SEPURI-SAI-KRISHNA opened a new pull request, #13126:
URL: https://github.com/apache/seatunnel/pull/13126
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #13124.
The Cassandra source read a NULL number or boolean as `0` or `false`, so
jobs succeeded with values that were never in the table. Other NULL types
failed the job. The sink failed on most NULL values and wrote a NULL `inet` as
`127.0.0.1`. The issue has the full table per type.
Three checks:
- `TypeConvertUtil#buildSeaTunnelRow` checks `row.isNull(i)` before the type
switch, so a NULL cell stays a NULL field.
- `TypeConvertUtil#reconvertAndInject` calls `statement.setToNull(index)`
for a null value, which is what the `text`, `time`, `date` and `map` setters
already end up doing.
- `CassandraSinkWriter` fails the write when a primary key column is NULL,
naming the table and column.
They belong in one PR. The sink only got by on numbers because the source
never handed it a NULL, so fixing the source alone would turn a succeeding
Cassandra to Cassandra job into a failing one.
The key check is needed because of the second change. The server rejects a
NULL key, and that rejection does not fail the job: `async_write = true` only
logs it, and with `async_write = false` the job still finishes after losing the
good rows in the batch. Without the check, a NULL `bigint` key would go from a
loud NullPointerException to silent loss, which is what a NULL `text` key does
on `dev` today. The writer reads the key columns from the driver's table
metadata, using the keyspace and table the prepared statement reports, so
`keyspace.table` names work. If that metadata is not available, the check does
nothing and the behavior for a NULL key is the server rejection above.
A NULL `list`, `set` or `map` now comes through as NULL rather than empty.
Cassandra does not store an empty collection separately from NULL, and this
keeps one rule for every type.
A NULL value overwrites what the target row already has in that column, as
it should for an upsert, and leaves a tombstone like any NULL written to
Cassandra. The `text` setter already behaves this way.
Not fixed here: a failed async write still does not fail the job in general.
I will raise that separately too.
Not in this PR: the sink still fails on a non-NULL `DECIMAL` or `VARINT`
from other sources, because it casts them to `Double` and `String`. I will
raise that separately.
### Does this PR introduce _any_ user-facing change?
Yes, and it is in `incompatible-changes.md` (EN and ZH). Jobs that received
`0`, `false` or an empty collection for a NULL cell now receive NULL, so a
target column declared `NOT NULL` will now reject these rows. Jobs that failed
on a NULL value now succeed, unless the NULL is in a primary key column, which
now always fails the job. That includes `text`, `ascii`, `time` and `date`
keys, where a job used to succeed while losing rows. Values that are not NULL,
including a stored `0` or `false`, do not change.
### How was this patch tested?
**Unit tests** (JDK 11 and 17): `TypeConvertUtilTest` covers NULL cells of
every type through a real driver `DefaultRow`, null values of every type
through `reconvertAndInject`, and a stored `0` and `false`. On `dev` exactly
the two NULL tests fail. `CassandraSinkWriterTest` is new, since the writer had
no test: a NULL key with async and sync writes and with a `keyspace.table`
name, and a NULL in another column. It needed a package-private constructor
that takes the session. Removing the key check fails exactly the three NULL key
tests. `connector-cassandra` passes (33 run, 0 failures).
**Real Cassandra 4.1.1 job on Zeta**, Cassandra to Cassandra, where source
row `id = 1` has every other column NULL and row `id = 2` holds real zeros,
`c_int = 7` and `false`. On `dev`:
```
numeric columns: status=FINISHED
id=1: c_bigint=0 c_boolean=false c_double=0.0 c_float=0.0 c_int=0
c_smallint=0 c_tinyint=0
id=2: c_bigint=0 c_boolean=false c_double=0.0 c_float=0.0 c_int=7
c_smallint=0 c_tinyint=0
all 21 column types: status=FAILED, java.lang.NullPointerException, 0 rows
written
```
With this PR:
```
numeric columns: status=FINISHED
id=1: all 7 columns NULL
id=2: c_bigint=0 c_boolean=false c_double=0.0 c_float=0.0 c_int=7
c_smallint=0 c_tinyint=0
all 21 column types: status=FINISHED, 2 rows written
id=1: all 21 columns NULL
id=2: the same 7 values, the other 14 columns NULL
```
**NULL keys on the same server, FakeSource to Cassandra**, one row with a
NULL key and one good row:
```
dev this PR
bigint key, async FAILED, NullPointerException FAILED,
Primary key column id of table key_big is NULL
bigint key, sync FAILED, NullPointerException FAILED, same
message
text key, async FINISHED, NULL key row lost FAILED,
Primary key column k of table key_text is NULL
text key, sync FINISHED, both rows lost FAILED, same
message
```
With this PR a NULL clustering column and a `keyspace.table` name fail the
same way, `async_write = false` writes the all-NULL row with every column NULL,
a NULL overwrites an existing value, and an empty `text` and `blob` stay empty
rather than becoming NULL.
**`CassandraIT`** gains a source row with only NULL cells, and
`testCassandra` now asserts that every column of that row is NULL in the sink.
`testNullPrimaryKeyFailsTheJob` is new. I compiled it locally. It runs on Zeta,
Flink and Spark in this branch's fork CI, not on my machine, and I have not
checked the Flink translation myself.
Also run: `spotless:check`.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs (no option changes;
behavior change is in `incompatible-changes.md`)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
