fatmanverse opened a new pull request, #12803:
URL: https://github.com/apache/seatunnel/pull/12803
Fixes #12799
### Purpose of this pull request
When the Jdbc sink writes with a key, `BufferReducedBatchStatementExecutor`
buffers rows in a `LinkedHashMap` keyed by the extracted key row, so that only
the last change of a key is written in a batch. `SeaTunnelRow#equals` uses
`Arrays.deepEquals`, so all rows whose key contains NULL are equal and collapse
into one entry. The other rows are dropped before any SQL runs.
A key can contain NULL when it comes from a nullable unique key, either
derived from the upstream table (no primary key) or configured in
`primary_keys`. In SQL, NULL never equals NULL: the database keeps every such
row, `INSERT ... ON CONFLICT (k)` does not conflict on them, and `DELETE ...
WHERE k = ?` does not match them.
This PR buffers a row whose key contains a NULL field under its own
identity, so it is not merged with other rows. Rows with a complete key are
reduced exactly as before. Insertion order of the buffer, and so the order of
upserts and deletes, is unchanged.
The first approach considered was to stop deriving `primary_keys` from a
nullable unique key. It was not taken because it switches such tables to the
plain-insert executor, which writes every row kind as an INSERT and would break
UPDATE/DELETE events from CDC sources. The issue is in the buffer, not in the
choice of key.
### Does this PR introduce _any_ user-facing change?
Yes. Rows whose sink key contains NULL are no longer silently dropped. In
the reported case (30 rows, 10 with a NULL unique key), all 30 rows are written
instead of 21.
### How was this patch tested?
- Added
`BufferReducedBatchStatementExecutorTest#testRowsWithSameKeyAreReduced`,
`#testRowsWithNullKeyAreNotReduced` and
`#testDeleteWithNullKeyKeepsOrderWithUpserts`.
- `./mvnw -pl seatunnel-connectors-v2/connector-jdbc test` on JDK 17: all
tests pass.
- On a deployed Zeta cluster, Jdbc MySQL -> Jdbc PostgreSQL with a
no-primary-key table and a nullable unique key (30 rows, 10 NULL) wrote 21 rows
before this change and 30 after it, with the same `ON CONFLICT ("code")`
statement.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs (not needed)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (not needed, this only fixes data loss)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
