SEPURI-SAI-KRISHNA opened a new pull request, #13250:
URL: https://github.com/apache/seatunnel/pull/13250

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Part of #13223, the `JDBC / normal` row: the UNIQUE-key gap that #13242 does 
not cover.
   
   When a table has no primary key and `primary_keys` is not set, 
`JdbcSinkFactory` fills `primary_keys` from the first UNIQUE key of the table, 
as `Jdbc.md` documents ("SeaTunnel attempts to inherit a primary key or the 
first unique key"). The sink then upserts and deletes by that key. But 
`JdbcSink.getPrimaryKeyIndex` read only the schema primary key, which stays 
empty for such a table, so `JdbcSinkWriter.primaryKey()` returned nothing. With 
`multi_table_sink_replica` above 1, `MultiTableSinkWriter` then spread the 
changes of one key over several writers at random, and they were applied out of 
order.
   
   Changes:
   
   - `JdbcSink.getPrimaryKeyIndex(tableSchema, sinkPrimaryKeys)`: the first 
column of the schema primary key, as before; when the schema has none, the 
first column of `primary_keys`, the key the sink writes by. A name that is not 
in the row gives no key instead of an exception.
   - `JdbcSink.routingKeyIndex()` (package-private) gives `createWriter` that 
index. `restoreWriter` of the normal writer goes through `createWriter`, so 
restored writers get it too.
   - The exactly-once writer is not changed; it has no routing key today (row 
`JDBC / exactly-once (XA)` of #13223).
   
   Routing by one column of the key keeps every change of one record on one 
writer, also for a key with several columns, because those changes carry the 
same value in each key column.
   
   Notes for reviewers:
   
   - #13242 changes `JdbcSinkWriter` (key tracked by column name after a schema 
change) and does not touch `JdbcSink.java`. Its tracking starts from the index 
`JdbcSink` passes, so it also follows the UNIQUE column. The two PRs do not 
conflict.
   - With #13243, a Jdbc table without a key is rejected when it has several 
replicas. This PR gives a UNIQUE-only table a key, so such a table is accepted 
again. Its docs say a Jdbc table "without a primary key" fails with replicas; 
whichever of the two lands second will also mention the UNIQUE key there (EN 
and ZH).
   - #11932 refactors the same lines of `JdbcSink.createWriter` for another 
purpose, so one of the two will rebase.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, for a Jdbc table with only a UNIQUE key and `multi_table_sink_replica` 
above 1: its changes of one key now go to one writer and are applied in order. 
Before, the job could fail on a key conflict, or finish with deleted rows back 
and old values. No option, default, API or state changes. Jobs with one 
replica, a primary key, or `primary_keys` set are not affected.
   
   ### How was this patch tested?
   
   - `JdbcSinkTest`: the schema primary key wins over `primary_keys`; the 
fallback to `primary_keys`; no key at all; a `primary_keys` name that is not in 
the row. The three existing cases now pass `null` for the new argument.
   - `JdbcSinkFactoryTest` (through the factory, as a job builds the sink): a 
table with only a UNIQUE key routes by its UNIQUE column; configured 
`primary_keys`, a schema primary key, and no key route as before.
   - 
`JdbcSinkCDCChangelogIT#testUniqueKeyTableKeepsTheLastChangeOfEachKeyWithReplicas`
 (Postgres, `jdbc_sink_cdc_changelog_unique_key_replicas.conf`): a UNIQUE-only 
key, 100 keys, 350 CDC rows (insert, update, delete of every second key), 
replica 2; the table must hold exactly the 50 remaining keys at their last 
value. Zeta: passes (2 writer queues); Flink runs in CI. With the fallback 
removed, the job still exits 0, but the table has 61 rows instead of 50: 11 
deleted keys back, 17 rows with the old value, 12 rows missing.
   - With the fallback removed, the two new UNIQUE-key unit tests fail as well.
   - `connector-jdbc` unit tests: 1238 run, all pass except 4 Oracle and 
Kingbase Testcontainers tests, which could not reach Docker in that run.
   - Real Zeta jobs, FakeSource to Jdbc on DuckDB, UNIQUE-only key, 1000 keys, 
3500 CDC rows, 500 rows expected:
   
   | build | replicas | result |
   | --- | --- | --- |
   | dev `535d4237dd` (same `JdbcSink` code as now) | 2 | job fails (`Conflict 
on tuple deletion!`); 713 rows, 225 deleted keys back, 208 old values |
   | the same build with this PR's `JdbcSink` | 1 | 500 rows, correct |
   | the same build with this PR's `JdbcSink` | 2 | 500 rows, correct (2 
queues) |
   | the same build with this PR's `JdbcSink` | 4 | 500 rows, correct (4 
queues) |
   
   - `spotless:check` and `verify -DskipTests` on `connector-jdbc` and 
`connector-jdbc-e2e-part-3`.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs (`Jdbc.md` already 
documents the UNIQUE key fallback)
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR. (not incompatible)
   * [ ] If you are contributing the connector code, please check that the 
following files are updated: (not a new connector)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to