SEPURI-SAI-KRISHNA opened a new pull request, #13250:
URL: https://github.com/apache/seatunnel/pull/13250
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Part of #13223, the `JDBC / normal` row: the UNIQUE-key gap that #13242 does
not cover.
When a table has no primary key and `primary_keys` is not set,
`JdbcSinkFactory` fills `primary_keys` from the first UNIQUE key of the table,
as `Jdbc.md` documents ("SeaTunnel attempts to inherit a primary key or the
first unique key"). The sink then upserts and deletes by that key. But
`JdbcSink.getPrimaryKeyIndex` read only the schema primary key, which stays
empty for such a table, so `JdbcSinkWriter.primaryKey()` returned nothing. With
`multi_table_sink_replica` above 1, `MultiTableSinkWriter` then spread the
changes of one key over several writers at random, and they were applied out of
order.
Changes:
- `JdbcSink.getPrimaryKeyIndex(tableSchema, sinkPrimaryKeys)`: the first
column of the schema primary key, as before; when the schema has none, the
first column of `primary_keys`, the key the sink writes by. A name that is not
in the row gives no key instead of an exception.
- `JdbcSink.routingKeyIndex()` (package-private) gives `createWriter` that
index. `restoreWriter` of the normal writer goes through `createWriter`, so
restored writers get it too.
- The exactly-once writer is not changed; it has no routing key today (row
`JDBC / exactly-once (XA)` of #13223).
Routing by one column of the key keeps every change of one record on one
writer, also for a key with several columns, because those changes carry the
same value in each key column.
Notes for reviewers:
- #13242 changes `JdbcSinkWriter` (key tracked by column name after a schema
change) and does not touch `JdbcSink.java`. Its tracking starts from the index
`JdbcSink` passes, so it also follows the UNIQUE column. The two PRs do not
conflict.
- With #13243, a Jdbc table without a key is rejected when it has several
replicas. This PR gives a UNIQUE-only table a key, so such a table is accepted
again. Its docs say a Jdbc table "without a primary key" fails with replicas;
whichever of the two lands second will also mention the UNIQUE key there (EN
and ZH).
- #11932 refactors the same lines of `JdbcSink.createWriter` for another
purpose, so one of the two will rebase.
### Does this PR introduce _any_ user-facing change?
Yes, for a Jdbc table with only a UNIQUE key and `multi_table_sink_replica`
above 1: its changes of one key now go to one writer and are applied in order.
Before, the job could fail on a key conflict, or finish with deleted rows back
and old values. No option, default, API or state changes. Jobs with one
replica, a primary key, or `primary_keys` set are not affected.
### How was this patch tested?
- `JdbcSinkTest`: the schema primary key wins over `primary_keys`; the
fallback to `primary_keys`; no key at all; a `primary_keys` name that is not in
the row. The three existing cases now pass `null` for the new argument.
- `JdbcSinkFactoryTest` (through the factory, as a job builds the sink): a
table with only a UNIQUE key routes by its UNIQUE column; configured
`primary_keys`, a schema primary key, and no key route as before.
-
`JdbcSinkCDCChangelogIT#testUniqueKeyTableKeepsTheLastChangeOfEachKeyWithReplicas`
(Postgres, `jdbc_sink_cdc_changelog_unique_key_replicas.conf`): a UNIQUE-only
key, 100 keys, 350 CDC rows (insert, update, delete of every second key),
replica 2; the table must hold exactly the 50 remaining keys at their last
value. Zeta: passes (2 writer queues); Flink runs in CI. With the fallback
removed, the job still exits 0, but the table has 61 rows instead of 50: 11
deleted keys back, 17 rows with the old value, 12 rows missing.
- With the fallback removed, the two new UNIQUE-key unit tests fail as well.
- `connector-jdbc` unit tests: 1238 run, all pass except 4 Oracle and
Kingbase Testcontainers tests, which could not reach Docker in that run.
- Real Zeta jobs, FakeSource to Jdbc on DuckDB, UNIQUE-only key, 1000 keys,
3500 CDC rows, 500 rows expected:
| build | replicas | result |
| --- | --- | --- |
| dev `535d4237dd` (same `JdbcSink` code as now) | 2 | job fails (`Conflict
on tuple deletion!`); 713 rows, 225 deleted keys back, 208 old values |
| the same build with this PR's `JdbcSink` | 1 | 500 rows, correct |
| the same build with this PR's `JdbcSink` | 2 | 500 rows, correct (2
queues) |
| the same build with this PR's `JdbcSink` | 4 | 500 rows, correct (4
queues) |
- `spotless:check` and `verify -DskipTests` on `connector-jdbc` and
`connector-jdbc-e2e-part-3`.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs (`Jdbc.md` already
documents the UNIQUE key fallback)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (not incompatible)
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]