SEPURI-SAI-KRISHNA opened a new pull request, #13243:
URL: https://github.com/apache/seatunnel/pull/13243

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Phase 1 of #13223 (explicit capability contract and routing validation). 
Reported in #13221.
   
   With `multi_table_sink_replica` above 1, a multi-table sink runs several 
writers of each table in one subtask. A writer without a key inherited the 
empty `primaryKey()` default, and an empty key was treated as permission to 
spread rows over those writers at random. Two changes of the same record could 
then be applied out of order: an older value could overwrite a newer one, or a 
deleted row could come back, while the job still finished.
   
   Changes, following the Phase 1 plan in #13223:
   
   - New `MultiTableSinkReplicaMode` (`UNSUPPORTED`, `KEYED`, `RANDOM`) and 
`SupportMultiTableSinkWriter.replicaMode()`, default `UNSUPPORTED`. 
`primaryKey()` is unchanged, and `KEYED` is never inferred from it.
   - `MultiTableSinkWriter` keeps `sinkPrimaryKeys` and adds 
`sinkReplicaModes`, filled together. One `validateReplicaRouting(...)` helper 
runs in initialization, before the queue workers start, so `createWriter()` and 
`restoreWriter()` share it:
     - one replica: every mode is accepted.
     - several replicas: `UNSUPPORTED` is rejected, `KEYED` needs a present, 
non-negative `primaryKey()`, `RANDOM` needs no key, and all writers of a table 
must declare the same mode and, for `KEYED`, the same key index.
     - the error names the option, the replica count, and each table with its 
connector, writer class and invalid declaration (at most 10 tables, then a 
count).
     - created writers and the shared resource are released when the check, a 
later writer, or the shared resource fails.
   - `write()` keeps the control row handling, table lookup, failure policy and 
error checks. It resolves the target table, computes one queue index and calls 
`offerRowElement(index, element)` once: queue 0 with one replica, the existing 
key hash for `KEYED`, a random queue for `RANDOM` (even when the writer has a 
key), and a rejection for `UNSUPPORTED`. The old empty-key random branch and 
the single-table random shortcut are removed.
   - Removing a table's writers removes it from both caches.
   - `JdbcSinkWriter` returns `KEYED`, so its keyed routing stays. A Jdbc table 
without a key (none in the schema and no `primary_keys`) is rejected with 
replicas by the `KEYED` check. `JdbcExactlyOnceSinkWriter` keeps the default.
   - The benchmark `MultiTableSinkFixture` writer declares `KEYED`.
   - The six sink doc examples (Feishu, Fluss, HugeGraph, InfluxDB, Prometheus, 
Pulsar) and `mysql_cdc_to_iceberg_for_schema_change.conf` now use 1, with their 
assertions unchanged. The Jdbc examples and E2E configs keep their replicas.
   
   Not in this PR, as agreed in #13223: restoring with a changed parallelism or 
replica count, state redistribution, the connector rows and their E2E coverage.
   
   Notes for reviewers:
   
   - Missing `tableId`: in a writer built for one table, a row whose table id 
is missing (`null` or empty) or names no table resolves to that table and uses 
its mode and key, so it is never made random. I kept the "names no table" case 
because the queue worker already falls back to the single writer for it 
(`MultiTableWriterRunnable`, `allowSingleWriterFallback`, from #10600), and 
`MultiTableSinkWriterTest#testSingleWriterFallbackAcceptsExplicitTableId` 
covers a row with table id `Optional[http]` for writer `http`. Sending such 
rows to the unknown-table handling would fail those jobs. In a writer with 
several tables, an unresolved row still goes to the unknown-table handling. I 
can narrow this to `null` and empty only if you prefer.
   - `selectQueueIndex` also checks that a `KEYED` index is inside the row, so 
a bad index fails with the table and index instead of an 
`IndexOutOfBoundsException`.
   - #13210 moves `MultiTableSinkFixture` and adds `FlushWriter`, which runs 
with more than one replica. Whichever of the two PRs lands second has to keep 
the `KEYED` override of the moved fixture and make `FlushWriter` declare a 
replica mode.
   - #13242 (the `JDBC / normal` row: `byte[]` key hash and key refresh after a 
schema change) changes the same routing code and `JdbcSinkWriter`, so whichever 
of the two lands second rebases. With both, a key refreshed to empty (the key 
column dropped) on a `KEYED` table with several replicas fails its next row in 
`selectQueueIndex`, because the replica check runs when the writer is built. 
That row can decide whether this is the wanted behavior.
   - #11206 creates writers while the job runs. It has to call 
`MultiTableSinkWriter.validateReplicaRouting` for them before they are 
registered.
   - #11077, #12366, #13201, #13060 and #10874 touch the same files, and so do 
#13235 and #13238 (#13232, replica count in `OptionRule`), which conflict with 
this PR only where both add tests in `MultiTableSinkWriterTest`. I will rebase 
on whichever lands first.
   - CI `benchmark-test` fails in `CheckpointStorageBenchmark` 
(`checkpointOverviewIncrementalUpdate` and `checkpointPersistenceTransaction`, 
"Checkpoint overview was not stored for job ..."). The cause is already on dev, 
from #13015: it stopped persisting `engine_checkpoint_monitor`, while 
`CheckpointStorageBenchmarkFixture.reloadOverviewSamples` still evicts the 
overviews and reloads them from that store. With only `FileMapStore` taken from 
before #13015, both pass (see Local runs). The job runs here only because this 
PR touches `seatunnel-benchmarks`. I can fix the fixture in a separate PR.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. A job that sets `multi_table_sink_replica` above 1 for a sink other 
than Jdbc, for Jdbc with `is_exactly_once`, or for a Jdbc table without a 
primary key now fails when the writer is created or restored, for example:
   
   ```
   Option 'multi_table_sink_replica' is 2, so each table gets 2 writers in one 
subtask, but this table does not support that: table 'main.t' (connector 
'Console', writer 
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter): its 
writer declares neither KEYED nor RANDOM. Set 'multi_table_sink_replica' = 1. A 
connector supports more than one writer per table only when its writer declares 
KEYED with a valid primaryKey(), or RANDOM.
   ```
   
   For a Jdbc table without a key the reason reads `its writer declares KEYED, 
but primaryKey() returns no key`. Jobs that keep the default of 1 are not 
affected. A custom connector whose writer relied on `primaryKey()` alone must 
also return `KEYED` from `replicaMode()`.
   
   Docs (EN and ZH): `incompatible-changes.md` entry with migration guide, 
`multi_table_sink_replica` in Sink Common Options, the multi-table architecture 
page, the Paimon and Iceberg option rows, and the six sink examples above.
   
   ### How was this patch tested?
   
   New or extended tests in `MultiTableSinkWriterTest`:
   
   - `testOneReplicaAcceptsEveryModeWithoutAKey`
   - `testWriterWithoutReplicaModeIsRejectedWithReplicasAndClosed`
   - `testKeyedWriterNeedsAValidKey` (missing and negative index)
   - `testReplicasOfOneTableMustAgree` (mode and key index)
   - `testKeyedRoutingKeepsTheLastChangeOfEachKeyWhenOneReplicaIsSlow` (40 
keys, 5 versions, replica 4, one slow replica; table id set, `null`, empty, and 
one that names no table)
   - `testRandomWriterSpreadsRowsEvenWhenItHasAKey`
   - `testEachAcceptedRowIsWrittenOnce` (`KEYED` and `RANDOM`, 1 and 3 replicas)
   - `testRowOfAnUnresolvedTableFollowsTheFailurePolicy` (`FAIL_FAST` and 
`CONTINUE_OTHER_TABLES`)
   - `testRestoreChecksRestoredAndFreshWriters` and 
`testRestoreWithTheSameLayoutRestoresEveryReplica`
   - `testCreatedWritersAreClosedWhenAnotherWriterCannotBeCreated` and 
`testWritersAreClosedWhenTheSharedResourceCannotBeCreated`
   - `testRuntimeWriteIsolatesFailedTableAfterRetryExhausted` now also checks 
that both caches drop the removed table
   - `testKeyOutsideTheRowFailsWithTheTableAndIndex`, 
`testReplicaErrorListsAtMostTenTables`, 
`testReplicaCheckCanBeRunForWritersAddedLater`
   
   and 
`JdbcSinkWriterTest#testReplicaModeIsKeyedAndATableWithoutAKeyHasNoRoutingKey`. 
The existing test writers declare `KEYED` or `RANDOM`.
   
   I also broke the change on purpose, one mutation at a time, and a test 
failed each time: `RANDOM` hashed by the key, no table resolution for a missing 
id, the old random shortcut for such rows, `KEYED` routed at random, a row 
offered twice, no queue 0 with one replica, a mode kept after table removal, 
`UNSUPPORTED` accepted, and `KEYED` without a key accepted.
   
   Local runs:
   
   - `seatunnel-api`: 426 tests pass. `connector-jdbc`: 1233 run, all pass 
except 4 Oracle and Kingbase Testcontainers tests, which could not reach Docker 
in that run.
   - `seatunnel-benchmarks` with `-Pbenchmark`: `MultiTableSinkFixtureTest` 4 
tests pass. `benchmarks.jar` with the CI arguments and only `FileMapStore` 
taken from before #13015: every benchmark passes, including 
`MultiTableSinkBenchmark.recordHandoff` with 1, 5 and 15 replicas. With the 
current `FileMapStore`, the two `CheckpointStorageBenchmark` cases above fail.
   - Zeta local jobs (FakeSource, replica 2): Console and Jdbc without a key 
are rejected with the messages above. Jdbc (SQLite) with `primary_keys = 
["id"]`, 200 keys with 5 versions each: 2 writer queues, 1000 rows written, 
every key at its last version.
   - E2E on Zeta: 
`MysqlCDCWithSchemaChangeIT#testMysqlCdcWithSchemaEvolutionCase` and 
`#testMysqlCdcWithMultiDatabaseSameTableSchemaEvolution` (Jdbc, replica 2), and 
`IcebergSinkCDCIT#testMysqlCdcCheckSchemaChangeE2e` and 
`#testMysqlCdcCheckMultiSchemaChangeE2e` (now 1) pass. Before this restructure, 
`PostgresSchemaChangeIT#testMysqlCdcWithSchemaEvolutionCase` (replica 2) also 
passed.
   - Fork CI 
(https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/43049980882): 
`jdbc-connectors-it-ddl`, which runs the SQL Server, DM and Postgres schema 
change ITs, passes on JDK 11 and 17, as do `mysql-cdc-connector-it` and the 
other Jdbc IT parts. Besides `benchmark-test` (above), the jobs that failed did 
so before any SeaTunnel job ran, in the same way as in other PRs' CI this week: 
`TiDBCDCIT` lost its connection to TiDB in `startUp` (also in #13228), 
`TestPythonTransformIT` could not set up its container because `apt` failed 
(also in #13228 and #13179, and the other 16 transform test classes passed), 
and `connector-file-local-it` on JDK 11 hit the 2 hour job limit on a slow 
runner (also in #13238; it passes on JDK 17). I will rerun them.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the 
following files are updated: (not a new connector)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to