SEPURI-SAI-KRISHNA opened a new pull request, #13243:
URL: https://github.com/apache/seatunnel/pull/13243
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Phase 1 of #13223 (explicit capability contract and routing validation).
Reported in #13221.
With `multi_table_sink_replica` above 1, a multi-table sink runs several
writers of each table in one subtask. A writer without a key inherited the
empty `primaryKey()` default, and an empty key was treated as permission to
spread rows over those writers at random. Two changes of the same record could
then be applied out of order: an older value could overwrite a newer one, or a
deleted row could come back, while the job still finished.
Changes, following the Phase 1 plan in #13223:
- New `MultiTableSinkReplicaMode` (`UNSUPPORTED`, `KEYED`, `RANDOM`) and
`SupportMultiTableSinkWriter.replicaMode()`, default `UNSUPPORTED`.
`primaryKey()` is unchanged, and `KEYED` is never inferred from it.
- `MultiTableSinkWriter` keeps `sinkPrimaryKeys` and adds
`sinkReplicaModes`, filled together. One `validateReplicaRouting(...)` helper
runs in initialization, before the queue workers start, so `createWriter()` and
`restoreWriter()` share it:
- one replica: every mode is accepted.
- several replicas: `UNSUPPORTED` is rejected, `KEYED` needs a present,
non-negative `primaryKey()`, `RANDOM` needs no key, and all writers of a table
must declare the same mode and, for `KEYED`, the same key index.
- the error names the option, the replica count, and each table with its
connector, writer class and invalid declaration (at most 10 tables, then a
count).
- created writers and the shared resource are released when the check, a
later writer, or the shared resource fails.
- `write()` keeps the control row handling, table lookup, failure policy and
error checks. It resolves the target table, computes one queue index and calls
`offerRowElement(index, element)` once: queue 0 with one replica, the existing
key hash for `KEYED`, a random queue for `RANDOM` (even when the writer has a
key), and a rejection for `UNSUPPORTED`. The old empty-key random branch and
the single-table random shortcut are removed.
- Removing a table's writers removes it from both caches.
- `JdbcSinkWriter` returns `KEYED`, so its keyed routing stays. A Jdbc table
without a key (none in the schema and no `primary_keys`) is rejected with
replicas by the `KEYED` check. `JdbcExactlyOnceSinkWriter` keeps the default.
- The benchmark `MultiTableSinkFixture` writer declares `KEYED`.
- The six sink doc examples (Feishu, Fluss, HugeGraph, InfluxDB, Prometheus,
Pulsar) and `mysql_cdc_to_iceberg_for_schema_change.conf` now use 1, with their
assertions unchanged. The Jdbc examples and E2E configs keep their replicas.
Not in this PR, as agreed in #13223: restoring with a changed parallelism or
replica count, state redistribution, the connector rows and their E2E coverage.
Notes for reviewers:
- Missing `tableId`: in a writer built for one table, a row whose table id
is missing (`null` or empty) or names no table resolves to that table and uses
its mode and key, so it is never made random. I kept the "names no table" case
because the queue worker already falls back to the single writer for it
(`MultiTableWriterRunnable`, `allowSingleWriterFallback`, from #10600), and
`MultiTableSinkWriterTest#testSingleWriterFallbackAcceptsExplicitTableId`
covers a row with table id `Optional[http]` for writer `http`. Sending such
rows to the unknown-table handling would fail those jobs. In a writer with
several tables, an unresolved row still goes to the unknown-table handling. I
can narrow this to `null` and empty only if you prefer.
- `selectQueueIndex` also checks that a `KEYED` index is inside the row, so
a bad index fails with the table and index instead of an
`IndexOutOfBoundsException`.
- #13210 moves `MultiTableSinkFixture` and adds `FlushWriter`, which runs
with more than one replica. Whichever of the two PRs lands second has to keep
the `KEYED` override of the moved fixture and make `FlushWriter` declare a
replica mode.
- #13242 (the `JDBC / normal` row: `byte[]` key hash and key refresh after a
schema change) changes the same routing code and `JdbcSinkWriter`, so whichever
of the two lands second rebases. With both, a key refreshed to empty (the key
column dropped) on a `KEYED` table with several replicas fails its next row in
`selectQueueIndex`, because the replica check runs when the writer is built.
That row can decide whether this is the wanted behavior.
- #11206 creates writers while the job runs. It has to call
`MultiTableSinkWriter.validateReplicaRouting` for them before they are
registered.
- #11077, #12366, #13201, #13060 and #10874 touch the same files, and so do
#13235 and #13238 (#13232, replica count in `OptionRule`), which conflict with
this PR only where both add tests in `MultiTableSinkWriterTest`. I will rebase
on whichever lands first.
- CI `benchmark-test` fails in `CheckpointStorageBenchmark`
(`checkpointOverviewIncrementalUpdate` and `checkpointPersistenceTransaction`,
"Checkpoint overview was not stored for job ..."). The cause is already on dev,
from #13015: it stopped persisting `engine_checkpoint_monitor`, while
`CheckpointStorageBenchmarkFixture.reloadOverviewSamples` still evicts the
overviews and reloads them from that store. With only `FileMapStore` taken from
before #13015, both pass (see Local runs). The job runs here only because this
PR touches `seatunnel-benchmarks`. I can fix the fixture in a separate PR.
### Does this PR introduce _any_ user-facing change?
Yes. A job that sets `multi_table_sink_replica` above 1 for a sink other
than Jdbc, for Jdbc with `is_exactly_once`, or for a Jdbc table without a
primary key now fails when the writer is created or restored, for example:
```
Option 'multi_table_sink_replica' is 2, so each table gets 2 writers in one
subtask, but this table does not support that: table 'main.t' (connector
'Console', writer
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter): its
writer declares neither KEYED nor RANDOM. Set 'multi_table_sink_replica' = 1. A
connector supports more than one writer per table only when its writer declares
KEYED with a valid primaryKey(), or RANDOM.
```
For a Jdbc table without a key the reason reads `its writer declares KEYED,
but primaryKey() returns no key`. Jobs that keep the default of 1 are not
affected. A custom connector whose writer relied on `primaryKey()` alone must
also return `KEYED` from `replicaMode()`.
Docs (EN and ZH): `incompatible-changes.md` entry with migration guide,
`multi_table_sink_replica` in Sink Common Options, the multi-table architecture
page, the Paimon and Iceberg option rows, and the six sink examples above.
### How was this patch tested?
New or extended tests in `MultiTableSinkWriterTest`:
- `testOneReplicaAcceptsEveryModeWithoutAKey`
- `testWriterWithoutReplicaModeIsRejectedWithReplicasAndClosed`
- `testKeyedWriterNeedsAValidKey` (missing and negative index)
- `testReplicasOfOneTableMustAgree` (mode and key index)
- `testKeyedRoutingKeepsTheLastChangeOfEachKeyWhenOneReplicaIsSlow` (40
keys, 5 versions, replica 4, one slow replica; table id set, `null`, empty, and
one that names no table)
- `testRandomWriterSpreadsRowsEvenWhenItHasAKey`
- `testEachAcceptedRowIsWrittenOnce` (`KEYED` and `RANDOM`, 1 and 3 replicas)
- `testRowOfAnUnresolvedTableFollowsTheFailurePolicy` (`FAIL_FAST` and
`CONTINUE_OTHER_TABLES`)
- `testRestoreChecksRestoredAndFreshWriters` and
`testRestoreWithTheSameLayoutRestoresEveryReplica`
- `testCreatedWritersAreClosedWhenAnotherWriterCannotBeCreated` and
`testWritersAreClosedWhenTheSharedResourceCannotBeCreated`
- `testRuntimeWriteIsolatesFailedTableAfterRetryExhausted` now also checks
that both caches drop the removed table
- `testKeyOutsideTheRowFailsWithTheTableAndIndex`,
`testReplicaErrorListsAtMostTenTables`,
`testReplicaCheckCanBeRunForWritersAddedLater`
and
`JdbcSinkWriterTest#testReplicaModeIsKeyedAndATableWithoutAKeyHasNoRoutingKey`.
The existing test writers declare `KEYED` or `RANDOM`.
I also broke the change on purpose, one mutation at a time, and a test
failed each time: `RANDOM` hashed by the key, no table resolution for a missing
id, the old random shortcut for such rows, `KEYED` routed at random, a row
offered twice, no queue 0 with one replica, a mode kept after table removal,
`UNSUPPORTED` accepted, and `KEYED` without a key accepted.
Local runs:
- `seatunnel-api`: 426 tests pass. `connector-jdbc`: 1233 run, all pass
except 4 Oracle and Kingbase Testcontainers tests, which could not reach Docker
in that run.
- `seatunnel-benchmarks` with `-Pbenchmark`: `MultiTableSinkFixtureTest` 4
tests pass. `benchmarks.jar` with the CI arguments and only `FileMapStore`
taken from before #13015: every benchmark passes, including
`MultiTableSinkBenchmark.recordHandoff` with 1, 5 and 15 replicas. With the
current `FileMapStore`, the two `CheckpointStorageBenchmark` cases above fail.
- Zeta local jobs (FakeSource, replica 2): Console and Jdbc without a key
are rejected with the messages above. Jdbc (SQLite) with `primary_keys =
["id"]`, 200 keys with 5 versions each: 2 writer queues, 1000 rows written,
every key at its last version.
- E2E on Zeta:
`MysqlCDCWithSchemaChangeIT#testMysqlCdcWithSchemaEvolutionCase` and
`#testMysqlCdcWithMultiDatabaseSameTableSchemaEvolution` (Jdbc, replica 2), and
`IcebergSinkCDCIT#testMysqlCdcCheckSchemaChangeE2e` and
`#testMysqlCdcCheckMultiSchemaChangeE2e` (now 1) pass. Before this restructure,
`PostgresSchemaChangeIT#testMysqlCdcWithSchemaEvolutionCase` (replica 2) also
passed.
- Fork CI
(https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/43049980882):
`jdbc-connectors-it-ddl`, which runs the SQL Server, DM and Postgres schema
change ITs, passes on JDK 11 and 17, as do `mysql-cdc-connector-it` and the
other Jdbc IT parts. Besides `benchmark-test` (above), the jobs that failed did
so before any SeaTunnel job ran, in the same way as in other PRs' CI this week:
`TiDBCDCIT` lost its connection to TiDB in `startUp` (also in #13228),
`TestPythonTransformIT` could not set up its container because `apt` failed
(also in #13228 and #13179, and the other 16 transform test classes passed),
and `connector-file-local-it` on JDK 11 hit the 2 hour job limit on a slow
runner (also in #13238; it passes on JDK 17). I will rerun them.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]