SEPURI-SAI-KRISHNA commented on issue #13223: URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6788416172
I would like to work on the two API rows: the capability contract, and the fail-fast validation on create and restore with the null `tableId` routing and cleanup. Audited on dev `75e19dadd7`; the sink API has not changed since `f5374addb3`. My last comment on #13221 crossed with this issue, which answers it. I will follow this plan: fail fast, `UNORDERED` only when a writer declares it, no key taken from `TableSchema`, and `byte[]` hashing and key refresh after a schema change left to follow-ups. Plan: - `SupportMultiTableSinkWriter` gets a default method that returns the capability: `KEYED` when `primaryKey()` is present, otherwise `UNSUPPORTED`. `primaryKey()` stays as it is, so the Jdbc keyed path is unchanged. - The check runs when `MultiTableSinkWriter` is built, which `createWriter` and every `restoreWriter` branch (also the fresh writer branch) go through, and before any queue worker starts. All replicas of a table must resolve to the same capability and key index, and `KEYED` needs a key index inside the row. On failure the writers already created are closed. The message names the writer class, the table, `multi_table_sink_replica`, its value and the fix. - A row without `tableId` in a single-table job uses the routing of that table. - Tests go into `MultiTableSinkWriterTest`, with controlled writer delays, following the acceptance list. Docs EN and ZH and `incompatible-changes.md`. Two questions: 1. With only this PR, `mysql_cdc_to_iceberg_for_schema_change.conf` (Iceberg, `multi_table_sink_replica = 2`) fails, and so do the doc examples with 2 for Feishu and Pulsar. Should this PR set them to 1, or should the Iceberg row land first? I can take the Iceberg row next. 2. Migration by lowering the option loses state on restore. I restored the state of a job with replica 2 and parallelism 2 through `MultiTableSink.restoreWriter` with replica 1, giving each subtask its own states. Only old index 0 was restored; the states of indexes 1, 2 and 3 were dropped without a message, and subtask 1 got a fresh writer. So `multi_table_sink_replica = 1` is safe for a new job but not for a restore. Should the restore also fail when a state id does not map to the new replica count, or should the migration guide say to start the job without restoring? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
