nzw921rx commented on issue #13223: URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6793626255
> [@nzw921rx](https://github.com/nzw921rx) Thanks, `JdbcSinkWriter` will return `KEYED` in this PR, so the five Jdbc E2E configs and the Jdbc examples keep their replicas. `JdbcExactlyOnceSinkWriter` keeps the default until its own row. > > Two last points before I write the code. > > 1. For a Jdbc table with no primary key in its schema and no `primary_keys`, `primaryKey()` is empty and rows go at random today. Should `replicaMode()` return `KEYED` only when the key is present and `UNSUPPORTED` otherwise, so such a table is rejected with replicas, or `RANDOM` to keep today's routing until the JDBC row checks it? I would take `UNSUPPORTED`. > 2. Writer state is also dropped when only the parallelism goes down, with the default replica 1. On Zeta, `CheckpointCoordinator.restoreTaskState` gives new subtask `s` the states of old subtasks `s`, `s + p`, `s + 2p` and so on (`p` is the new parallelism, and since [[Fix][Zeta] Checkpoint restore remap duplicates subtask state for parallelism > 1 #12195](https://github.com/apache/seatunnel/pull/12195) each state reaches exactly one subtask), but `MultiTableSink.restoreWriter` only looks up the ids of its own layout, `subtask * replica + i`. I replayed that assignment against `MultiTableSink` from dev `f5374addb3`: > > old parallelism, replica new parallelism, replica old writer states restored > 2, 1 1, 1 1 of 2 > 4, 1 2, 1 2 of 4 > 2, 2 1, 2 2 of 4 > 2, 2 2, 1 1 of 4 > 2, 1 2, 2 1 of 2 > A sink that is not wrapped loses nothing here, because Zeta passes all of these states to its `restoreWriter`. So a check that fails whenever a state id is outside the new layout would also fail every scale-down restore of a multi-table sink, at replica 1 too. Two ways: > > * Fail fast as you said, and document that a scale-down restore of a multi-table sink fails until the job starts without state. > * Keep every state: give each state of a table to that table's writer at slot `index % replica` on this subtask. This is the merge Zeta already does for a sink that is not wrapped, so `restoreWriter` already has to accept states from several old writers. State of a table that is no longer configured is ignored as today. > > I prefer to keep every state, with a test for each row above. Which one do you want, and in this PR or a separate issue? Thanks for investigating these cases. 1. For JDBC, I agree with `UNSUPPORTED` when `primaryKey()` is empty. Return `KEYED` only when a key is present. 2. Restoring with a changed parallelism or replica count is outside the scope of this PR. Adding state redistribution would increase complexity beyond the capability contract and routing fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
