nzw921rx commented on issue #13223:
URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6793626255

   > [@nzw921rx](https://github.com/nzw921rx) Thanks, `JdbcSinkWriter` will 
return `KEYED` in this PR, so the five Jdbc E2E configs and the Jdbc examples 
keep their replicas. `JdbcExactlyOnceSinkWriter` keeps the default until its 
own row.
   > 
   > Two last points before I write the code.
   > 
   > 1. For a Jdbc table with no primary key in its schema and no 
`primary_keys`, `primaryKey()` is empty and rows go at random today. Should 
`replicaMode()` return `KEYED` only when the key is present and `UNSUPPORTED` 
otherwise, so such a table is rejected with replicas, or `RANDOM` to keep 
today's routing until the JDBC row checks it? I would take `UNSUPPORTED`.
   > 2. Writer state is also dropped when only the parallelism goes down, with 
the default replica 1. On Zeta, `CheckpointCoordinator.restoreTaskState` gives 
new subtask `s` the states of old subtasks `s`, `s + p`, `s + 2p` and so on 
(`p` is the new parallelism, and since [[Fix][Zeta] Checkpoint restore remap 
duplicates subtask state for parallelism > 1 
#12195](https://github.com/apache/seatunnel/pull/12195) each state reaches 
exactly one subtask), but `MultiTableSink.restoreWriter` only looks up the ids 
of its own layout, `subtask * replica + i`. I replayed that assignment against 
`MultiTableSink` from dev `f5374addb3`:
   > 
   > old parallelism, replica   new parallelism, replica        old writer 
states restored
   > 2, 1       1, 1    1 of 2
   > 4, 1       2, 1    2 of 4
   > 2, 2       1, 2    2 of 4
   > 2, 2       2, 1    1 of 4
   > 2, 1       2, 2    1 of 2
   > A sink that is not wrapped loses nothing here, because Zeta passes all of 
these states to its `restoreWriter`. So a check that fails whenever a state id 
is outside the new layout would also fail every scale-down restore of a 
multi-table sink, at replica 1 too. Two ways:
   > 
   > * Fail fast as you said, and document that a scale-down restore of a 
multi-table sink fails until the job starts without state.
   > * Keep every state: give each state of a table to that table's writer at 
slot `index % replica` on this subtask. This is the merge Zeta already does for 
a sink that is not wrapped, so `restoreWriter` already has to accept states 
from several old writers. State of a table that is no longer configured is 
ignored as today.
   > 
   > I prefer to keep every state, with a test for each row above. Which one do 
you want, and in this PR or a separate issue?
   
   Thanks for investigating these cases.
   
   1. For JDBC, I agree with `UNSUPPORTED` when `primaryKey()` is empty. Return 
`KEYED` only when a key is present.
   
   2. Restoring with a changed parallelism or replica count is outside the 
scope of this PR. Adding state redistribution would increase complexity beyond 
the capability contract and routing fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to