zephyr45 commented on code in PR #13060:
URL: https://github.com/apache/seatunnel/pull/13060#discussion_r4599791579
##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/SinkContextProxy.java:
##########
@@ -44,6 +44,14 @@ public int getIndexOfSubtask() {
return index;
}
+ @Override
Review Comment:
Yes for retries with the same replica count; not as a permanent identity
across a replica-count change.
The calculation is `parentStableIndex * replicaNum + replicaOffset`. For
parent index 1 and two replicas, the stable indexes are 2 and 3 even when
Spark's task-attempt ID changes. With four replicas they become 4–7; with one
replica the index is 1. `getNumberOfParallelSubtasks()` instead returns the
total count, `parentParallelism * replicaNum`.
I checked your 2→4 and 4→1 examples against the actual
`MultiTableSink.restoreWriter` using a local Java 11 probe with a recording
child sink. With one parent subtask, 2→4 restores states 0/1 and creates fresh
writers 2/3. On 4→1, only state 0 reaches the child sink; states 1/2/3 are not
redistributed. The [existing exact-index
lookup](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java#L197-L223)
is unchanged by this PR. So this accessor does **not** establish safe stateful
replica rescaling. Supporting that needs a general multi-table state
redistribution contract, as you suggest.
On the broader compatibility concern, the proxy change only forwards the new
identity while retaining the replica offset. The existing index, writer count,
state keys and routing stay unchanged. The default accessor delegates to the
existing index; Spark overrides it with the partition ID, and Kafka is the only
connector currently reading it. Other sinks keep their historical attempt-based
identifiers. This does not add Kafka multi-table support.
The existing `MultiTableSinkWriterTest` suite passed again today: 32 tests,
zero failures/errors/skips. The separate probe checked distinct stable indexes
at replica counts 1/2/3/4 across retries; its restore checks demonstrate the
limitation above, not a successful rescale. I would keep the shared API change
limited to retry identity and leave replica-state redistribution as an explicit
separate design question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]