nzw921rx commented on PR #10964:
URL: https://github.com/apache/seatunnel/pull/10964#issuecomment-4591594758
Apologies, my earlier comments may not have been clear enough. Let me
reorganize my thoughts here.
I've identified a few points in the current approach that I think are worth
discussing:
- Architectural consistency
The `Sink→Sink` point-to-point shuffle effectively introduces an
additional data channel outside the engine's existing data exchange, checkpoint
alignment, and back-pressure paths, which conflicts with Zeta's current
unidirectional DAG architecture. Moreover, this approach lacks reusability — if
future data flows need to introduce scenarios such as primary-key-based
routing, it would be nearly impossible to extend from this foundation.
- Cluster stability risk
The current hash routing lacks a back-pressure feedback loop, making it
easy for hot keys to overload a single writer. Combined with the synchronous
blocking RPC (the dispatch side's `.get()` blocks the task thread), local skew
can escalate into a global stall. On the receiving side, although
`SinkPartitionExchangeOperation` obtains the target task through
`TaskExecutionService`, the actual write executes synchronously on the HZ
operation thread — if downstream responses slow down, it directly occupies HZ
operation threads. More critically, this overhead scales as a product of the
number of jobs and parallelism. When the cluster runs many concurrent jobs
(e.g. 100), HZ operation threads can become heavily saturated. Since this
thread pool is shared across the entire cluster (heartbeats, checkpoints,
cluster management all depend on it), once exhausted, it may trigger heartbeat
timeouts, false node-down detection, and cascading failover.
- Failover data loss
-
Because the `Sink→Sink` point-to-point channel is outside the engine's
checkpoint scope, data that has been sent but not yet persisted may be lost
during failover, and the existing recovery mechanism cannot replay it. If
exactly-once or at-least-once recovery needs to be supported in the future, the
room for retrofitting would be extremely limited.
- A thought on direction
My personal inclination is not to add a point-to-point channel on the
Sink side, nor to introduce a standalone exchange operator at this stage. A
direction worth considering is to have `SourceSplitEnumerator` handle routing
and split strategies on the Source side, which would align better with the
engine's existing mechanisms.
Based on the above considerations, my suggestion is to produce a detailed
design first, covering at least the following:
- The complete `document_id` data flow (assignment, routing, delivery,
landing), including the addressing relationship between each parallelism
instance and vector database partitions
- Checkpoint alignment can be considered as a follow-up iteration (state
boundaries, snapshot timing, recovery consistency)
The above is solely my personal opinion and I welcome any corrections
@yzeng1618 @davidzollo @zhangshenghang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
