yzeng1618 commented on PR #10964: URL: https://github.com/apache/seatunnel/pull/10964#issuecomment-4598289911
I re-checked the current implementation against these concerns. The current SinkPartitionExchange path does introduce a sink-side point-to-point channel: remote delivery goes through Hazelcast task operations, the dispatch side waits on the operation future, and the receive side can synchronously enter the target SinkWriter path from the operation call stack. The exchange also has local buffering/alignment state that is not part of checkpoint state. So I agree this should not be treated as the production Knowledge Sync routing direction without a fuller design. I have drafted a design document first, focusing on the complete document_id flow: assignment, routing, delivery, landing, and the relationship between route buckets, parallel subtasks, and vector-store partitions/logical buckets. The proposed direction is to move the first production routing path toward SourceSplitEnumerator/source-side document assignment, and defer both the sink-side point-to-point channel and a standalone keyed exchange operator until checkpoint, back-pressure, and recovery semantics are designed as first-class engine behavior. Checkpoint alignment and recovery consistency are called out as a follow-up iteration with explicit state boundaries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
