yzeng1618 commented on PR #10964: URL: https://github.com/apache/seatunnel/pull/10964#issuecomment-4737657636
> It seems that our original design intended to have a unified API on the engine side for hash-based routing. In our first version, we abandoned the sink → sink approach because it had compatibility issues with the existing engine and performance concerns. However, looking at the second version's design approach, it directly adds new configuration to the source connector-file — if enabled, it performs document hash-based routing. So if in the future, beyond document synchronization, other use cases want to reuse this hash routing mechanism, this approach appears to be a dead end. > > Here are my questions: > > 1. Does this align with the scope/boundaries we defined for the Phase 1 MVP? > 2. If the routing strategy needs to be extended to other connectors in the future, how should we handle it? I see the concern. My intention for this PR is not to define the final generic hash-routing API for SeaTunnel. This PR is only the first concrete Phase 1 implementation slice for Knowledge Sync: bounded Markdown file sources derive a stable document_id and let the source enumerator assign whole-document splits to deterministic route buckets. This matches the current Phase 1 boundary in #10889 / #10914: source-side document routing first, Markdown file source first, and no sink-to-sink exchange or standalone engine keyed exchange in this PR. The implementation is intentionally gated by Markdown + markdown_rag_metadata_enabled, so existing non-Markdown sources and Markdown sources without RAG metadata keep the current round-robin behavior. For future connectors, I think we should not copy connector-file-specific code blindly. The extension path should be a follow-up common source-side document routing contract, for example: 1. a stable route key/document_id contract from the metadata work, 2. a source/enumerator capability for whole-document split assignment, 3. planner/lifecycle-sink validation that the source supports document-level routing and route alignment, 4. a later engine-level keyed exchange only if we need generic routing beyond source-side document assignment. So I would treat this PR as the Markdown/file implementation of the Phase 1 source-side routing direction, not as the final reusable API. I can also make that scope explicit in the PR description/design issue if that helps avoid implying this is the long-term generic routing mechanism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
