DanielLeens commented on issue #10889: URL: https://github.com/apache/seatunnel/issues/10889#issuecomment-4465014884
Thanks for putting this together. This is a valuable direction, and it reads much more like a design/STIP-scale umbrella than a normal feature request. After checking the current codebase, I agree there are already several useful building blocks in place: the Markdown path already exposes `source_uri` / `document_id` / `chunk_id` / `chunk_index` / `content_hash`, the `Embedding` transform already exists, and Milvus already consumes `row.options[Partition]`. But those pieces still do not form a stable knowledge-sync contract end to end. The main boundaries I think we should keep explicit before implementation starts are: 1. semantic contract first: `DocumentId` / `DocumentHash` / `ChunkId` / `ChunkHash` should be settled before lifecycle sink behavior, because the current Markdown path still couples `chunk_id` to `content_hash`, so chunk identity is not stable across content refreshes; 2. engine contract separately: `SinkPartitionStrategy` and Zeta translation should be treated as their own API/engine slice, because they also define the non-Zeta failure contract; 3. Phase 1 should stay narrow: Zeta only, one concrete source path, one concrete parse/chunk path, and one lifecycle sink path first; 4. multi-engine wording should stay strict: if a non-empty sink partition strategy is only supported in Zeta first, Flink/Spark should fail explicitly rather than silently degrade. So from my side, the most useful next step is not to start from the whole PR train at once, but to make PR-A / Gate 0 the first reviewable artifact: - scope and non-goals - semantic model and hash contracts - Zeta-first / non-Zeta behavior - the first MVP chain and what is explicitly out of scope Once that foundation is written down, the follow-up PR slices will be much easier to review and much less likely to blur architecture boundaries. Happy to review the Gate 0 / ADR package first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
