DanielLeens commented on issue #11801: URL: https://github.com/apache/seatunnel/issues/11801#issuecomment-5315843084
Thanks for the PoC work in #11813. I re-checked the current `dev` file-source boundary before replying here. The current repo already has two stable contracts we should preserve: - `file_format_type = markdown` -> `MarkdownReadStrategy` - `file_format_type = pdf` -> `PdfReadStrategy` with optional RAG metadata So I agree with keeping anydoc as an explicit, opt-in integration rather than folding it into the existing PDF path or silently changing current behavior. For the first production slice, I would keep the runtime model conservative: 1. use a managed executable boundary first; 2. make it explicitly opt-in via config; 3. fail fast with a clear availability / version / platform check at startup; 4. define timeout, temp-file / cleanup, and concurrency limits explicitly; 5. document that scanned / image-only PDFs still require OCR and are out of scope for this backend. I would not start with WASM as the primary production path yet. The current question is not only whether it can run, but how we package, observe, and debug it on SeaTunnel workers across the supported environments. A managed executable is easier to reason about as a first slice, while still leaving room for a later WASM or sidecar evolution if the community wants it. On the design-record location: keeping the design note in the published developer docs is reasonable for now as long as it stays clearly marked as an open proposal and is updated or removed when the runtime model in #11801 is accepted, replaced, or rejected. The current PoC PR is already moving in that direction. So my recommendation is: - keep #11813 as the compatibility / design PoC; - keep existing PDF and Markdown behavior unchanged; - narrow v1 to an opt-in managed-executable backend that converts supported documents to Markdown and then reuses the current Markdown row / RAG path; - once that boundary is settled, split the production implementation into smaller reviewable steps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
