DanielLeens opened a new issue, #11801: URL: https://github.com/apache/seatunnel/issues/11801
### Search before asking - [x] I searched existing issues and found related document/RAG work, but no issue that specifically discusses an `anydoc`-backed document-to-Markdown parser backend. Related issues: - #10889 - #9861 - #10608 - #9716 - #10916 - #10920 ### Description SeaTunnel already has an active Knowledge Sync / RAG direction, and the current `dev` branch has document-oriented file source support: - `FileFormat` includes `MARKDOWN` and `PDF` read strategies. - `PdfReadStrategy` extracts PDF document elements through PDFBox and can append RAG metadata fields. - `MarkdownReadStrategy` parses Markdown into document elements and can append RAG metadata fields. - `markdown_rag_metadata_enabled` and `pdf_rag_metadata_enabled` already expose the RAG metadata switch at file-source option level. This issue proposes adding an optional `anydoc`-backed parser backend for document-to-Markdown conversion, then reusing the existing Markdown/document element path where possible. `anydoc` is a Rust library from Firecrawl that converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and text-based PDF files to GitHub-Flavored Markdown. It currently publishes Node.js, Python, Rust, and WebAssembly bindings, and its repository is MIT licensed: https://github.com/firecrawl/anydoc Why this may fit SeaTunnel: 1. SeaTunnel's current PDF support is PDF-specific and PDFBox-based. It is useful, but it does not provide one shared document-to-Markdown path for Office documents, RTF, EPUB, CSV, and PDF. 2. SeaTunnel already has a Markdown reader that can turn Markdown into structured rows, so a document-to-Markdown parser backend can avoid duplicating structure extraction logic per document format. 3. Knowledge Sync needs stable document/chunk metadata and a predictable text representation before chunking, embedding, and vector sink lifecycle handling. Markdown is a reasonable intermediate contract for that layer. ### Suggested Scope Add an optional document parser backend, for example: ```hocon source { LocalFile { path = "/path/to/documents" file_format_type = "document" document_parser = "anydoc" document_output_format = "markdown" document_rag_metadata_enabled = true } } ``` Possible implementation shape: 1. Add a new `DOCUMENT` file format or a dedicated document parser option instead of overloading the existing `PDF` behavior. 2. Let the parser convert supported documents to Markdown first. 3. Reuse or share the Markdown document-element parsing path to emit rows with fields such as `element_id`, `element_type`, `heading_level`, `text`, `source_uri`, `document_id`, `chunk_id`, `chunk_index`, and `content_hash`. 4. Keep existing `file_format_type = pdf` behavior unchanged for compatibility. 5. Gate anydoc behind an explicit option so deployments do not take a native/external runtime dependency unless they opt in. ### Integration Concerns This should be treated as an integration proposal, not as a simple Maven dependency bump: - SeaTunnel is JVM/Maven based, while anydoc currently exposes Rust/Node/Python/WASM bindings rather than a normal Java artifact. - A PoC could use an external command adapter, but production support should define a clear runtime packaging story, failure semantics, concurrency model, and cross-platform support. - Scanned or image-only PDFs still require OCR. anydoc handles text-based PDFs locally but does not replace OCR. - Existing PDFBox-based `PdfReadStrategy` should remain available unless the community explicitly decides otherwise. ### Acceptance Criteria - A design is documented for where the parser backend belongs: file source format, transform, or shared document parser SPI. - Existing PDF and Markdown behavior remains backward compatible. - The supported format list is explicit, including the no-OCR limitation for scanned PDFs. - The emitted schema is aligned with current Markdown/PDF RAG metadata and the Knowledge Sync metadata contract. - Runtime dependency packaging is documented and tested for at least Linux and macOS, with clear behavior when the backend is unavailable. - Unit tests cover text-based PDF, DOCX or PPTX, Markdown handoff, unsupported/scanned PDF behavior, and error handling. - At least one e2e example shows document files being read, converted to Markdown/structured rows, chunked, and written to a downstream sink. ### Usage Scenario Enterprise RAG and knowledge indexing pipelines often receive mixed document sets in object storage or local/HDFS paths: - PDFs - DOC/DOCX - PPT/PPTX - XLS/XLSX - ODT/ODS/ODP - RTF - EPUB A SeaTunnel pipeline should be able to discover those documents, normalize them into Markdown or document elements, add stable document/chunk metadata, then continue through chunking, embedding, and vector sink lifecycle handling. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
