DanielLeens opened a new issue, #12437: URL: https://github.com/apache/seatunnel/issues/12437
## Purpose This is a design discussion, not an implementation proposal. We would like feedback on a cohesive File Connector direction that makes parallel reads correct for structured files and creates a production-safe ingestion contract for images, audio, and video. Alternative designs and a formal STIP are very welcome. ## Current observations The File Connector already has useful foundations: - `FileSourceSplit` carries a byte range and has `fileIdentity` and `endContentAnchor` fields: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSourceSplit.java - `enable_file_split` currently plans text-like ranges by finding the next raw row delimiter, while Parquet uses a separate strategy: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/AccordingToSplitSizeSplitStrategy.java - `JsonReadStrategy` deserializes one physical line at a time: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/JsonReadStrategy.java - `BinaryReadStrategy` emits the stable three-field binary contract `[data, relativePath, partIndex]`: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java - The bounded file reader currently consumes a whole split in one `pollNext` call and snapshots queued splits, rather than a record-level cursor: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/BaseFileSourceReader.java This leaves several design questions. A raw newline is not necessarily a CSV record boundary when quoted fields may contain newlines. JSON Lines and JSON arrays have different split semantics. Large media assets should not be arbitrarily byte-split or assembled as full byte arrays in the JVM heap. ## Design goals 1. Preserve existing jobs and old checkpoints. `enable_file_split=false` remains the default, and the existing binary three-field schema must not change. 2. Never silently produce incomplete, duplicated, or malformed records because a split boundary is unsafe. 3. Support bounded reads with progress checkpoints at complete-record boundaries. 4. Keep connector responsibilities clear: file/object discovery and file reading belong to the File Connector; media decoding belongs to an optional media provider. 5. Avoid moving large image/audio/video payloads through ordinary SeaTunnel rows unless the input is intrinsically a binary message/BLOB stream. ## Proposed direction for discussion ### 1. Format-aware split planning Introduce a common record-boundary planning SPI rather than treating every supported format as line-delimited. | Format | Proposed split policy | |---|---| | Text | Split on the configured complete row delimiter. | | CSV | Use a quote/escape-aware scanner and split only after a complete RFC-style CSV record. Header resolution must happen once before readers run. | | JSONL | Split on complete physical lines only. | | JSON array / one JSON document | Do not permit byte-range splitting. Use a single-file streaming parser or reject incompatible configuration. | | Parquet | Retain RowGroup splitting. | | Compressed/archive/binary/PDF/Excel/XML/media | Do not byte-split; distribute whole files. | For CSV, a quote-aware scanner may need to stream the file once to create boundaries or maintain a reusable boundary index keyed by file identity. We should prefer correctness over an unsafe fast path. ### 2. Recoverable FileReadCursor For split-enabled bounded reads, introduce a cursor state such as: ```text file identity + planned range + last committed record end offset + resolved header/schema + end-content anchor ``` A reader should emit a bounded amount of data per poll and checkpoint only after a complete record. On restore, it reopens from the last committed record boundary and validates that the file/object identity and content anchor did not change. A changed object must fail or be re-enumerated by an explicit policy; it must never combine bytes from different versions. This should be introduced as a new opt-in reader path for split-enabled bounded Text/CSV/JSONL/Parquet reads, keeping the existing full-split and continuous-tailing behavior untouched initially. ### 3. Media-aware File Source path For LocalFile/HdfsFile/S3File/OssFile, consider a new `media` file format/reader (exact configuration is intentionally open for design). It would process one complete file per split and emit one ordinary `MediaAsset` row rather than binary chunks: ```text source_uri, relative_path, file_id/version, byte_size, modified_time, content_hash, detected_mime_type, media_kind, format, width, height, duration_ms, frame_rate, sample_rate, channels, codec, parse_status, parse_error_code, provider_properties ``` The File Connector would provide stable file/object identity and streaming input. An optional media provider would detect and inspect the content. The first provider could be image metadata only; native audio/video decoding would be a separately installed provider with strict resource limits and explicit failure when unavailable. Media files remain whole-file units. Their parallelism comes from independent files, not arbitrary byte ranges inside a video or audio asset. ### 4. Keep generic binary compatibility The existing binary source contract remains exactly `[data, relativePath, partIndex]` for binary-to-binary synchronization. A later `MediaPayload` Transform can adapt binary data from Kafka, HTTP, or database BLOBs: validate chunk order, spool under resource limits, compute identity/hash, call the same media provider, and emit the same `MediaAsset` row. It should not require changing Binary Source or Binary Sink schemas. ## Explicit non-goals - This discussion does not propose a Ray-like execution engine or a new distributed runtime. - It does not make a Tensor/media logical type a prerequisite for v1. Metadata should initially use existing scalar and map types. - It does not merge object-storage CDC, event notifications, delete/changelog semantics, or on-demand fetch into this work. - It does not claim a generic byte-range split is safe for all file formats. ## Related work - #10326 added large-file parallel splitting foundations and is the relevant predecessor. - #10608 discusses a PDF parser and multimodal labeling operator; this issue proposes the lower-level file/media ingestion contract those features could build upon. - #11758 discusses object-storage CDC and lazy content fetch. Its event/changelog semantics should stay separate from this bounded-file correctness work. ## Questions for maintainers and contributors 1. Is a shared format-aware split-planning SPI the right boundary, or should each read strategy own its complete planning logic? 2. Should `FileReadCursor` be represented in `FileSourceSplit` or as separate reader state to keep split identity immutable? 3. What is the preferred checkpoint and file-change policy for object stores without a reliable ETag/version? 4. Should media inspection be a File Connector format, a reusable Source SPI/provider, or primarily a Transform with a specialized File adapter? 5. Which minimal image-only output schema would be useful enough for a first PR, without prematurely creating a public media type system? 6. Would maintainers prefer an STIP before new public configuration and provider/SPIs are introduced? A contributor who would like to drive a better design or submit an STIP is very welcome. The goal here is to agree on a small, backward-compatible first slice before any broad implementation starts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
