JeremyXin commented on issue #12437: URL: https://github.com/apache/seatunnel/issues/12437#issuecomment-6209253094
@DanielLeens This is a valuable direction. I suggest delivering split correctness, recoverable reads, and media ingestion as separate slices. Suggested plan: - 1. Start with format-aware planning inside the existing File Connector split factory. Text/JSONL should split only at complete records; CSV needs a quote/escape-aware scanner using the same parsing rules as the reader; Parquet can retain RowGroup splits. Use whole-file splits or reject a requested split when a safe boundary cannot be established. - 2. Add an opt-in bounded reader path that passes the actual FileSourceSplit to the read strategy, emits a limited number of complete records per poll, and checkpoints the active split plus its next-record position. Keep split identity/range stable and preserve deserialization of old checkpoint state. - 3. Validate file/object identity before resuming. Where a reliable version identity is unavailable, the recoverable path should fail closed rather than combine bytes from different object versions. Parquet progress needs a format-specific position rather than a text byte offset. - 4. Keep media as a separate first slice: one whole file per split, streaming image-metadata inspection, and a small scalar output schema. Discuss new public configuration/provider APIs in a STIP before adding audio/video or Binary-payload adaptation. One existing call path needs attention: BaseFileSourceReader currently consumes a whole split per pollNext and calls the string-based read method, while the multiple-table reader snapshots only queued splits. Both paths should be considered when defining the new checkpoint behavior. I’d like to try taking ownership of this work, starting with the format-aware split planning and its correctness tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
