DanielLeens commented on issue #10129: URL: https://github.com/apache/seatunnel/issues/10129#issuecomment-6595081253
Hi @taowenjun, thanks for offering to continue this. A follow-up PR for ORC is welcome. Here is where things stand on current `dev`. **What is already merged.** File split is driven by `FileSplitStrategyFactory.initFileSplitStrategy` in connector-file-base. `FileFormat.supportFileSplit()` returns true only for `CSV`, `TEXT`, `JSON`, and `PARQUET`. Parquet uses `ParquetFileSplitStrategy`, which splits on row-group boundaries (#10239); the text formats use `AccordingToSplitSizeSplitStrategy` (#10142). The factory is wired into `LocalFileSource`, `HdfsFileSource`, and `S3FileSource` (#10450). ORC and XML currently fall back to whole-file reading with a warning, and compressed text input does too. The status table in this issue is out of date: S3 is done. **Why #10325 was closed.** It was closed automatically as stale on 2026-09-02, after a changes-requested review on 2026-08-25. The stripe-based approach itself was accepted. The two blockers were: 1. It conflicted with the current split architecture. It still targeted the removed `LocalFileSplitStrategyFactory`. 2. The split read path in `OrcReadStrategy` replaced `reader.options().schema(schema)` with a fresh `Reader.Options().range(...)`, which drops the projected schema. **What a new PR should cover:** - An `OrcFileSplitStrategy` in connector-file-base, selected through `FileSplitStrategyFactory`, with `ORC` added to `supportFileSplit()`. That way LocalFile, HdfsFile, and S3File all get it from one place. Like `ParquetFileSplitStrategy`, it should read stripe metadata through `HadoopFileSystemProxy` with the connector's `HadoopConf`, so Kerberos and remote filesystems keep working. Do not use a plain `new Configuration()`. - Splits made of whole stripes, with boundaries aligned to stripe offsets. ORC's `range()` selects stripes by start offset, so unaligned ranges could skip or duplicate a stripe. - One `Reader.Options` in `OrcReadStrategy` that always carries the projected schema, adding `range(start, length)` only for real splits. - No changes to `FileSourceSplit` serialization or its legacy `start`/`length` handling, so existing checkpoints still restore. - Tests: a unit test of the stripe split calculation, and a comparison of split and non-split output with `read_columns`. Add an ORC case to the existing LocalFile split E2E with parallelism greater than 1 and exact row-count assertions, next to the current `*_enable_split_to_assert.conf` cases. Update the `enable_file_split` docs in both `docs/en` and `docs/zh`. Since #10325 is the starting point, please credit the original author in the PR, for example with a `Co-authored-by` trailer. I will suggest adding you as an assignee for the ORC part. In the meantime, please link your PR here when it is ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
