goutamadwant opened a new issue, #12977: URL: https://github.com/apache/seatunnel/issues/12977
### Search before asking - [x] I searched open and closed issues and PRs and found no same-purpose physical Parquet column-pruning implementation. Searches checked on 24 October 2026: `Parquet projection`, `Parquet pruning`, `Parquet read_columns`, and `Parquet column`. The adjacent timestamp-precision PR #12956 is separate from this proposal and may overlap the reader file. ### Description The shared file-base Parquet reader supports selecting output fields through `read_columns`, but the baseline reader still reads the unselected physical column chunks. A job returning a few small fields from a wide Parquet file can therefore read large payload columns it never emits. This was reproduced through the production `ParquetReadStrategy` for both the Avro reader and its native Group fallback, not through a standalone Parquet-library projection example. Baseline: `9ab97e15f33cd1383c5216e50e77a2fe53d212c3`. Locally validated candidate: `eb56896d57e4b2cfc9144b3e5d0d74312cdfda53`. [Review branch and six-file diff](https://github.com/apache/seatunnel/compare/dev...goutamadwant:improve/parquet-column-pruning). No upstream PR has been opened. This issue records the workload, evidence and proposed scope for community review; it does not assume approval of the optimization. ### Usage Scenario Read a narrow subset of top-level fields from wide Parquet files, for example identifiers and timestamps while excluding large binary/text payloads. The implementation belongs in the shared file-base reader rather than adding connector-specific behavior. ### Before Example Parquet schema: ```text id INT32, required unused BINARY, required ``` Example source configuration: ```hocon source { LocalFile { path = "data/wide.parquet" file_format_type = "parquet" read_columns = [id] parse_partition_from_path = false } } ``` With 128 rows containing IDs 0–127 and a 32 KiB unused binary payload per row, the output already contains only the IDs. However, the reader still reads approximately 4.2 MB in the measured data-read phase: ```text Output: [0], [1], ... [127] Avro reader bytes read: 4,195,735 Native Group reader bytes read: 4,196,080 ``` The native-reader fixture uses the field name `id-value`, with `read_columns = ["id-value"]`, to exercise the existing illegal-Avro-name fallback. ### After: proposed behavior Use the same configuration and return the same 128 rows, selected values and ordering. Request only selected top-level physical fields from the underlying Parquet reader: ```text Output: [0], [1], ... [127] Avro reader bytes read: 863 Native Group reader bytes read: 1,208 ``` No new option is needed. Selected nested fields remain complete fields; nested-field pruning is not included. The candidate preserves the original read path when an explicit Parquet/Avro read schema, Avro projection, filter predicate or legacy record filter is configured. Filters can depend on columns absent from the output, so automatic pruning must not remove those columns. Empty/all-missing selections and all-column reads retain the original requested schema. Output reordering, missing-field handling, partitions, table IDs, split boundaries and schema metadata remain covered by compatibility tests. ### Reproduction and measurements The deterministic regression fixture writes 128 rows using a fixed random seed, uncompressed Parquet and dictionary encoding disabled. Each row contains an integer ID plus 32 KiB of random binary data. Both readers are exercised. The byte counter wraps the filesystem used by the production reader. It is reset after row-type discovery and before the data-read call. These figures therefore describe that read phase, not total job I/O including earlier schema discovery. The same final fixture, test code, reader dependencies, JDK and measurement boundary were used for five baseline/candidate repetitions. For baseline measurements, only the production reader implementation was replaced with the unchanged baseline version. Every repetition produced: - Avro projected read: **4,195,735 → 863 bytes**. - Native projected read: **4,196,080 → 1,208 bytes**, including the existing Avro fallback attempt. - Avro all-column control: **4,195,735 bytes** on both revisions. - Native all-column control: **4,196,080 bytes** on both revisions. - All 128 IDs matched in order; checksum **8128**. Full-column controls also compared every payload byte with the generated input. Baseline file sizes were 4,195,796 bytes for the Avro fixture and 4,195,808 bytes for the native fixture. Cumulative read bytes can exceed file size because of metadata/repeated reads. The regression first passed its row/value assertions on the baseline and then failed the physical-I/O assertion for both readers. The candidate passes both. These are repeatable synthetic filesystem-I/O measurements, **not wall-clock throughput benchmarks, production speedups, memory measurements or JMH results**. Remote-storage latency, realistic compressed workloads and workload-level throughput remain outside this evidence. ### Tests and validation Java 11 was used. The broad file-base run used an x86-compatible runtime because the existing Snappy dependency lacks the required ARM native library. Completed locally: - **370 file-base tests passed**, including **30 Parquet tests**. - Focused coverage includes both readers, full-read controls, reordered/missing/all-missing fields, nested/logical/null values, per-file schemas, row-group splits, partition values, table IDs, explicit schemas, filters, Avro metadata and configuration isolation. - A predicate-on-an-unselected-column regression caught an initial candidate returning zero rows instead of one. The final explicit-filter fallback resolves that regression in both readers. - **Four LocalFile projection E2E invocations passed**, zero failures/errors/skips: Flink 1.18, Flink 1.20, Zeta and Spark 3.3. - E2E used the existing five-row Snappy fixture, selected fields in a different order, exact row-count bounds and non-null field assertions. - Local engine validation used architecture-matched amd64 Java 11 images via image-selection-only test overrides. No reader, fixture, assertion, discovery or exception-handling implementation was replaced. - Scoped packaging, formatting and diff checks passed. Independent correctness and security reviews found no remaining actionable findings. Relevant candidate tests: - [Parquet reader regression and compatibility tests](https://github.com/goutamadwant/seatunnel/blob/eb56896d57e4b2cfc9144b3e5d0d74312cdfda53/seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/ParquetReadStrategyTest.java#L105-L500). - [LocalFile projection E2E configuration](https://github.com/goutamadwant/seatunnel/blob/eb56896d57e4b2cfc9144b3e5d0d74312cdfda53/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/parquet/local_file_parquet_projection_to_assert.conf). From the candidate checkout, with Java 11: ```sh mvn -pl seatunnel-connectors-v2/connector-file/connector-file-base -am -DskipTests install mvn -pl seatunnel-connectors-v2/connector-file/connector-file-base \ '-Dtest=ParquetReadStrategyTest#testReadColumnsPrunesPhysicalColumns' test mvn -pl seatunnel-connectors-v2/connector-file/connector-file-base verify ``` With Docker and architecture-compatible engine/Java 11 images: ```sh TEST_IN_PR=true RUN_ALL_CONTAINER=false RUN_ZETA_CONTAINER=true RUN_ENGINES=all \ mvn -pl :connector-file-local-e2e -am -DskipUT=true -DskipIT=false \ '-Dit.test=LocalFileIT#testLocalFileParquetProjection' \ -Dit.failIfNoSpecifiedTests=false -De2e.dependency.skip=false -Dskip.ui=true verify ``` The commands above identify the checks; an ARM host may require compatible engine images as described above. Full repository/hosted CI has not been established as passing. Java 8 was not tested; the tested development branch targets Java 11. ### Advantages - Avoid reading unused wide columns for narrow Parquet projections. - Improve the shared reader rather than duplicating logic in individual connectors. - Reuse existing configuration and decoding libraries. - Retain conservative behavior for explicit schemas and filters. ### Breaking changes and limits No intentional breaking changes: no removed/renamed options, default changes, new dependencies, public API changes, output/state-format changes or required migration. Rows, values and ordering must remain identical for supported inputs. Selecting a nested field does not prune its children. This proposal adds no predicate pushdown, schema-evolution feature or distributed-engine lifecycle change. Local correctness tests do not prove compatibility with every third-party Parquet producer or establish a throughput benefit. This issue is for scope/workload review under the project's performance-contribution guide; benchmark-first publication and broader CI remain separate gates before an optimization PR is submitted. ### Related issues No dedicated same-purpose issue was found in the searches above. #12956 is adjacent timestamp work, not the same change. ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
