Jakob-al28 opened a new pull request, #2528: URL: https://github.com/apache/systemds/pull/2528
[SYSTEMDS-3929] Speed up Parquet frame reader/writer The old reader created a `Group` object per row, boxing every decoded value into a wrapper object inside it, even though SystemDS frames are always flat. The parallel reader also converted every boxed value to a string and re-parsed it back to its target type in `FrameBlock.set`. This PR reads columns via parquet's internal column API (`ColumnReadStoreImpl`/`ColumnReader`) instead, avoiding the `Group` allocation. On TPC-H lineitem (sf=5, ~30M rows, median of 7 runs on a cloud VM), read time went from 88.9s to 34.5s (2.6x faster). <img width="700" alt="tpch_read" src="https://github.com/user-attachments/assets/c0ca74ba-6b92-4df8-b9ec-93afe7d04106"; /> Also rewrites the writer to use a custom `WriteSupport` with tuned defaults, adds INT96 timestamp read support, and fixes row offsets for the parallel reader. The old writer had a tunable batch of 1000 rows. The batch size was benchmarked looking for a better default, but `ParquetWriter` already batches internally by buffered byte size, making the manual row-count batch on top of it redundant, so the new writer drops it and instead exposes that internal buffer size as the tunable row-group size. Compression codec and dictionary-encoding were benchmarked as well: `ZSTD` and per-column dictionary encoding (`STRING_ONLY`) are used as defaults. Row group size was benchmarked too and kept at parquet's default of 128MB. Note the batch-size comparison runs uncompressed to match the legacy writer's codec, while the row-group benchmark uses the ZSTD default, so absolute times differ between the two plots. <img width="700" alt="tpch_batch_sizes" src="https://github.com/user-attachments/assets/140643ec-9c5f-4a0f-9c60-57bef11deb77"; /> <img width="700" alt="tpch_row_group_sizes" src="https://github.com/user-attachments/assets/86ea6d12-2edd-482b-84da-bc2c22fcb6b0"; /> <img width="700" alt="tpch_compression" src="https://github.com/user-attachments/assets/c3f12c56-cb4a-4be4-98ef-fbc419603301"; /> <img width="700" alt="tpch_encoding" src="https://github.com/user-attachments/assets/6eae95b0-47e9-4d24-b16b-a2fb95dd0461"; /> TPC-H lineitem's comment column is ~84% unique, so dictionary encoding on it doesn't pay off, which is why `ALL_OFF` wins on this benchmark. Typical frame workloads are assumed to have low-cardinality string columns where dictionaries pay off, so `STRING_ONLY` is kept as the default. Also adds public Parquet test files under `src/test/resources/datasets/parquet/` (duckdb, apache/parquet-testing, Titanic from HuggingFace) as a check against real-world files, and tests covering INT96 decoding, null handling and parallel/sequential equivalence. Possible next step: row-group-level parallel reads within a single file. Limitations: INT96 decoding truncates nanosecond precision to milliseconds, and there's no logical-type mapping for dates/timestamps since FrameBlock has no such types. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
