Jakob-al28 commented on code in PR #2528: URL: https://github.com/apache/systemds/pull/2528#discussion_r3523624843
########## src/test/resources/datasets/parquet/all.parquet: ########## Review Comment: Looked into the decode bottleneck in #2515. Delta Kernel's default engine decodes through parquet-mr's row-record API (ParquetReader + ReadSupport/RecordMaterializer), which is built to reassemble nested records, each cell goes through a PrimitiveConverter callback. Frames are flat in SystemDS, so it doesn't apply here. I built a thin Engine decorator on top of #2515 that overrides readParquetFiles: flat schemas decode through the same column API this PR uses (ColumnReadStoreImpl/ColumnReader) instead, everything else (predicates, nested/decimal/timestamp columns, unknown metadata columns) falls through to the wrapped default engine. On the same 30M row TPC-H lineitem (median of 7 runs, cloud VM): 103.9s to 55.2s (1.88x). On the DeltaFrameRead benchmark from #2515: 1.68x serial, 1.50x parallel (k=8). <img width="1050" height="750" alt="delta_tpch_30m" src="https://github.com/user-attachments/assets/b994a910-1254-403a-aedb-707e64ebfd2a"; /> <img width="1800" height="750" alt="deltaframeread" src="https://github.com/user-attachments/assets/88a37ebe-2eb9-4f54-9d2c-018126ac065e"; /> Two cases needed explicit handling: tables with deletion vectors enabled ask every read for a synthetic _metadata.row_index column that isn't actually in the file, and older files in a table that's had a column added later don't contain that column at all. Both would have failed otherwise; now the first is computed and the second comes back as null, covered by tests. There's also a config kill switch (sysds.io.delta.reader.columnapi, default on). If you're interested, I can open this as a new PR now that #2515 is in, happy to do that whenever works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
