Jakob-al28 opened a new pull request, #2535: URL: https://github.com/apache/systemds/pull/2535
Follow-up to #2515, as discussed in #2528. Delta Kernel's default engine decodes parquet through parquet-mr's row-record API (ParquetReader + ReadSupport/RecordMaterializer), which is built to reassemble nested records; each cell goes through a PrimitiveConverter callback. Frames are flat in SystemDS, so this adds a thin Engine decorator that overrides ParquetHandler.readParquetFiles: flat schemas decode through the column API (ColumnReadStoreImpl/ColumnReader). Everything else (predicates, nested/decimal/timestamp columns, unknown metadata columns) falls through to the wrapped default engine, and deletion vectors, partition values and column mapping are still applied by the kernel above the handler. On TPC-H lineitem, 30M rows (median of 7 runs, cloud VM, n2-highmem-8, 40GB heap): 101.0s to 53.3s (1.89x). On the DeltaFrameRead benchmark from #2515: 1.68x serial, 1.50x parallel (k=8). | Format / Reader | Median (ms) | |-----------------|------------:| | Parquet: Legacy (Group) | 102,099 | | Parquet: Column API | 53,381 | | Delta: default engine | 101,028 | | Delta: column-API engine | 53,324 | | Binary Block | 21,576 | All runs at the same settings: ZSTD, no dictionary for Parquet and Delta; Binary Block uncompressed. <img width="1650" height="900" alt="matched_encoding_all" src="https://github.com/user-attachments/assets/a8b6d871-04fd-4bc6-b083-7ac09248fb12"; /> <img width="1800" height="750" alt="deltaframeread" src="https://github.com/user-attachments/assets/02442684-bf10-4779-9d24-cacddfdd6b55"; /> Two cases needed explicit handling: tables with deletion vectors enabled ask every read for a synthetic _metadata.row_index column that isn't actually in the file, and older files in a table that's had a column added later don't contain that column at all. Both would have failed otherwise; now the first is computed and the second comes back as null, covered by tests (DeltaFrameSparkContractTest). There's also a config kill switch (sysds.io.delta.reader.columnapi, default on). Notes: the ParquetHandler contract has grown across kernel versions (the row_index requirement came with deletion vector support), so delta-kernel version bumps should re-run the contract tests. The fast path materializes one row group at a time, a bigger per-batch memory footprint than the default engine's fixed-size batches. The decode relies on parquet-mr's column API (org.apache.parquet.column.impl); its signatures are unchanged from parquet 1.13 through 1.17. The write path stays on the default engine. Two checks suggest there is little to gain there: the parquet writer rewrite in #2528, which removed the per-row Group materialization, only moved write time from 92s to 87s, and with matched codecs the Delta write is within about 8% of that plain parquet writer (2.7s Delta vs 2.5s parquet on 1M lineitem rows, both set to snappy for this check). Neither points to a large win available on the write side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
