Jakob-al28 commented on PR #2535: URL: https://github.com/apache/systemds/pull/2535#issuecomment-4946530041
The inline comments are applied. For the shapes and sizes I added DeltaFrameShapeCoverageTest, which round-trips 27 shapes from 1x1 up to 1M rows and 1 to 1000 columns, including the 4096 writer batch boundaries, all-null string columns, NaN/infinity/min/max values and unicode strings. For the speed I benchmarked this branch against the kernel read path on main, same on-disk tables, same JVM flags, alternating runs, best of three each. On a 10Mx3 table with the default file layout (208MB, string/long/double with nulls) the direct decode is 1.7x faster serial and 1.5x parallel. Smaller and wider tables gain more, between 1.8x and 5.2x across the grid, no shape was slower. While validating this I found and fixed one gap: type widening can leave INT32 data files under a column that is now bigint. The kernel converts those on read, the typed decode cannot. decodeDataFileInto now checks each file's physical types up front and falls back to the kernel read for that file only; readTypeWidenedIntFilesAsLongColumn builds such a table and fails on the previous code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
