Jakob-al28 commented on PR #2528: URL: https://github.com/apache/systemds/pull/2528#issuecomment-4971306512
Thanks, addressed all comments: The reader decodes directly into pre-allocated column backing arrays, with the same structure and naming as the Delta column decode, the row offset is moved out of the decode loops. Columns whose parquet physical type does not match the requested frame type fall back to a per-cell conversion with the previous FrameBlock.set semantics (e.g. numeric files read with a string schema). INT96 support is removed. Requesting an INT96 column fails with a error while the remaining columns of the same file stay readable via projection. Legacy files are removed. Tests now compare against Spark as the reference implementation in both directions: Spark-written inputs are compared against Spark's own read, and SystemDS-written single- and multi-part outputs are read back with Spark. The sequential reader also expands part-file directories now, and both readers validate the file row counts against the expected dimensions. Benchmarks are moved to org.apache.sysds.performance.frame, the batch-size comparison was dropped together with the legacy writer it measured. Tests use TestUtils.compareFrames/generateRandomFrameBlock and redundant tests were removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
