Ichbinkiana commented on PR #2509: URL: https://github.com/apache/systemds/pull/2509#issuecomment-5110401806
## Summary of changes Latest commits moved the HDF5 benchmark away from JUNIT and also introduced two HDF5 I/O optimizations as below: - Added HDF5 benchmarks which by default disabled - My Benchmark records write/read time, file size, number of files, heap usage, GC stats, parallelism, and correctness status in both Json and CSV format - Added batched HDF5 writing for both dense and sparse inputs - Cleaned up HDF5 write to avoid unnecessary boxed Double and used Java primitive types instead. - Now the write and read are sparse-aware and this can be enabled for sparse matrices with below args: * Dsysds.hdf5.write.sparse.layout=coo * Dsysds.hdf5.read.sparse.layout=coo - Added a fallback so ReaderHDF5Parallel uses the sparse-aware sequential reader when Coordinate layout is enabled for sparse test cases. ------ ## Results ### For dense sequential writing: - Baseline: ~7.1 s - After batching: ~4.4 s - Improvement: ~37% faster ### For sparse matrix writing with 125000 x 1000 and 125000 nonzeros: - Baseline dense-layout file size: ~1.0 GB - optimized file size: ~3.0 MB - Baseline sparse write: several seconds - Optimized sparse write: ~23 ms - Optimized sparse read: ~18 ms ------ ## Interpretations/ Reasoning - Batching improved dense write performance by calling write less and in batches instead of row-by-row. - The large sparse improvement came from writing only nonzero entries instead of all entries. ## Why did i use synthetic Data - SystemDS HDF5 implementation works with matrices, which only support doubles. Real HDF5 files can contain non-double entries that are outside the current matrix reader/writer scope. I also used synthetic dense and sparse matrices to isolate and validate my optimized code behavior under controlled conditions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
