Baunsgaard commented on PR #2511:
URL: https://github.com/apache/systemds/pull/2511#issuecomment-4799568110
### Native Delta read performance vs binary reader
Benchmark (`org.apache.sysds.performance` `DeltaIO`), 32 vCPU, JDK 17,
`-Xmx16g`,
20 timed repetitions after warmup, isolated (no competing load).
**Workload:** 5,000,000 × 16 dense FP64 matrix (~640 MB in memory, sparsity
1.0).
| Read path | Time (ms) | Disk throughput |
|----------------------|----------------|-----------------|
| Delta (serial) | 5411 ± 79 | 79 MB/s |
| **Delta (parallel)** | **875 ± 19** | **490 MB/s** |
| Binary (parallel) | ~100–170 | 3.7–6.0 GB/s |
**Takeaways**
- The parallel reader decodes data files concurrently and is **~6.2× faster
than the
serial Delta reader** (5411 ms → 875 ms).
- The binary reader is still **~5–9× faster** than parallel Delta. This is
expected:
binary is a near-raw `double[]` dump, while Delta Kernel 3.3.0 decodes
parquet
per-value (row-by-row converters, not vectorized). File-level parallelism
is the
main lever available, which is what this reader exploits.
- **On disk Delta is smaller:** 450 MB vs 645 MB for binary (0.70×), thanks
to parquet
encoding — so the read-time cost buys a ~30% smaller footprint.
For reference, the write path is encode-bound: Delta write ~7.0 s vs binary
~0.15 s
(parquet encoding dominates); not the focus of this PR but noted for
completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
