YannByron commented on PR #9466: URL: https://github.com/apache/paimon/pull/9466#issuecomment-5519671704
Local ACT benchmark after rebasing this PR onto `origin/master@2788fe596` (which includes the path-key BLOB index cache from #9547). The benchmark was recorded at functional commit `c1dbea125`; the only later change in the current PR head (`9e0340a82`) documents the measurement boundaries. ### Method - CPU-only local workstation run. - 5 independent HDF5/Paimon pairs, with execution order alternated between pairs. - Each result contains 3 rounds; the comparison aggregates 10 result files. - The same experiment, normalization, seed, model, snapshot, and logical window sequence were used for both backends. - The OS page cache was warmed and was not dropped. These numbers represent steady-state local reads, not cold-cache or distributed production performance. ### Machine and runtime | Item | Value | | --- | --- | | CPU | Apple M2 Pro, 10 logical CPUs | | Architecture / OS | arm64 / macOS 26.3 | | Python | 3.13.12 | | PyTorch | 2.10.0, 6 intra-op threads, 10 inter-op threads | | LeRobot | 0.4.4 | | PyArrow / h5py | 23.0.1 / 3.16.0 | | NumPy / Pillow | 2.2.6 / 12.1.1 | ### Experiment parameters | Parameter | Value | | --- | ---: | | Dataset | RoboMIND AgileX | | Action horizon / ACT chunk size | 32 | | Batch size | 2 | | Physical fetch depth | 8 batches | | Warm-up / timed batches | 1 / 32 | | Optimizer steps | 2 | | Repeats per result | 3 | | Image size | 64 x 80 | | Seed | 20260825 | | Learning rate / weight decay | 1e-4 / 1e-4 | | ACT model | ResNet-18, dim 64, 4 heads, latent dim 16, 1 encoder/decoder/VAE layer | | Parameters | 11,375,790 | | KL weight / pretrained backbone | 10.0 / none | | Paimon frames snapshot | 2 | Identity hashes: - experiment: `83a3b801d33ab5829911e2c5c241c72e63f1fab1d03c38c8b5b6289c9246d7ed` - source: `21c5567398d11f0e125d87e75d8670443cdcf0fe14a7a07bc978eb54a676de5b` - normalization: `9cea7907bea1fb85eaf4b9b6734b66e02d4b4fe32f1d701fbc41e771cdb7bb02` - window plan: `f15474db91ddfd8c60648ae278b90c8d8976ac8bcbc7d28dc25796f6c2a64532` - sample sequence: `2ee1b13d2a5d26b85bf985045ad63d3cb86f478944e110b5b45614926673df1d` ### Results | Metric (median) | HDF5 | Paimon | Comparison | | --- | ---: | ---: | ---: | | Batch fetch throughput | 175.57 samples/s | 170.73 samples/s | Paimon/HDF5 **97.24%** | | First batch | 12.42 ms | 19.52 ms | Paimon 1.57x HDF5 | | Fixed optimizer steps, including fetch | 202.65 ms | 213.76 ms | Paimon 1.05x HDF5 | | Measured wall time | 905.60 ms | 967.83 ms | Paimon 1.07x HDF5 | | Python peak allocation | 179.37 MiB | 181.48 MiB | Paimon 1.01x HDF5 | | Dataset construction | 0.003 ms | 31.81 ms | one-time construction cost | Correctness comparison: `SUCCEEDED`. - All 70 checked windows produced the same tensor fingerprint: `a90df976c27ab8d27ca9b60c0c1fb0f982abbbb74a8446c4542361501c8e1842`. - Both backends produced the same train losses: `[31.77707862854004, 31.60108757019043]`. - Both backends produced the same validation loss: `32.9738883972168`. For reference, a direct pre-/post-rebase Paimon A/B run on the same machine measured 168.64 -> 170.63 samples/s (+1.18%), so the rebase did not introduce a Paimon-side throughput regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
