wangzhigang1999 commented on PR #10148: URL: https://github.com/apache/paimon/pull/10148#issuecomment-5806630815
Thanks @JingsongLi for reviewing and merging! Following up on the benchmark suggestion: I reran the final PR head `33b6dfe1751092c9ba1ca69a156affab8d3c75d7` against its parent `c23edd8dbdf27c13784f80d109a0785245d0596d`, so this comparison isolates the final change, including the cleanup. The run used the same four frozen datasets and full-MAP read cases on Linux with Python 3.11.13 / PyArrow 19.0.1, reading local append-only Parquet tables. Each side had one warmup and three measured rounds, with alternating old/new order and a shared timing lock. Times below are medians; negative changes mean less elapsed time. | Dataset | Default batch: before → after | Change | 32-row batch: before → after | Change | |---|---:|---:|---:|---:| | int64 MAP | 0.239358 → 0.200792 s | -16.11% | 0.447900 → 0.425891 s | -4.91% | | Moving-key string MAP | 0.189527 → 0.146412 s | -22.75% | 0.399424 → 0.364728 s | -8.69% | | Amazon | 0.992082 → 0.551238 s | -44.44% | 2.206791 → 1.830320 s | -17.06% | | Sparse int64 MAP, width 64 | 0.398998 → 0.171782 s | -56.95% | 0.849088 → 0.623576 s | -26.56% | All 4 smoke samples and 64 formal-stage samples (including warmups) passed the independent correctness oracle, and source/data/harness integrity checks passed. The focused full-restore and related shared-shredding tests also passed: **16 tests and 15 subtests**. The final head retains the measured gains. I would treat the int64 32-row result as approximately flat under the report’s ±5% display band, rather than claim a significant improvement. No measured regressions appeared in these cases. These results cover local Parquet reads; they do not add end-to-end ORC or OSS coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
