wangzhigang1999 opened a new pull request, #10798:
URL: https://github.com/apache/paimon/pull/10798
### Purpose
`DataWriter._get_column_stats` currently calls Arrow `min` and `max`
separately for the same column. Use one `min_max` reduction and extract both
values, avoiding repeated work without changing statistics fields, null
handling, unsupported-type guards, table options, or file formats.
ARM64 exploration with PyArrow 22.0.0 (4 cores, 3 GiB; 40k initial rows,
eight buckets, five overlapping 10k-row updates) showed lower statistics CPU
time: 7.10 to 5.89 ms for key-only statistics and 33.47 to 25.49 ms for
full-column statistics in separate diagnostic runs. Four uninstrumented runs
per variant/configuration did **not** establish a consistent end-to-end
speedup. This is a small reduction of redundant statistics work, not a
throughput claim.
### Tests
- Added 47 parameterized regression cases for batches and multi-chunk
tables: numeric/temporal/string/binary/boolean values, empty and all-null
columns, unsupported nested/large-binary columns, NaNs, infinities, and signed
zeros. The cases pass before and after the change.
- ARM64/Python 3.12/PyArrow 22: **167 passed, 18 skipped** (the skipped
lifecycle variants require the optional `pypaimon-rust` package):
```sh
python -m pytest -q pypaimon/tests/write/data_writer_stats_test.py \
pypaimon/tests/write/table_write_test.py \
pypaimon/tests/write/append_writer_lifecycle_test.py \
pypaimon/tests/test_write_merge_buffer.py
```
- Upstream Flake8 and license checks pass. The new test file also passes
formatting and type checking.
- The repeated-update experiment passed 240 read-back comparisons across 20
runs; diagnostic statistics digests matched between both implementations for
each configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
