jhorstmann opened a new pull request, #7824:
URL: https://github.com/apache/arrow-rs/pull/7824
# Which issue does this PR close?
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax.
- Closes #7822. The benchmark update in #7823 should be merged first to get
a fair baseline.
# Rationale for this change
The changes in this PR improve parquet writing performance for primitives
without bloom filters by between 15% and 30%.
# What changes are included in this PR?
There was not a single bottleneck to fix, instead several small improvements
contributed to the performance increase:
- Optimize counting of values and nulls by replacing a loop with code that
can be vectorized by the compiler. The number of nulls can also be calculated
from the lengths of the array and the number of values to write, instead of
being counted separately.
- Change asserts in `BitWriter::put_value` to `debug_assert` since these
should never be triggered by users of the code and are not required for
soundness.
- Use slice iteration instead of indexing in flush_bit_packed_run to avoid
a bounds check.
- Separate iteration for def_levels and non_null_indices using specialized
iterators. Range iteration is `TrustedLen` and so avoids multiple capacity
checks and `BitIndexIterator` is more optimized for collecting non-null indices.
- Cache logical nulls of the array to avoid clones or repeated
recomputation. This should avoid a pathological case when writing lists of
arrays that need logical nulls.
# Are these changes tested?
Logic should be covered by existing tests.
# Are there any user-facing changes?
No, all changes are to implementation details and do not affect public apis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
