neilconway commented on PR #25607:
URL: https://github.com/apache/datafusion/pull/25607#issuecomment-5816220097
@comphead Thanks! I made the following changes:
- Reuse the string copier for Binary and LargeBinary. Generalized it to
GenericByteArray. The Binary benchmarks improved by 33–46%.
- Switch to one-pass copier for utf-8 and binary types. Per discussion
above, this wins on some workloads and loses on others.
- Release unused primitive output capacity. Added shrink_to_fit() after
copying. The mostly-NULL-row example now retains 2.25 KiB instead of 192 KiB.
This reduces retained capacity; the initial reservation is unchanged.
- Remove both inlining annotations. The final combined implementation
supported removing both after benchmarking.
- Simplify bitmap padding. Replaced manual length checks and padding
calculations with BooleanBufferBuilder::resize, while retaining batching across
consecutive NULL rows.
- Clarify names and comments. Renamed the construction wrapper to
build_compacted_list, clarified element counts versus byte lengths, documented
visible-span indexing, and explained why the unchecked string constructor
preserves UTF-8 validity.
- Improve and modestly trim tests. Added adjacent retained strings, spans
crossing row boundaries, empty strings, Unicode, sliced bitmaps, hidden NULL
payloads, arbitrary binary bytes, masked invalid dictionary keys, and
retained-capacity checks. Removed redundant numeric/all-NULL cases while
preserving representative types and both
offset widths.
I also revised the benchmarks in #25566. There's more that we could do here
but I think the current state of the PR is definitely a major improvement,
additional improvements can be taken on as followups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
