rambleraptor opened a new pull request, #51405:
URL: https://github.com/apache/arrow/pull/51405
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's
Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing
Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
Please remove this line and the above text before creating your pull request.
### Rationale for this change
When serializing a zero-length slice of a variable-length array that has a
non-empty offsets buffer, the Arrow IPC writer previously retained the entire
unsliced offsets buffer. For use cases which consumed ArrowRecordBatches in
sub-batches, this caused unnecessary buffer amplification and sub-batch
splitting.
The writer now returns either a single zero offset for zero-length slices
with non-empty offsets buffers or the null or empty value_offsets if the
value_offsets is null or empty. Tests in Arrow IPC and SuperluminalRowReader
are updated to verify this behavior.
We're hoping to understand if this was an intended behavior / not. This was
originally changed in https://github.com/apache/arrow/pull/46408
### What changes are included in this PR?
Change to behavior + unit tests
### Are these changes tested?
Unit tests.
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** (If there are any
breaking changes to public APIs, please explain which changes are breaking. If
not, you can remove this.)
There is a behavior change, but this doesn't appear to be a public API.
**This PR contains a "Critical Fix".** (If the changes fix either (a) a
security vulnerability, (b) a bug that caused incorrect or invalid data to be
produced, or (c) a bug that causes a crash (even when the API contract is
upheld), please provide explanation. If not, you can remove this.)
### Was AI used for this PR?
In accordance to the [AI generation
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
