felipecrv commented on PR #41700:
URL: https://github.com/apache/arrow/pull/41700#issuecomment-2299183276
> Note that there's something inherently suboptimal in this PR: we're
trading the concatenation of the chunked values (essentially allocating a new
values array) against the resolution of many chunked indices (essentially
allocating _two_ new indices arrays). This is only beneficial if the value
width is quite large (say a 256-byte FSB) _or_ the number of indices is much
smaller than the number of values.
I don't remember getting bad results like this on an ARM Macbook. I will try
benchmarking on x86 as well.
> _(the fact that our take benchmarks currently use the same number of
indices as values doesn't help :-))_
Indeed. My assumption when going for the allocation of indices was that
indices are cheaper to allocate than the concatenation of values.
> In the end, perhaps we want to guard this with a heuristic based on total
byte size of values and length of indices.
Yeah, but this kind of branching is really bad for the confidence in unit
tests. I will see what can be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
