ZhangHuiGui commented on code in PR #41975:
URL: https://github.com/apache/arrow/pull/41975#discussion_r1627073697
##########
cpp/src/arrow/compute/exec.cc:
##########
@@ -1022,25 +1048,7 @@ Status CheckCanExecuteChunked(const VectorKernel*
kernel) {
class VectorExecutor : public KernelExecutorImpl<VectorKernel> {
public:
Status Execute(const ExecBatch& batch, ExecListener* listener) override {
- // Some vector kernels have a separate code path for handling
- // chunked arrays (VectorKernel::exec_chunked) so we check if we
- // have any chunked arrays. If we do and an exec_chunked function
- // is defined then we call that.
- bool have_chunked_arrays = false;
- for (const Datum& arg : batch.values) {
- if (arg.is_chunked_array()) have_chunked_arrays = true;
- }
-
- output_num_buffers_ =
static_cast<int>(output_type_.type->layout().buffers.size());
-
- // Decide if we need to preallocate memory for this kernel
- validity_preallocated_ =
Review Comment:
We could use more precise pre-allocation strategy by check the input batches
as what `ScalarExecutor` do.
We could prevent some unnecessary pre-allocation by this precise strategy
(checked the input batches status). And this could make an improvement for the
situations like below:
before:
```shell
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/1000 4395737 ns
4393473 ns 138 items_per_second=119.333M/s null_percent=0.1
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/10 5109157 ns
5106714 ns 129 items_per_second=102.666M/s null_percent=10
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/2 6586773 ns
6583881 ns 104 items_per_second=79.6321M/s null_percent=50
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/1 1868864 ns
1867719 ns 364 items_per_second=280.71M/s null_percent=100
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/0 3133277 ns
3131282 ns 248 items_per_second=167.436M/s null_percent=0
size=524.288k
```
after:
```shell
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/1000 1813275 ns
1812657 ns 392 items_per_second=289.237M/s null_percent=0.1
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/10 2711716 ns
2710747 ns 255 items_per_second=193.411M/s null_percent=10
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/2 4460442 ns
4459029 ns 154 items_per_second=117.579M/s null_percent=50
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/1 248396 ns
248270 ns 2695 items_per_second=2.11176G/s null_percent=100
size=524.288k
TakeChunkedFlatInt64RandomIndicesWithNulls/524288/0 878407 ns
878028 ns 769 items_per_second=597.12M/s null_percent=0 size=524.288k
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
