ZhangHuiGui commented on issue #41923:
URL: https://github.com/apache/arrow/issues/41923#issuecomment-2143796671
> However, some code implicit shows that it might contains chunked array,
see:
In the expression system, ChunkedArray can be processed normally as input,
and for some kernel functions, chunked array will be returned as output.
If the input type is chunked array, chunked array is returned in
`ExecuteScalarExpression` as follows:
1. `NullHandling::COMPUTED_NO_PREALLOCATE` is registered when `AddKernel` is
added
2. The output type of kernel function is `nested-type` or `dictionary-type`
3. The first two cases are not satisfied, but `can_write_into_slices =
false` is set.
`ExecuteScalarExpression`'s processing logic for chunked array is as follows:
1. ExecBatch mainly saves `vector<Datum>`, and Datum can be chunked-array
type.
2. Chunked array is processed in the kernel of ExecuteScalarExpression. The
processing logic is mainly in [1]. There are three processing scenarios:
1). If `preallocating_all_buffers_ = false`, that is, we do not enable
preallocation (`NullHandling::COMPUTED_NO_PREALLOCATE` or nested-types or
dictionary-type), the chunks in chunked-array will be processed one by one and
returned as chunked-array. [2]
2). If `preallocate_contiguous_ = true`, it means that the output type of
this kernel function supports preallocation (non-nested types), the chunks in
chunked-array will be processed one by one through span_iterator_, and an array
will be concatenated and returned. [3]
3). If `preallocate_contiguous_ = false`, the processing logic is the same
as `1)`, and chunked-array will be returned. [4]
[1]
https://github.com/apache/arrow/blob/0d9d699250ad98cdcc5c99735e1379b1e79bc195/cpp/src/arrow/compute/exec.cc#L424C1-L439C6
[2]
https://github.com/apache/arrow/blob/0d9d699250ad98cdcc5c99735e1379b1e79bc195/cpp/src/arrow/compute/exec.cc#L907C1-L927C6
[3]
https://github.com/apache/arrow/blob/0d9d699250ad98cdcc5c99735e1379b1e79bc195/cpp/src/arrow/compute/exec.cc#L856C1-L861C8
[4]
https://github.com/apache/arrow/blob/0d9d699250ad98cdcc5c99735e1379b1e79bc195/cpp/src/arrow/compute/exec.cc#L869C1-L874C9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]