snizovtsev opened a new issue, #34538:
URL: https://github.com/apache/arrow/issues/34538
### Describe the enhancement requested
Hello,
On my project I faced the need to emit a pair of columns from my compute
kernel. Since kernel's signatures allows only 1 output argument I wrapped
multiple columns under Struct type. Then I realize that
`MemAllocation::PREALLOCATE` doesn't allocate child buffers where struct's
fields are stored:
```cpp
Result<std::shared_ptr<ArrayData>>
KernelExecutorImpl::PrepareOutput(int64_t length) {
auto out = std::make_shared<ArrayData>(output_type_.GetSharedPtr(),
length);
out->buffers.resize(output_num_buffers_);
...
for (size_t i = 0; i < data_preallocated_.size(); ++i) {
const auto& prealloc = data_preallocated_[i];
if (prealloc.bit_width >= 0) {
ARROW_ASSIGN_OR_RAISE(
out->buffers[i + 1],
AllocateDataBuffer(kernel_ctx_, length + prealloc.added_length,
prealloc.bit_width));
}
}
// out->child_data untouched
return out;
}
```
and then
```cpp
arrow::Status UdfExec(cp::KernelContext* ctx, const cp::ExecSpan& batch,
cp::ExecResult* out)
{
CHECK_EQ(out->array_data()->buffers.size(), 1); // validity bitmap
allocated
CHECK_EQ(out->array_data()->child_data.size(), 0); // no data for struct
fields allocated
return arrow::Status::OK();
}
```
As workaround have to set `kernel.mem_allocation =
cp::MemAllocation::NO_PREALLOCATE;` and use regular builders in my kernel:
```cpp
arrow::Status UdfExec(cp::KernelContext* ctx, const cp::ExecSpan& batch,
cp::ExecResult* out)
{
std::unique_ptr<arrow::ArrayBuilder> builder;
RETURN_NOT_OK(arrow::MakeBuilder(ctx->memory_pool(),
out->array_data()->type,
&builder));
auto struct_ = dynamic_cast<arrow::StructBuilder*>(builder.get());
auto x = dynamic_cast<arrow::UInt32Builder*>(builder->child(0));
auto y = dynamic_cast<arrow::UInt64Builder*>(builder->child(1));
RETURN_NOT_OK(builder->Reserve(batch.length));
RETURN_NOT_OK(x->Reserve(batch.length));
RETURN_NOT_OK(y->Reserve(batch.length));
const uint64_t* pn = batch[0].array.GetValues<uint64_t>(1);
for (int64_t i = 0; i < batch.length; ++i) {
RETURN_NOT_OK(x->Append(pn[i] % 10));
RETURN_NOT_OK(y->Append(pn[i] ^ 1231231231231));
RETURN_NOT_OK(struct_->Append());
}
RETURN_NOT_OK(builder->FinishInternal(&std::get<std::shared_ptr<arrow::ArrayData>>(out->value)));
return arrow::Status::OK();
}
...
void RegisterUdf() {
auto out_type = arrow::struct_({
arrow::field("x", arrow::uint32()),
arrow::field("y", arrow::uint64()),
});
cp::ScalarKernel kernel({arrow::uint64()}, out_type, UdfExec);
...
}
```
I think it would be useful to support such case in kernel's memory
preallocation. Or my workaround would help someone else.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]