alamb commented on issue #8938:
URL: https://github.com/apache/arrow-rs/issues/8938#issuecomment-3711585459
> TBC whilst I do have concerns about the size of the change to the arrow
codebase, my larger concern with this initiative is with making large-scale
breaking changes to the public API. This has the potential to be extremely
disruptive to the ecosystem and will require a lot more buy-in from the various
stakeholders.
I don't think the features contemplated here **require** a large breaking
change
We could, for example, have a default global memory pool that the existing
stateless kernels use, as well as new kernels that take an explicit context:
```rust
// new kernel with context
fn new_binary(context: &Context, arr: &dyn Array, arr2: &dyn Array) ->
ArrayRef { ... }
// Old kernel / backwards compatible API
fn binary(arr: &dyn Array, arr2: &dyn Array) -> ArrayRef {
// calls through to new kernel with context using a global allocator
new_binary(GLOBAL_CONTEXT.get(), arr, arr2)
}
```
> That's not to say it isn't possible, but it would take multiple highly
motivated individuals to see it through.
Yes, this is my biggest concern. I think we would need to muster a
significant amount of effort to get this feature in -- in contributors and
reviewers. And leaving it half way done would be pretty bad.
That being said, something I have been thinking about for a long time is a
kernel API that allows buffer reuse (for example, producing the output in the
same allocation as the input, when possible). The current signature of taking a
`&` doesn't really allow for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
