tustvold opened a new pull request, #4393:
URL: https://github.com/apache/arrow-rs/pull/4393
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
Closes #1047
Relates to #3999
Relates to #2837
Relates to #2766
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in
the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your
changes and offer better suggestions for fixes.
-->
This is a proposal on how to represent scalars within arrow-rs kernels, with
a view to providing a more consistent experience for users, and allowing us to
consolidate a non-trivial amount of dispatch logic currently found in
[downstreams](https://github.com/apache/arrow-datafusion/blob/d9e91d187c8af7f39999f8b1a11d53383826a471/datafusion/physical-expr/src/expressions/binary.rs).
There are a couple of aspects worth highlighting
## Arrays are Preeminent
Arrow is a specification for arrays and not scalars. As such this approach
makes a conscious decision to not define a parallel specification for
representing scalars. This has a number of advantages:
* Can use the existing stories for serialization, FFI, display, casting,
etc...
* Avoids ambiguity over how to represent scalars with non-native encodings,
e.g. dictionaries, run-end encoded, etc...
* Allows sharing more logic, by not bifurcating the types
* Supports all array types with no additional effort
It does come with some obvious downsides compared to a first-class scalar
approach
* Less efficient representation
* Potentially more confusing / less ergonomic to use
However, it is my assertion that preserving the arrow-ness of the
representation is more important than either of these
## Non-Owning
Related to the above, data is **only** stored in arrays. `Datum` and
`Scalar` solely act as wrappers to influence the dispatch within kernels, **you
cannot store a scalar value**.
This reflects both the desire to keep arrays as the canonical
representation, and also to discourage use of the scalar abstraction within
data structures. We want to encourage the use of arrays, not constructions like
`Vec<Scalar>` or `HashMap<Scalar, _>`. Permitting such usage not only creates
confusion, but performing type-erasure per field in this way is grossly
inefficient both from a memory and performance standpoint.
## Type-Erasure / Dynamic Dispatch
Both `Scalar` and `Datum` are abstractions relying on type-erasure. This
reflects a couple of goals:
* Designing generics for many operations, like arithmetic, is extremely hard
and represents a significant API commitment, type-erasure presents a much
smaller API surface that is therefore easier to evolve without breaking changes
* Avoiding generics helps to reduce code bloat from monomorphisation and
hopefully improve compile times
* Is consistent with the vast majority of kernels which use `&dyn Array`
The implied assumption is that the overheads of this dynamic dispatch will
be irrelevant when amortised over the number of values in an array. Or to
phrase it differently, **we are explicitly not optimising for performance of
purely scalar operations**. IMO such operations are outside the remit of a
vectorised execution engine.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is
sometimes worth providing a summary of the individual changes in this PR.
-->
Adds `Scalar` and `Datum` along with examples of their usage
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking
change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
