rkrishn7 opened a new pull request, #27499:
URL: https://github.com/apache/datafusion/pull/27499
## Which issue does this PR close?
- Closes #27498.
## Rationale for this change
See #27498.
## What changes are included in this PR?
- Adds a required `GroupValues::find`, which writes `Some(group_id)` for
each stored row and `None` otherwise, and never inserts.
- Implements it for every built-in `GroupValues`, mostly by reusing each
implementation's `intern` probe path and skipping the insert:
- `GroupValuesPrimitive`, `GroupValuesBoolean`, `GroupValuesBytes`,
`GroupValuesBytesView`
- `GroupValuesRows`, which applies the same float normalization as
`intern` before converting rows
- `GroupValuesColumn`:
- Streaming mode compares values while probing, because colliding hashes
there are stored as separate map entries.
- Vectorized mode collects every candidate (row, group) pair, including
colliding groups, and checks them with one `vectorized_equal_to` pass per
column.
- The nested-float normalization `intern` already did is moved into a
helper that both methods call.
- `GroupValuesClustered` has no hash table, so `find` splits the input
into runs of equal rows and scans the buffered groups, newest first.
- Adds lookup-only `find` methods to `ArrowBytesMap` and
`ArrowBytesViewMap`, which back the single-column bytes implementations.
- `GroupValuesColumn::emit(EmitTo::All)` now clears its hash map and
collision lists, so `find` can't return ids of groups that were already emitted.
## What is the testing strategy for this PR?
Unit tests check that, for every `GroupValues` implementation, `find`
returns the id `intern` assigned to each stored row, returns `None` for missing
keys, and never inserts. They also cover nulls and lookups after `emit` and
`clear_shrink`. The tests also pass with the `force_hash_collisions` feature,
which sends every lookup through the collision handling.
## Are there any user-facing changes?
There are no changes to query behavior.
API changes (please add the `api change` label):
- **Breaking:** `GroupValues::find` is a new required trait method, so
custom `GroupValues` implementations must add it.
- New public methods: `ArrowBytesMap::find` and `ArrowBytesViewMap::find` in
`datafusion-physical-expr-common`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]