jayshrivastava commented on issue #23436:
URL: https://github.com/apache/datafusion/issues/23436#issuecomment-6221631960
What if we added a custom partitioning scheme which stores an identifier
which helps check equality?
```
#[derive(Debug, Clone)]
pub enum Partitioning {
RoundRobinBatch(usize),
Hash(Vec<Arc<dyn PhysicalExpr>>, usize),
Range(RangePartitioning),
/// NEW: denote that we are partitioned on these columns using an
identifier to combine the partitioning scheme.
Custom(Vec<Arc<dyn PhysicalExpr>>, num_partitions: usize, identifier:
usize),
UnknownPartitioning(usize),
}
```
It would be nice to support regarding dynamic filtering in co-partitioned
joins, I have a proposal here that can help avoid CASE hash
https://github.com/apache/datafusion/issues/23817 so we don't necessarily need
to define the split points.
The benefit of the custom partitioning is to be able to do 1-step
aggregations on the partitioning key columns and to do co-partitioned joins
without a repartition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
