gene-bordegaray commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5483942555
> > All we need upstream is a property that can be propagated to AggregateExec to indicate this. > > Shouldn't this property already be inferrable from column statistics? If the `[min_key, max_key]` intervals are disjoint across input partitions, and we have `OrderingEquivalanceClass` with `[key ASC/DESC, date_bin(time) ASC/DESC]` then we should be able to stream the aggregation without additional machinery right? This will work. And datafusion checks for this right now and will stream. But this property streatches further then that use case. The main idea is concatting sorted runs can lose the global sort but still preserve group contiguousness (wow first time I have used that word and surprisinly it is a real word 😆 ): ```text run 1: AAA BBB run 2: CCC DDD concat (2 -> 1): CCC DDD AAA BBB ``` Every group is contiguous. Thusm it can be emitted incrementally but the combined stream cannot advertise sorted order. `Exact`, disjoint per-run stats could infer this property, but merged/inexact stats cannot. So stats should be one way to derive the property but we still need a property to represent and propagate the guarantees to agg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
