alamb commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5528238771
I was speaking with people at VLDB this week, and we were discussing how to represent this notion with our existing infrastructure. Goetz Graefe suggested a formalized version of what I think @xudong963 and @gene-bordegaray describe as 'group-contiguous' in https://github.com/apache/datafusion/issues/24438#issuecomment-5473519696 and https://github.com/apache/datafusion/issues/24438#issuecomment-5483942555. The idea is to model this as an extension of the existing [`SortProperties`](https://docs.rs/datafusion/latest/datafusion/logical_expr/sort_properties/enum.SortProperties.html) -- `ASC`, `DESC` and (the new) `GROUPED`. Here is how these relate, via example **`ASC`**: rows are sorted by ascending `month` ```text month | value ------+------ Jan | 100 Jan | 42 Feb | 77 Feb | 12 Mar | 55 Mar | 91 ``` **`DESC`**: rows are sorted by descending `month` ```text month | value ------+------ Mar | 55 Mar | 91 Feb | 77 Feb | 12 Jan | 100 Jan | 42 ``` **`GROUPED`** (new): all rows with the same `month` are contiguous, but the months themselves appear in no particular order ```text month | value ------+------ Feb | 77 Feb | 12 Mar | 55 Mar | 91 Jan | 100 Jan | 42 ``` `GROUPED` is strictly weaker than `ASC` / `DESC` (any sorted input is also grouped), but I think it is sufficient for the various streaming operations we have in DataFusion. For example, for streaming aggregation: once the value of `month` changes, the aggregator knows that group is complete and can emit it. One challenge with this approach would be that SortProperties I think is in arrow-rs and updating it / wrapping it in DataFusion would be fairly invasive / a breaking API cange The changes from @xavlee in https://github.com/apache/datafusion/pull/24698 is less disruptive, but is also basically a special case -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
