suremarc commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5469717323
Actually, maybe this is possible with existing machinery today? If you `GROUP BY time_partition, key, date_bin(time)` then the aggregate becomes fully streaming. I tested this with some fake data in `datafusion-cli`: https://gist.github.com/suremarc/a32b8cdcfc1227d977db8b87969c7f08 Now, I wish we didn't have to manually add `time_partition` to the group key, but the fact this is possible in stock DataFusion with an alternate spelling makes me think a solution here should require no additional execution machinery. This also won't handle use cases where group keys straddle partitions, but I don't think that was in scope in the first place. I think if DataFusion had more awareness of data partitioning, it should be possible to do a rewrite of `key, date_bin(time)` -> `time_partition, key, date_bin(time)` automatically with an optimizer rule. But it's not clear to me yet the right way to implement this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
