xudong963 commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5473519696
Thanks @suremarc for looping me into the issue. Yes, I think this generalization would be useful for Massive. We have tables partitioned by day or month and hash-binned by key, with each logical time partition sorted by `(key, time)`. When Atlas combines many logical partitions into one DataFusion execution partition, the effective ordering becomes `(time_partition, key, time)`. A query such as `GROUP BY key, date_bin(time)` therefore loses a globally usable `(key, time)` ordering, even though its groups remain non-overlapping when the time buckets do not cross partition boundaries. This is a direct use case for the original proposal. We also have many materialization queries that read time-ordered data and compute OHLC rollups grouped by key and time bucket using ordered `FIRST_VALUE` and `LAST_VALUE`. Those inputs are not necessarily fully group-contiguous because keys may interleave, but knowing that time is ordered when the key is fixed would allow the aggregate to retain only the current bucket per key. Today we compensate with additional execution partitions or hash-repartitioned aggregation, both of which have costs. From Massive's perspective, the conditional/per-logical-partition ordering model looks broadly useful for materialization, window functions, and potentially streaming execution. I'll follow up the issue and the related PRs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
