2010YOUY01 commented on issue #24603: URL: https://github.com/apache/datafusion/issues/24603#issuecomment-5565002223
> Long-lived aggregation state needs to delete groups that are no longer active. A streaming engine may evict keys after a TTL or watermark while keeping every other group available for future updates. I think the current groups accumulator physical design is only optimized for throughput, so it's really hard to make eviction efficient. Reusing the `GroupsAccumulator` s can make both batched and streaming execution harder to evolve. The existing `Accumulator` interface seems even more suitable than `GroupsAccumulator` to extend for streaming use cases. I'm wondering whether you've considered extending `Accumulator`, or adding a parallel accumulator implementation optimized for streaming use cases, and what the blockers are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
