rluvaton commented on PR #15591: URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5426889129
One of the problem with removing `EmitTo::First(n)` is: lets say batch size is 1000 the input is ordered by group by keys so you know when a group is finished (or other case where we can emit early) and lets say we get to group 900 and in the middle of it and then we no longer have more memory we won't be able to say emit first 899 as we can only emit next block, and when calling emit next block it will treat all current items in the block as finished, which is wrong Did not solve that problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
