westonpace commented on PR #40722:
URL: https://github.com/apache/arrow/pull/40722#issuecomment-2030845396
> LGTM as a fixing here, seems currently we don't understand why this
happens?
Thanks for the review. I do understand why this was happening.
* We can't pause immediately because Acero is push-based. It takes time
for the pause signal to travel from sink to source. More tasks may be
scheduled during this time (and other tasks may be in flight).
* Previously, we did not finish queued tasks before unpausing. This means
we would unpause and let in more queued tasks and then pause again real quick.
This leads to the # of queued tasks growing without bound.
* The fix does not unpause until all queued tasks have run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
