viirya commented on a change in pull request #1921:
URL: https://github.com/apache/arrow-datafusion/pull/1921#discussion_r820135554
##########
File path: datafusion/src/execution/memory_manager.rs
##########
@@ -340,7 +341,13 @@ impl MemoryManager {
} else if current < min_per_rqt {
// if we cannot acquire at lease 1/2n memory, just wait for
others
// to spill instead spill self frequently with limited total
mem
- self.cv.wait(&mut rqt_current_used);
+ let timeout = self
+ .cv
+ .wait_for(&mut rqt_current_used, Duration::from_secs(5));
Review comment:
> Each task is processing a finite partition of data. If no consumers
release and cause a stuck, I prefer to treat it as a potential bug, and we
should fix it instead of letting it slip through our fingers. We are not
dealing with a black box of operators like UDFs, but all controlled physical
operators in the engine.
I must say that you don't convince me as I don't think the dynamic during
consumers interaction is understood here and should be treated like that. It
looks like an issue in the model, not a potential bug in some consumers. As
seems I cannot also convince you at the point so the discussion will be
endless. I don't prepare to continue on it. So I'd close this, leave this as
is, and thanks for above discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
