viirya commented on a change in pull request #1921:
URL: https://github.com/apache/arrow-datafusion/pull/1921#discussion_r820135554
##########
File path: datafusion/src/execution/memory_manager.rs
##########
@@ -340,7 +341,13 @@ impl MemoryManager {
} else if current < min_per_rqt {
// if we cannot acquire at lease 1/2n memory, just wait for
others
// to spill instead spill self frequently with limited total
mem
- self.cv.wait(&mut rqt_current_used);
+ let timeout = self
+ .cv
+ .wait_for(&mut rqt_current_used, Duration::from_secs(5));
Review comment:
> Each task is processing a finite partition of data. If no consumers
release and cause a stuck, I prefer to treat it as a potential bug, and we
should fix it instead of letting it slip through our fingers. We are not
dealing with a black box of operators like UDFs, but all controlled physical
operators in the engine.
I must say that you don't convince me (because basically you're unable to
answer the question) as I don't think the dynamic during consumers interaction
is understood here and should be treated like that. It looks like an issue in
the model, not a potential bug in some consumers. As seems I cannot also
convince you at the point so the discussion will be endless. I don't prepare to
continue on it. So I'd close this, leave this as is, and thanks for above
discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
