pepijnve commented on issue #23447:
URL: https://github.com/apache/datafusion/issues/23447#issuecomment-4952493546
Still tracing where things are getting stuck. Current lead is that in
`SpillPool`, the `SpillPoolReader` is returning `Pending` under the assumption
the writing side is still active, but all `SpillPoolWriter` have already been
dropped, so nothing ever gets you out of this situation.
This can be observed using the following patch
```
Index: datafusion/physical-plan/src/spill/spill_pool.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/datafusion/physical-plan/src/spill/spill_pool.rs
b/datafusion/physical-plan/src/spill/spill_pool.rs
--- a/datafusion/physical-plan/src/spill/spill_pool.rs (revision
360a56dfe48edfb5b7391b911b75d22959e33c6a)
+++ b/datafusion/physical-plan/src/spill/spill_pool.rs (date
1783884975812)
@@ -24,7 +24,7 @@
use arrow::datatypes::SchemaRef;
use arrow::record_batch::RecordBatch;
-use datafusion_common::Result;
+use datafusion_common::{internal_err, Result};
use datafusion_execution::{RecordBatchStream, SendableRecordBatchStream,
SpillFile};
use super::in_progress_spill_file::InProgressSpillFile;
@@ -696,6 +696,9 @@
// File not ready yet (waiting for writer)
// Register waker so we get notified when writer
adds more batches
let mut shared = self.shared.lock();
+ if shared.active_writer_count == 0 {
+ return
Poll::Ready(Some(internal_err!("SpillPoolFile not ready, but all
SpillPoolWriters have been dropped")))
+ }
shared.register_waker(cx.waker().clone());
return Poll::Pending;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
