pepijnve commented on issue #23447: URL: https://github.com/apache/datafusion/issues/23447#issuecomment-4952972447
Digging a bit deeper I think there's a race condition in that code. Suppose there's an active spill file and then the following sequence happens: - `SpillPoolReader#pool_next` calls `SpillPoolFile#pool_next` - `SpillPoolFile#pool_next` acquires the `ActiveSpillFileShared` mutex, catches up to the writer and takes the `else` code path in step 1 - `SpillPoolFile#pool_next` returns `Pending` and releases the `ActiveSpillFileShared` mutex - Now the last `SpillPoolWriter` is dropped, it acquires the `ActiveSpillFileShared` mutex does some cleanup, it tries to wake the registered waker, but at this point there is none. - Finally we're back to `SpillPoolReader#pool_next` where the `SpillPoolFile#pool_next` call returns with `Pending`. A waker is registered, but this one is only triggered if another `SpillPoolWriter` is dropped or pushes a batch. The last `SpillPoolWriter` has already been dropped so this will never happen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
