crepererum commented on code in PR #14028:
URL: https://github.com/apache/datafusion/pull/14028#discussion_r1905684304
##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -478,7 +478,12 @@ impl<F: FileOpener> FileStream<F> {
reader,
)),
partition_values,
- }
+ };
+ // Return control to the runtime
when we're ready to open the next file
+ // to prevent uncancellable
queries in scenarios with many large files.
+ // This functions similarly to a
`tokio::task::yield_now()`.
+ cx.waker().wake_by_ref();
+ return Poll::Pending;
Review Comment:
I think this may not be correct. The `wake_by_ref` should be called AFTER
returning `Pending`. I think you may be lucky that this works, but it's
undefined behavior. If I understand the `yield_now` impl. correctly, it tries
to delay the "wake" call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
