Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
alamb closed issue #16396: Blog post about DataFusion Async / Stream execution model / cancellation URL: https://github.com/apache/datafusion/issues/16396 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
pepijnve commented on issue #16396: URL: https://github.com/apache/datafusion/issues/16396#issuecomment-2974057031 @alamb I have a first draft written up at https://github.com/apache/datafusion-site/pull/75. I'm not a great blog writer, so any help in getting this over the finish line would be appreciated. This was written under the assumption that all the stars align, the Tokio PR lands, the DataFusion PR lands, etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
alamb commented on issue #16396: URL: https://github.com/apache/datafusion/issues/16396#issuecomment-2970443116 BTW to make a blog post, we make PRs to this repo - https://github.com/alamb/datafusion-site -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
zhuqi-lucas commented on issue #16396: URL: https://github.com/apache/datafusion/issues/16396#issuecomment-2970103102 Very good point for pushing a blog for this topic! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
pepijnve commented on issue #16396: URL: https://github.com/apache/datafusion/issues/16396#issuecomment-297302 👍 I'll try to go for 'how **did** we improve this' then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
alamb commented on issue #16396: URL: https://github.com/apache/datafusion/issues/16396#issuecomment-296978 > [@alamb](https://github.com/alamb) the part I'm still working on is 'so how do we fix this?'. I'll let you know here when that's ready. SOunds good -- I think the background knowledge is also really valuable, though the narrative arc of "here is how we improved cancellation" would be more compeling -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
pepijnve commented on issue #16396: URL: https://github.com/apache/datafusion/issues/16396#issuecomment-2969885438 @alamb the part I'm still working on is 'so how do we fix this?'. I'll let you know here when that's ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[I] Blog post about DataFusion Async / Stream execution model / cancellation [datafusion]
alamb opened a new issue, #16396: URL: https://github.com/apache/datafusion/issues/16396 ### Is your feature request related to a problem or challenge? The use of Rust `async` and `Stream` for execution is not obvious to everyone working with DataFusion and takes some getting used to. The more end user facing documentation we have about the subject the easier it is for others to get involved and make DataFusion work for them ### Describe the solution you'd like I would like a blog post describing how the DataFusion `Stream`/ `Future`s execution model works -- basically a tutorial that 1. Introduces the background knowledge needed to use Streams 2. Applies that background knowledge to writing Datafusion operators (aka Streams) 3. Adds an example of using Streams ### Describe alternatives you've considered - As part of https://github.com/apache/datafusion/issues/16353#issuecomment-2964042212, @pepijnve wrote up a great document describing the DataFusion execution model (as background to describe the cancellation mechanism) which I think has 75% of a blog post already https://github.com/pepijnve/datafusion/blob/cancel_spec/dev/design/cancellation.md Here is the text from the document above: # Query Cancellation ## The Challenge of Cancelling Long-Running Queries Have you ever tried to cancel a query that just wouldn't stop? This document explores why that happens in DataFusion and what we can do about it. ### Understanding Rust's Async Model Rust's asynchronous programming model is built around the `Future` trait, which works quite differently from async models in other languages. Unlike systems where async code runs continuously on background threads, a Rust `Future` represents a lazy calculation that only makes progress when explicitly polled. When you call the `poll` function on a `Future`, you're asking it to advance its calculation as much as possible. The future responds with either: - `Poll::Pending` if it needs to wait for something (like I/O) before continuing - `Poll::Ready` when it has completed and produced a value When a future returns `Pending`, it saves its internal state so it can pick up where it left off the next time you poll it. This state management is what makes Rust's futures memory-efficient and composable. Rust's `async` keyword provides syntactic sugar over this model. When you write an `async` function or block, the compiler transforms it into an implementation of the `Future` trait. This transformation makes async code much more readable while maintaining the same underlying mechanics. The `await` keyword complements this by letting you pause execution until a future completes. When you write `.await` after a future, you're essentially telling the compiler to poll that future until it's ready, and then continue with the result. ### From Futures to Streams The `futures` crate extends this model with the `Stream` trait, which represents a sequence of values produced asynchronously rather than just a single value. A `Stream` has a `poll_next` method that returns: - `Poll::Pending` when the next value isn't ready yet - `Poll::Ready(Some(value))` when a new value is available - `Poll::Ready(None)` when the stream is exhausted ### How DataFusion Executes Queries In DataFusion, query execution follows this async pattern. When you run a query: 1. The query is compiled into a tree of `ExecutionPlan` nodes 2. Calling `ExecutionPlan::execute` returns a `SendableRecordBatchStream` (essentially a `Box>`) 3. This stream is actually the root of a tree of streams where each node processes data from its children Query execution progresses each time you call `poll_next` on the root stream. This call typically cascades down the tree, with each node calling `poll_next` on its children to get the data it needs to process. Here's where things get tricky: some operations (like aggregations, sorts, or certain join phases) need to process a lot of data before producing any output. When `poll_next` encounters one of these operations, it might need to do substantial work before returning. ### Tokio and Cooperative Scheduling DataFusion runs on top of Tokio, which uses a cooperative scheduling model. This is fundamentally different from preemptive scheduling: - In preemptive scheduling, the system can interrupt a task at any time to run something else - In cooperative scheduling, tasks must voluntarily yield control back to the scheduler This distinction is crucial for understanding our cancellation problem. When a Tokio task is running, it can't be forcibly interrupted - it must cooperate by periodically yielding control. When you call `JoinHandle::abort()` on a Tokio task, you're not immedia
