alamb commented on code in PR #8986: URL: https://github.com/apache/arrow-datafusion/pull/8986#discussion_r1466259955
########## datafusion/core/src/lib.rs: ########## @@ -364,8 +368,26 @@ //! //! [`RepartitionExec`]: https://docs.rs/datafusion/latest/datafusion/physical_plan/repartition/struct.RepartitionExec.html //! [Volcano style]: https://w6113.github.io/files/papers/volcanoparallelism-89.pdf +//! [Morsel-Driven Parallelism]: https://db.in.tum.de/~leis/papers/morsels.pdf +//! [DataFusion paper submitted SIGMOD]: https://github.com/apache/arrow-datafusion/files/13874720/DataFusion_Query_Engine___SIGMOD_2024.pdf //! [implementors of `ExecutionPlan`]: https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#implementors //! +//! ## Thread Scheduling +//! +//! DataFusion incrementally computes output from a [`SendableRecordBatchStream`] +//! with `target_partitions` threads. Parallelism is implementing using multiple +//! [Tokio] [`task`]s, which are executed by threads managed by a tokio Runtime. +//! While tokio is most commonly used Review Comment: I should probably also point out that this phrasing came from the SIGMOD paper and I think at least @yjshen and @ozankabak had a part in authoring it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
