alamb commented on code in PR #8022:
URL: https://github.com/apache/arrow-datafusion/pull/8022#discussion_r1380847568
##########
datafusion/physical-expr/src/partitioning.rs:
##########
@@ -15,14 +15,94 @@
// specific language governing permissions and limitations
// under the License.
-//! [`Partitioning`] and [`Distribution`] for physical expressions
+//! [`Partitioning`] and [`Distribution`] for `ExecutionPlans`
use std::fmt;
use std::sync::Arc;
use crate::{expr_list_eq_strict_order, EquivalenceProperties, PhysicalExpr};
-/// Partitioning schemes supported by operators.
+/// Output partitioning supported by [`ExecutionPlan`]s.
+///
+/// When `executed`, `ExecutionPlan`s produce one or more independent stream
of
+/// data batches in parallel, referred to as partitions. The streams are Rust
+/// `aync` [`Stream`]s (a special kind of future). The number of output
+/// partitions varies based on the input and the operation performed.
+///
+/// For example, an `ExecutionPlan` that has output partitioning of 3 will
+/// produce 3 distinct output streams as the result of calling
+/// `ExecutionPlan::execute(0)`, `ExecutionPlan::execute(1)`, and
+/// `ExecutionPlan::execute(2)`, as shown below:
+///
+/// ```text
+/// ... ...
...
+/// ... ▲ ▲
▲
+/// │ │
│
+/// ▲ │ │
│
+/// │ │ │
│
+/// │ ┌───┴────┐ ┌───┴────┐
┌───┴────┐
+/// ┌────────────────────┐ │ Stream │ │ Stream │ │
Stream │
+/// │ ExecutionPlan │ │ (0) │ │ (1) │ │
(2) │
+/// └────────────────────┘ └────────┘ └────────┘
└────────┘
+/// ▲ ▲ ▲
▲
+/// │ │ │
│
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │
│
+/// Input │ │ │
│
+/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │
│
+/// ▲ ┌ ─ ─ ─ ─ ┌ ─ ─ ─ ─ ┌ ─
─ ─ ─
+/// │ Input │ Input │
Input │
+/// │ │ Stream │ Stream │
Stream
+/// (0) │ (1) │
(2) │
+/// ... └ ─ ▲ ─ ─ └ ─ ▲ ─ ─ └ ─
▲ ─ ─
+/// │ │
│
+/// │ │
│
+/// │ │
│
+///
+// ExecutionPlan with 3 output 3 (async) streams, one for
each
Review Comment:
I am struggling with how to draw this -- the `ExecutionPlan` has a single
child input (another `ExecutionPlan`) but both its child and itself produce 3
output partitions / streams
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
