GitHub user aglinxinyuan edited a discussion: A control-variable port on every operator: design and prototype
## Summary Every operator gets one more input port, the **control-variable port**, drawn in purple. Whatever arrives on it becomes the operator's control variables, and a property that holds `$name` takes the value of the variable `name`. An existing operator can then run once per loop iteration on a value computed upstream. Today File Scan needs a second "from input" operator for this. A prototype that runs end to end is on [`aglinxinyuan/texera:control-variable-port`](https://github.com/aglinxinyuan/texera/tree/control-variable-port). It is a prototype for discussion, so there is no PR. ## Example: read every file of a folder ``` Text Input ──► Loop Start ══(purple)══► File Scan ──► Loop End (one path init: i = 0 fileName = $file update: i += 1 per row) file = D.line[0] file = D.line.get(i) output: None condition: i < len(D) ``` File Scan is a source. Its only change is that it resolves the file name at run time, once `$file` is bound. In each iteration, Loop Start sends `{i, file}` into File Scan's port. File Scan binds `$file`, opens that file, and its rows flow on to Loop End. ## Design 1. **One port per operator, no new edge type.** Any output port may link to it, and the port does the conversion. 2. **No schema.** A control message merges into the operator's control variables. A tuple's columns become variables of the same name. 3. **Binding is by name.** A name that never arrives is a run-time error. 4. **The port's input ends before the operator starts.** The edge into the port is materialized. The receiver's data edges can still be pipelined. 5. **The later value wins.** Several edges may enter the port. Everywhere, if two values of one name reach an operator at the same loop depth before it first uses them, the later one replaces the earlier one. This replaces today's conflict error. 6. **Port-fed values travel with the block.** They are forwarded through the rest of the loop block, like any other control variable. ## Prototype - The port is the reserved `PortIdentity(-1)`. The compiler adds it only when a link targets it. - The worker starts the port's reader first and holds data channels until the port's input ends. - Loop Start's `init()` can read `D`. - The frontend draws the port and its links in purple and sends each link as port -1. **PASTA's search needs no change.** The prototype adds the port link to `getBlockingAndDependeeLinks`, so the sender's region runs before the receiver's. The receiver's own data edge can still be pipelined. **End-to-end test.** `ControlVariablePortIntegrationSpec` runs the example over three files that hold the numbers 1 to 2, 3 to 5 and 6, one per line. Loop End receives `1` through `6`. It passes locally. ## Limits - **Among file readers, only File Scan.** The CSV reader infers its schema from the file at compile time, and with `$file` there is no file yet. - **No receivers that run code** (UDFs, Python-based operators, Loop Start, Loop End). They are rejected at compile time. - **Only the first tuple that reaches the port is converted.** Later tuples, from any edge, are dropped. - **One output port cannot feed both the port and a data port** of one operator. The compiler rejects this. - **The branch includes open PRs** #8640, #8642 and #8750, plus the merged #8639. The prototype itself is [these commits](https://github.com/aglinxinyuan/texera/compare/3d524b4d9f08edbfdf541afcac97627a717dad1a...control-variable-port). GitHub link: https://github.com/apache/texera/discussions/8915 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
