GitHub user aglinxinyuan edited a discussion: A control-variable port on every operator: design and prototype
## Summary Every operator gets one more input port, the **control-variable port**, drawn in purple. Whatever arrives on it becomes the operator's control variables, and a property that holds `$name` takes the value of the variable `name`. An existing operator can then run once per loop iteration on a value computed upstream. Today File Scan needs a second "from input" operator for this. A prototype that runs end to end is on [`aglinxinyuan/texera:control-variable-port`](https://github.com/aglinxinyuan/texera/tree/control-variable-port), with draft PR #8917. ## Example: read every file of a folder ``` Text Input ──► Loop Start ══(purple)══► File Scan ──► Loop End (one path init: i = 0 fileName = $file update: i += 1 per row) output: file = D.line[i] condition: i < len(D) ``` File Scan is a source. Its only change is that it resolves the file name at run time, once `$file` is bound. In each iteration, Loop Start's Output sets `file` to that iteration's path and Loop Start sends `{i, file}` into File Scan's port. File Scan binds `$file`, opens that file, and its rows flow on to Loop End. ## Design 1. **One port per operator, no new edge type.** Any output port may link to it, and the port does the conversion. 2. **No schema.** A control message merges into the operator's control variables. A tuple's columns become variables of the same name. 3. **Binding is by name.** A name that never arrives is a run-time error. 4. **The port's input ends before the operator starts.** The edge into the port is materialized. The receiver's data edges can still be pipelined. 5. **The later value wins.** Several edges may enter the port. Everywhere, if two values of one name reach an operator at the same loop depth before it first uses them, the later one replaces the earlier one. This replaces today's conflict error. 6. **Port-fed values travel with the block.** They are forwarded through the rest of the loop block, like any other control variable. 7. **Loop Start's Output is a Python cell.** Every name it assigns becomes a control variable of that iteration. The value of a trailing bare expression is the data sent to the body; without one, the body gets no data. A single expression such as `table.iloc[[i]]` works as before. ## Prototype - The port is the reserved `PortIdentity(-1)`. The compiler adds it only when a link targets it. - The worker starts the port's reader first and holds data channels until the port's input ends. - Loop Start's Initialization and Output can read `D`. Output runs once per iteration, before Loop Start sends its variables, so the body and the port see what it assigns. - The frontend draws the port and its links in purple and sends each link as port -1. The property panel offers the names a Loop Start's Initialization and Output assign. **PASTA's search needs no change.** The prototype adds the port link to `getBlockingAndDependeeLinks`, so the sender's region runs before the receiver's. The receiver's own data edge can still be pipelined. **End-to-end test.** `ControlVariablePortIntegrationSpec` runs the example over three files that hold the numbers 1 to 2, 3 to 5 and 6, one per line: once with `file` set in Output as above, and once with it set in Initialization and Update. Loop End receives `1` through `6` both times, in CI on Linux and macOS. The workflow also runs in a local deployment. ## Limits - **Among file readers, only File Scan.** The CSV reader infers its schema from the file at compile time, and with `$file` there is no file yet. - **No receivers that run code** (UDFs, Python-based operators, Loop Start, Loop End). They are rejected at compile time. - **Only the first tuple that reaches the port is converted.** Later tuples, from any edge, are dropped. - **One output port cannot feed both the port and a data port** of one operator. The compiler rejects this. - **The branch includes open PRs** #8640 and #8642. The prototype itself is [these commits](https://github.com/aglinxinyuan/texera/compare/3d524b4d9f08edbfdf541afcac97627a717dad1a...control-variable-port). GitHub link: https://github.com/apache/texera/discussions/8915 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
