GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, you can type > `$K` into a property of any operator in the body, and every iteration runs > that operator with the current value of the loop variable `K`. **Pull requests** (reviewed and merged separately; the panel needs the engine first): | Part | Pull request | Issue | |---|---|---| | Engine: binds `$K` in operator properties | [#8639 feat(amber): bind $K loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 | | Property panel: accepts and suggests `$K` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 | | Python operators: `self.loop_state` in UDFs | [#8642 feat(pyamber): register loop_state before process_state](https://github.com/apache/texera/pull/8642) | #8641 | --- ## Why A loop declares its variables in Loop Start's **Initialization** (for example `K = 2`) and updates them in Loop End's **Update** (`K += 1`). Today only those two operators can read the variables. The operators in between, the ones you actually loop for, take constants. So a parameter sweep such as "run the same pipeline for K = 2, 3, ..., 10" has to be rewired by hand or rewritten as a Python UDF. ## What it looks like Example: which reviews pass each rating threshold. ```mermaid flowchart LR R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"] LS --> F["Filter<br/>stars >= $min_stars"] F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"] ``` 1. Declare the variable in Loop Start's Initialization: `min_stars = 1`. 2. Open any operator between Loop Start and Loop End. In a property, type `$`: the panel suggests the variables of the enclosing loop, here `$min_stars`. Pick one. 3. Run. The first iteration filters with `stars >= 1`, the second with `stars >= 2`, and so on up to 5. Loop End collects the rows of every iteration. The operator itself does not change. That is the point: it works for built-in operators that were written long before loops existed. ### What you see in the property panel | Situation | What happens | |---|---| | Operator inside a loop | Text, number and true/false fields accept `$name`. Typing `$` suggests the loop's variables. | | `$foo`, but no enclosing loop declares `foo` | The field is flagged: "$foo is not a variable of an enclosing block". The workflow cannot run until it is fixed. | | The variable is renamed in Loop Start | Operators still using the old name are flagged right away. | | Operator moved out of the loop | Its fields go back to normal. `$K` in a number field is an error again, and a text such as `$AAPL` is plain text, exactly as today. | ### In your own Python UDF For a UDF, the variable belongs in your code rather than in a property. A UDF inside the loop receives the loop's variables when they arrive, the same way it receives each tuple: ```python class ProcessTupleOperator(UDFOperatorV2): def process_state(self, state, port): self.min_stars = state["min_stars"] # called when the loop's variables arrive return state def process_tuple(self, tuple_, port): if tuple_["stars"] >= self.min_stars: yield tuple_ ``` With #8642, the same values are also available anywhere in the UDF as `self.loop_state`. ## Not supported yet - **Operators that generate Python code**, such as the sklearn and chart operators, and the property fields of Python UDFs. The panel accepts `$K` there, but the run stops with "cannot refer to loop variables yet". - **An operator that also takes input from outside the loop**, for example a join whose other side comes from before Loop Start. - **Properties that name a column**, such as the attributes of Projection. - **Text that starts with `$` inside a loop.** A cashtag such as `$AAPL` always means a variable there. Outside loops nothing changes. - **Operators inside an inner loop** of nested loops. The panel suggests the variables of both loops there, but the run does not bind them reliably yet. - **The Form View** does not offer `$K` yet. ## Feedback wanted 1. Is `$name` the right syntax? And how should a literal `$` be written inside a loop, for example `$$AAPL`? 2. Inside a loop, true/false fields become text boxes so that they can hold `$flag`. Would you rather keep the checkbox and add a "use a loop variable" toggle? 3. Should the panel refuse `$K` up front on operators that cannot use it yet, instead of the run reporting it? 4. Which operators should come next? The sklearn operators are the obvious candidates, for example K-means with `$K` clusters. GitHub link: https://github.com/apache/texera/discussions/8643 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
