GitHub user aglinxinyuan created a discussion: Use a loop's variables in any
operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, you can type
> `$K` into a property of any operator in the body, and every iteration runs
> that operator with the current value of the loop variable `K`.
Issues: #8635, #8636, #8641. PRs: #8639 (engine), #8640 (property panel), #8642
(Python operators).
---
## Why
A loop declares its variables in Loop Start's **Initialization** (for example
`K = 2`) and updates them in Loop End's **Update** (`K += 1`). Today only those
two operators can read the variables. The operators in between, the ones you
actually loop for, take constants. So a parameter sweep such as "run the same
pipeline for K = 2, 3, ..., 10" has to be rewired by hand or rewritten as a
Python UDF.
## What it looks like
Example: which reviews pass each rating threshold.
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output:
table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
1. Declare the variable in Loop Start's Initialization: `min_stars = 1`.
2. Open any operator between Loop Start and Loop End. In a property, type `$`:
the panel suggests the variables of the enclosing loop, here `$min_stars`. Pick
one.
3. Run. The first iteration filters with `stars >= 1`, the second with `stars
>= 2`, and so on up to 5. Loop End collects the rows of every iteration.
The operator itself does not change. That is the point: it works for built-in
operators that were written long before loops existed.
### What you see in the property panel
| Situation | What happens |
|---|---|
| Operator inside a loop | Text, number and true/false fields accept `$name`.
Typing `$` suggests the loop's variables. |
| `$foo`, but no enclosing loop declares `foo` | The field is flagged: "$foo is
not a variable of an enclosing block". The workflow cannot run until it is
fixed. |
| The variable is renamed in Loop Start | Operators still using the old name
are flagged right away. |
| Operator moved out of the loop | Its fields go back to normal. `$K` in a
number field is an error again, and a text such as `$AAPL` is plain text,
exactly as today. |
### In your own Python UDF
For a UDF, the variable belongs in your code rather than in a property. A UDF
inside the loop receives the loop's variables when they arrive, the same way it
receives each tuple:
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_state(self, state, port):
self.min_stars = state["min_stars"] # called when the loop's variables
arrive
return state
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.min_stars:
yield tuple_
```
With #8642, the same values are also available anywhere in the UDF as
`self.loop_state`.
## Not supported yet
- **Operators that generate Python code**, such as the sklearn and chart
operators, and the property fields of Python UDFs. The panel accepts `$K`
there, but the run stops with "cannot refer to loop variables yet".
- **An operator that also takes input from outside the loop**, for example a
join whose other side comes from before Loop Start.
- **Properties that name a column**, such as the attributes of Projection.
- **Text that starts with `$` inside a loop.** A cashtag such as `$AAPL` always
means a variable there. Outside loops nothing changes.
- **Operators inside an inner loop** of nested loops. The panel suggests the
variables of both loops there, but the run does not bind them reliably yet.
- **The Form View** does not offer `$K` yet.
## Feedback wanted
1. Is `$name` the right syntax? And how should a literal `$` be written inside
a loop, for example `$$AAPL`?
2. Inside a loop, true/false fields become text boxes so that they can hold
`$flag`. Would you rather keep the checkbox and add a "use a loop variable"
toggle?
3. Should the panel refuse `$K` up front on operators that cannot use it yet,
instead of the run reporting it?
4. Which operators should come next? The sklearn operators are the obvious
candidates, for example K-means with `$K` clusters.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
