aglinxinyuan opened a new pull request, #8645:
URL: https://github.com/apache/texera/pull/8645
### What changes were proposed in this PR?
This draft puts the whole loop-variable feature in one branch so it can be
tried and tested end to end: inside a Loop Start ... Loop End block, any
operator's property can hold `$K`, and every iteration runs the operator with
the current value of the loop variable `K`.
It combines three PRs, one commit each, unchanged:
| Commit | PR | What it adds |
|---|---|---|
| `feat(amber): bind $K loop-variable references in operators` | #8639 | The
engine accepts `$K` in string and numeric/boolean properties, keeps it only for
operators inside a loop block, and binds it at each iteration's first tuple. |
| `feat(pyamber): register loop_state before process_state` | #8642 | Python
operators, UDFs included, can read the current iteration's variables as
`self.loop_state`. |
| `feat(frontend): accept $variable references in loop blocks` | #8640 |
Inside a block, the property panel accepts `$K` in any primitive field,
suggests the enclosing loop's variables when you type `$`, and flags a name no
enclosing loop declares. |
```
Reviews -> Loop Start (min_stars = 1) -> Filter (stars >= $min_stars) ->
Loop End (min_stars += 1; min_stars <= 5)
iteration 1 filters stars >= 1, iteration 2 stars >= 2, ... up to
5
```
Outside every loop nothing changes: a text such as `$AAPL` stays plain text.
### Any related issues, documentation, discussions?
Combines #8639, #8642 and #8640. Related to #8635, #8641 and #8636.
User-facing overview: https://github.com/apache/texera/discussions/8643
### How was this PR tested?
On this branch (the three commits on current `main`):
```
sbt "project WorkflowCore" "testOnly *StateReferenceModuleSpec
*LateBoundExecutorSpec *CoreExecutorReflectionSpec"
sbt "project WorkflowCompiler" test
sbt "project WorkflowOperator" "testOnly *OperatorMetadataGeneratorSpec"
sbt "project WorkflowCompilingService" Test/compile
sbt "project WorkflowExecutionService" Test/compile "testOnly
*DataProcessorSpec"
cd amber/src/main/python && python -m pytest
../../test/python/core/models/test_operator.py
../../test/python/core/runnables/test_data_processor.py
../../test/python/core/runnables/test_main_loop.py -q
cd frontend && yarn exec tsc --noEmit -p tsconfig.json
cd frontend && yarn ng test --watch=false --include=<the 6 loop specs>
```
| Check | Result |
|---|---|
| WorkflowCore: `StateReferenceModuleSpec`, `LateBoundExecutorSpec`,
`CoreExecutorReflectionSpec` | 54/54 |
| WorkflowCompiler full `test` | 83/83 |
| WorkflowOperator: `OperatorMetadataGeneratorSpec` | 2/2 |
| workflow-compiling-service and amber `Test/compile` (incl.
`LoopIntegrationSpec`) | compiled |
| amber: `DataProcessorSpec` | 8/8 |
| pyamber: `test_operator.py`, `test_data_processor.py`, `test_main_loop.py`
| 110 passed |
| frontend: `tsc --noEmit`; the 6 loop spec files | clean; 6/6 files passed |
Each part's own tests are described in #8639, #8642 and #8640.
`LoopIntegrationSpec`'s new case (`TextInput -> Loop Start -> Filter(line = $i)
-> Loop End`, 3 rows) runs in the `amber-integration` CI job.
Manual steps (Chrome, desktop): link `Loop Start -> Limit -> Loop End` and
set the Loop Start initialization to `K = 2`. Open `Limit`: typing `$` in its
integer field offers `$K`, and choosing it keeps the operator valid. Run the
workflow, and each iteration applies the current `K`.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5.1, Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
