aglinxinyuan opened a new pull request, #8671:
URL: https://github.com/apache/texera/pull/8671
### What changes were proposed in this PR?
Inside a control block (`Loop Start ... Loop End`), a text property of an
operator whose Scala descriptor generates Python, such as a sklearn trainer's
hyperparameter value, can now hold a loop-variable reference such as `$i`.
Until now the compiler rejected every reference on these operators, because
their code is generated before the loop runs.
The generated code already decodes each text property when the operator
runs. Inside a block, the decode of a `$i` text becomes a read of the
iteration's state, which each Python operator registers before `process_state`
(#8642):
```
Loop Start (i = 1) -> SVR trainer (C = "$i") -> Loop End
generated code, outside a block: C = float
(self.decode_python_template('JGk=')) # the text "$i"
generated code, inside a block: C = float (self.loop_variable_text('i'))
# 1, then 2, then 3
```
| Piece | What it does |
|---|---|
| pybuilder `decoderExpression` | The exact expression pyb renders a text
as; the renderer now uses it too, so the two cannot drift. |
| `PythonOperatorDescriptor` | When the descriptor holds text references
(only inside a block), each reference's decode expression in the generated code
becomes `self.loop_variable_text('<name>')`. Nothing else in the code changes. |
| `WorkflowCompiler` | A generated operator is accepted when every text
reference was rewritten. The compile error now names each property it cannot
bind and why: a numeric or boolean property (its value is written into the
code), a text the code pastes in as it is, a text the code never reads, or a
text that names a new output column. UDFs keep rejecting references, with their
own reason. |
| pyamber `Operator.loop_variable_text` | Returns the variable's current
value as text, merged over every state message the operator received (the
latest wins, as the JVM's `LateBoundExecutor` merges them), and spelled as the
JVM binds a text property: `true` / `false`, and scalars only. It fails with a
clear message when no state carried the name. |
Outside every block nothing changes: the sidecar is empty, so the code is
the same, and a text such as `$AAPL` stays text.
**Dependencies.** This builds on #8639 (the compiler's references) and #8642
(`loop_state`). Both are merged into this branch until they land on `main`. The
change in this PR is the top commit, fd46a3f977.
### Any related issues, documentation, discussions?
Closes #8669
### How was this PR tested?
New and updated tests. Each new test was run red against the pre-change code
first.
| Spec | Covers |
|---|---|
| `PythonTemplateBuilderApiSpec` | `decoderExpression` equals what pyb
renders for text, empty text, unicode, quotes, backslashes and newlines;
different texts give different expressions. |
| `StateReferenceModuleSpec` | `textReferences` keeps only the pointers
whose value is the `$name` text; typed placeholders, other texts and missing
pointers are dropped. |
| `PythonOperatorDescriptorSpec` (new) | On the real SVC trainer: a text
reference is rewritten everywhere it is decoded; an empty sidecar leaves the
code as generated; other texts (`$j`, `$index`, `rbf`) stay decoded; two
references; a typed placeholder is not rewritten; a failed generation keeps its
error. |
| `WorkflowCompilerSpec` | Sort and the SVR trainer bind a text reference
inside a block; outside a block `$USD` stays decoded; each rejection reason,
alone and together (typed, pasted, never read, names an output column); UDF,
Java UDF and R UDF references stay rejected; a failed generation reports its
own error. |
| `LoopIntegrationSpec` | End to end: `Text Input (1.0, 2.0, 3.0) -> Loop
Start (i = 1) -> SVR trainer (C = $i) -> Loop End` produces the fits `C = 1.0`,
`C = 2.0`, `C = 3.0`, one per iteration. It spawns Python workers, so it runs
in CI. |
| `test_operator.py`, `test_data_processor.py`, `test_main_loop.py` |
`loop_variable_text` for int, float, string and bool values; merged states;
errors before any state, for a missing name and for a non-scalar; a snippet
shaped like the generated code, run against a real operator. |
```
sbt "WorkflowCompiler/test"
# 97 passed
sbt "PyBuilder/test"
# 192 passed
sbt "WorkflowOperator/testOnly
org.apache.texera.amber.operator.PythonOperatorDescriptorSpec" # 11 passed
sbt "WorkflowCore/testOnly org.apache.texera.amber.core.state.*"
# 28 passed
sbt "WorkflowCompiler/scalafmtCheckAll" "WorkflowCompiler/scalafixAll
--check" # clean (and the other changed modules)
sbt "WorkflowExecutionService/Test/compile"
# LoopIntegrationSpec compiles
cd amber && pytest src/test/python/core/models/test_operator.py \
src/test/python/core/runnables/test_data_processor.py \
src/test/python/core/runnables/test_main_loop.py
# 131 passed
ruff format --check src/main/python src/test/python && ruff check
src/main/python src/test/python # clean
```
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
