carloea2 opened a new issue, #8319:
URL: https://github.com/apache/texera/issues/8319
## Feature Summary
Texera users currently have to rewrite Python programs manually to execute
them as workflows. Add a compiler path that accepts a `.py` file or executable
Jupyter notebook code cells and produces an ordinary Texera workflow of Python
UDF operators.
The initial scope groups complete top-level statements. Supported compound
statements such as `if`, `for`, `while`, `try`, and `with` remain single
placement units; their bodies are not distributed. Unsupported Python semantics
return source-located diagnostics. Compilation performs static analysis and
does not execute user code.
## Proposed Solution or Design
```text
.py or .ipynb code cells
|
v
ANALYZE -> GROUP -> VERIFY -> BUILD
|
v
ordinary Texera workflow -> standard Amber execution
```
| Layer | Responsibility |
| --- | --- |
| Compiler | Parse source, derive statement dependencies, choose legal
groups, verify boundaries, and emit workflow JSON |
| PyTexera runtime | Import and export selected Python values through a
Cloudpickle envelope |
| Texera integration | Invoke the compiler, report safe diagnostics, create
the workflow, and execute standard Python UDF operators |
The default grouping strategy targets `min(statement_count,
ceil(sqrt(physical LOC)))` contiguous groups. Every proposal must remain
acyclic and satisfy dependency, local reconstruction, and boundary transport
checks. A one-statement-per-operator strategy remains available as a diagnostic
baseline.
Initial scope:
| Supported | Deferred with a source-located diagnostic |
| --- | --- |
| Imports, assignments, expressions, and ordinary calls | Function and class
definitions |
| Complete `if`, `for`, `while`, `try`, and `with` statements | Lambdas,
async constructs, `global`, and `nonlocal` |
| Python files and executable notebook code cells | Wildcard and relative
imports |
| Transport of downstream-required values | Intrastatement and control-body
decomposition |
Acceptance criteria:
- `.py` and `.ipynb` inputs compile through the same path.
- Generated JSON imports as a normal Texera workflow.
- Generated UDFs execute in an Amber integration test.
- Boundary transport preserves required values, aliases, cycles, and field
presence.
- Known unsupported source returns an HTTP 422 response with a source
location.
- Unexpected failures return a generic HTTP 500 response while details
remain server-side.
- The frontend displays compiler diagnostics and creates a workflow on
success.
- The deployed service contains the compiler runtime and does not depend on
a repository checkout.
Design discussion: https://github.com/apache/texera/discussions/8160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
