GitHub user carloea2 created a discussion: py2wf: a compiler from plain Python
scripts to operator DAG workflows (.py → .pyt → workflow)
## Summary
I have an MVP of `python_to_workflow` (under
`py2udf/src/main/python/python_to_workflow/`): a compiler that takes an
ordinary single-file Python script and turns it into a **runnable operator
DAG**, using one simple, predictable rule:
> **Every statement becomes one operator.** Then, optionally, a heuristic fuses
> **N contiguous statements into one operator** to bring the graph down to a
> human-scale workflow.
The output is a `.pyt` file, still plain, executable Python, but structured so
each operator is a function with explicit input/output ports, and from there a
workflow JSON of stock `PythonUDFV2` operators that the engine accepts today.
## The 1-line-1-operator form
`pipeline.compile(source)` emits the finest-grained form, which we call **P⊥**:
- A single `_main()` orchestrator keeps control flow **visible**,
`for`/`while`/`if` stay as ordinary statements in `_main`.
- **Every value-producing line becomes one operator function** `_<var>_N(ctx,
live_in) -> live_out`, including loop predicates and iterables.
- **Ports are computed, not declared:** a liveness analysis determines each
operator's inputs (names read that were bound earlier) and outputs (names
written that are still live afterwards). A variable rebound after a branch *is*
the merge, no special merge nodes.
- Imports, class/function definitions, and constants go to a shared **context**
(`ctx`), registered by reference so operators can read them without threading.
Example (input → P⊥ shape):
```python
# input.py
data = load_rows()
total = 0
for r in data:
total += r.amount
print(total)
```
```python
# P⊥ .pyt (abridged)
def _data_1(ctx): return load_rows()
def _total_2(ctx): return 0
def _total_3(ctx, total, r): return total + r.amount
def _print_4(ctx, total): print(total)
def _main():
ctx = _Ctx(); ...
data = _data_1(ctx)
total = _total_2(ctx)
for r in data:
total = _total_3(ctx, total, r)
_print_4(ctx, total)
_main()
```
P⊥ is deliberately fine-grained.
## The grouping heuristic: N contiguous lines = 1 operator
One operator per statement is too many operators for a real canvas. The MVP
coarsening rule is intentionally simple:
- A grouping maps each statement to a block id; **consecutive statements
sharing a block id fuse into a single operator**.
- Fusing is a re-realization, not new codegen: the fused operator's body is the
statements spliced together, and its ports are recomputed by the same liveness
rule (inputs = reads entering the block, outputs = writes still live after the
block).
- The default heuristic is a fixed chunk size N (with control-flow statements
acting as natural boundaries), exposed as a knob. Contiguity in program order
guarantees the fused graph stays acyclic.
So the whole MVP surface is: `compile(source)` → P⊥, and `compile(source,
group=N)` → the same program as ⌈lines/N⌉-ish fused operators. Both outputs
run, and both are verified against the original script.
## What the MVP requires of the input
To make "one statement = one operator" well-defined, a small normalization
front-end runs first (all source-to-source, behavior-preserving):
comprehensions become loops, functions are inlined toward the main body, and
`global` state is rewritten into explicit parameter/return threading so every
dependency between two lines is a visible variable. Scripts using features
outside that subset are rejected rather than miscompiled.
## Non-goals (for this MVP)
- No data-parallel sharding decisions; each operator is a plain UDF.
- Single-file, synchronous Python only.
## Questions for discussion
- I know there's already work on translating python to workflows with LLMs. Is
there room for a compiler-based version too?
- Are we ok merging something this big? It's around 20k lines of compiler passes
- If there's interest, Is anyone interested in reviewing this?
GitHub link: https://github.com/apache/texera/discussions/6618
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
