GitHub user kz930 edited a discussion: CI for workflow-to-Python verification: pooled Python workers
Design of the verification itself is in #6072 — this is only about why it was slow and what fixed it. ### How we check one operator Three steps, and each one that needs Python used to start a brand-new Python process: 1. **Run it the normal Texera way** — the ground truth. Scala operators run in the JVM, no Python needed. Python-based operators (Sort, most charts) start a Python process. 2. **Run the standalone Python script** our translator generated for it — always Python. 3. **Compare the two outputs** — a small pandas script, also Python. So one check = **2 Python starts for a Scala operator, 3 for a Python-based one**. And we do one check per enum option (line chart mode = line / dots / line+dots), so a single operator often means several checks. ### The problem - Starting Python + importing pandas/pyamber: **~300 ms**. - The operator's actual work on our small test table: **~a few ms**. - So **~96% of the time was imports** — repaid on every single run, across 148 discovered operators. ### The fix - **Keep Python alive.** Start it once, import once, then feed it jobs: one job in on stdin, result out on stdout. Every job after the first skips the imports. - **Batch.** Many operators share one worker, so imports are paid per worker, not per run. - **One pool per role** — Texera path, generated script, comparison. Three different worker scripts (and the first needs its own `PYTHONPATH`), so workers are never mixed between them. - **Run 4 at a time.** Each pool keeps up to 4 workers; a worker takes one job at a time, since it changes directory per job. - **If a worker crashes**, we drop it and run that one job the old way — never worse than before. A normal test failure isn't a crash; the worker reports it and stays alive. `VERIFY_PYTHON_WORKER=0` turns pooling off entirely. - **In CI**, it's a step in the existing platform-integration job for workflow-compiling-service (Python and the service are already there). The unit job skips it by tag and stays fast. ### The result Same suite, same 53 operators, same machine: | | test run | wall clock | CPU used | |---|---|---|---| | workers on | **8.0 s** | 14.2 s | 61 s | | workers off | 32.8 s | 39.3 s | 201 s | - **~4× faster, ~3× less CPU.** - CI is green — 53 operators at the last run. ### How to test it Point `UDF_PYTHON_PATH` at a Python 3.12 that has pandas, plotly, pyarrow and betterproto, then: `UDF_PYTHON_PATH="$PWD/../venv312/bin/python" sbt "WorkflowCompilingService/testOnly *OperatorBehaviorSpec"` GitHub link: https://github.com/apache/texera/discussions/6975 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
