GitHub user kz930 edited a discussion: Scaling verification of workflow to Python translation — looking for input
We've been working on translating Texera workflows into standalone Python
scripts and verifying that the translation is correct. We have a plan for
making the verification scale, but **we're
not yet sure a shared CSV fixture is the right idea**, so we'd like people's
input and advice before committing to it.
## Background: the two workflows
**Translation.** A Texera workflow is converted into a standalone Python
script. The UI sends the workflow to the translator, the translator asks each
operator for its code fragment, and the fragments are stitched together into
one `.py` file that is returned to the user.
<img width="1072" height="362" alt="Screenshot 2026-07-02 at 11 40 55 PM"
src="https://github.com/user-attachments/assets/c184e924-e2dc-4ed7-a88c-bd23a217118b";
/>
**Verification.** To check the translation is correct, we run each operator
two ways: first through the native Texera execution path (the ground truth),
then through the generated Python script. We compare the outputs from both
paths. If they match, the translation is considered faithful.
<img width="1086" height="530" alt="Screenshot 2026-07-02 at 11 41 09 PM"
src="https://github.com/user-attachments/assets/7a612aff-21d7-4b48-b8b2-fe99e2413649";
/>
## Problem: verification does not scale
The current verification approach requires **one hand-written test case per
operator**, and each test case needs its own input data and operator
configuration.
This does not scale. Texera has around **150 operators**, and more are added
over time. Manually writing and maintaining a fixture for every operator is too
much work.
## Options considered
**1. Keep hand-writing a fixture per operator (status quo).** Accurate, but
the maintenance cost grows with every new operator — the thing we're trying to
get away from.
**2. One shared CSV fixture + automatic configuration.** Instead of a
separate fixture per operator, build a single large shared CSV and
auto-configure each operator against it, so a new
operator needs little or no additional hand-written test code. (This is the
direction we currently lean toward — details below.)
**3. Per-operator generated/synthetic input.** Generate input tailored to each
operator's declared schema on the fly. More precise per operator, but closer in
spirit to the status quo and
harder to keep meaningful across domains.
## Our current position / proposed plan
For most table-based operators, use **one large shared CSV fixture**. Each
operator is tested against this shared input table, and its configuration is
filled automatically.
**Designing the shared CSV.** We scan operator fields and identify what each
field needs — numeric columns, text columns, date columns, country codes,
p-values in (0,1), price-related columns,
and other domain-specific values — and add meaningful columns to cover those
needs.
**Testing an operator.** Feed the shared CSV into the operator, auto-fill its
configuration, run both the native path and the generated Python path, and
compare outputs. Since both paths use
the same input and the same configuration, any output difference should
indicate a translation bug.
**Choosing a column for a field (two steps):**
1. **Match by type** — numeric fields get numeric columns, text fields get
text columns, date fields get date columns, and so on.
2. **Match by domain when type isn't enough** — e.g. a field that expects a
country code should get a country-code column, not just any text column.
If a field can't be filled meaningfully, we **flag it** rather than
silently passing the test.
**Filling non-column settings (by type):**
- Enum / mode fields → test each option.
- Boolean fields → test both true and false when appropriate.
- Numeric settings → use the operator's declared default, falling back to 1
if none.
## Example: `ChoroplethMap`
`ChoroplethMap` draws a world map. It has two column-reference fields:
- `locations` — needs a **country code**
- `color` — needs a **numeric** column
Against the shared CSV:
| id | name | score | iso_country |
|----|-------|-------|-------------|
| 1 | alice | 0.5 | USA |
| 2 | bob | 1.0 | CHN |
| 3 | carol | 1.5 | JPN |
auto-config fills the two fields using the two steps above:
- `color` → `score` — **match by type** (first numeric column)
- `locations` → `iso_country` — **match by domain** (a plain text column like
`name` would render nothing, so it needs the country-code column)
This gives the config `{ locations: "iso_country", color: "score" }`. Both
paths then run on the **same input + same config**:
```mermaid
flowchart LR
IN["shared CSV + auto-config<br/>locations=iso_country, color=score"] -->
A["native Texera"]
IN --> B["generated Python"]
A --> OA["output (Plotly JSON)"]
OA --> C{"equal?"}
OB --> C
C -->|yes| P["PASS — faithful"]
```
If the outputs match, the translation is faithful. `locations = iso_country`
is exactly the "match by domain" case: with any text column the map draws
nothing and the test would be vacuous — the country-code column makes it a
real test.
## Special cases
- **Image and ML/sklearn operators** don't fit a plain table CSV — they need
binary data, image data, or model/training-style inputs. These should use
separate fixtures.
- **Whole-workflow correctness.** Every operator passing individually doesn't
prove the *stitched* Python script is correct. So we'd also test a small set of
complete workflows (native vs.
generated). Only a few should be needed, since the stitching mainly has to
cover a handful of shapes — simple chains, fan-out, joins, and multiple-output
operators.
## Next steps (tentative)
1. Create one shared CSV file for testing.
2. Write code that automatically fills in each operator's settings.
3. Add small annotations to operators where needed.
## What we'd like input on
- Is a **single shared CSV** the right foundation, or does it fall apart for
too many operators?
- Is automatic type/domain-based configuration robust enough to trust, or
will it produce too many false passes?
- Are there better alternatives we haven't considered for making per-operator
verification scale?
GitHub link: https://github.com/apache/texera/discussions/6072
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
