Brb6754 opened a new pull request, #8525:
URL: https://github.com/apache/texera/pull/8525
> Draft on purpose. The panel works end to end, but where a kept run should
live
> is an open design question and I would rather settle it with the committers
> than guess. See the last section.
### What changes were proposed in this PR?
A panel, opened from a button in the workspace menu, that shows the user what
Texera already records about a workflow and currently keeps to itself: every
save, and every run.
Nothing here is new data. `workflow_version` holds one inverse JSON patch per
save, so the whole history of a canvas is recoverable by walking the patches
backwards. A run's results are dropped after thirty seconds, but its runtime
statistics are not: they stay in an Iceberg table with a row per operator per
sample, which is enough to say how many rows went in and out of every step of
every run ever made.
The panel reads both and answers four questions in plain language, which is
what
its five views are named after:
- **How does this work?** The whole flow as one drawing, laid out by the
panel
rather than taken from the canvas, with each pipe as thick as the rows that
run through it, so where the flow narrows reads before any number does. A
step
inspector alongside gives one step at a time: what it does, rows in and
out,
its settings in words, and sample rows.
- **What did I change?** Every save as a block on a ribbon, coloured by what
kind of save it was, with the canvas rebuilt at whichever block is picked
and
that save's edits listed in words.
- **What has been tried?** Every run read as an experiment rather than a log
line, so the same settings run five times count as one thing tried. On the
workflow I developed against, twenty-one runs collapse to four.
- **Why did my results change?** Two runs compared step by step, naming the
first step whose output differs and the edit behind it.
- **Report.** The same facts written up as a handover document, asked for by
hand because it costs a model call.
Two rules hold throughout, and they are the part I would most like reviewed:
- **Every claim is derived and checkable.** A model is used only to put
already-derived facts into prose. It never decides which step diverged,
which
edit caused it, or what any count means. With no model configured the panel
loses its sentences and nothing else, so it does not become a feature that
stops working when a key expires.
- **Ordering is not a change.** A step returning the same rows in a different
order is reported as exactly that. The engine splits work between workers
non-deterministically, and an earlier version of this did blame a join for
it,
hiding the real edit further downstream.
Everything the panel writes is in plain language rather than in the system's
own
terms: canvas names instead of identifiers, property paths translated through
the same JSON schema that draws the operator form (so a filter reads
`amount > 400`, never its JSON), and dates in words. The technical form is
kept,
one disclosure down.
**Files.** One new component under
`frontend/src/app/workspace/component/spy/`,
three added lines' worth of button in the workspace menu, and a new
`spy-service` that answers it: it reads the database directly, holds no
Texera
session of its own, and its single writing route carries the token of the
user
who has Texera open, so whoever cannot run a workflow cannot run one from
here.
### Any related issues, documentation, discussions?
Proposed in #8524, which has the reasoning and the open question in full.
`spy-service/README.md` documents the modules and every environment variable.
### How was this PR tested?
No automated tests yet, which is the other thing I would like guidance on:
the
parts worth testing are the patch application and the topological comparison,
and I would rather write them where the committers expect them to live than
guess at a home for them.
What was verified, by hand, against a workflow with 61 saves and 21 runs:
- **The rebuild is exact.** All 61 versions rebuild, and each one matches
what
`GET /api/version/{wid}/{vid}` returns, compared one by one. That check
turned
up a real bug in the process: Texera sometimes writes the same inverse
patch
on several consecutive saves, typically `remove
/operators/0/operatorProperties/envName`,
which appears because opening a user function's property panel writes the
key
empty and the next save removes it. The first patch applies and the second
finds nothing to remove, which broke the chain and made everything earlier
look unrecoverable. Removing what is already gone leaves exactly the
document
the patch describes, so it is tolerated and counted. With that, one
workflow
went from 2 readable versions out of 56 to all 56. Worth flagging on its
own:
restoring an old version has the same exposure.
- **The comparison names the right edit** in two prepared cases: a filter
threshold moved from 100 to 400, and a grouping key moved from segment to
city where one branch changes and the other does not.
- **Browser check** of all five views, the step inspector, the ribbon, and
the
run-and-keep button (9 steps, 2,731 rows kept).
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
### The open design question
Comparing two runs row by row needs the rows, and Texera drops them thirty
seconds after the workflow goes idle. This draft keeps a copy as one JSON
file
per run on the service's own disk. That is the weakest part of it: it does
not
survive a restarted container, it does not respect the workflow's access
control, and nothing ever cleans it up. It should not survive review.
The rows do not need copying at all. When a run finishes they are already in
an
Iceberg table. They disappear because
`WorkflowService.clearExecutionResources`
clears them once `executionStateCleanUpInSecs` has passed. So keeping a run
for
comparison is a flag rather than a write: mark the execution as retained and
let
the cleanup skip it, the way it already skips per-user warehouses through
`WarehouseReadGuard.skipWhileDisabled`. A retention policy would be needed
alongside, since nothing would otherwise free the space.
That is likely worth having beyond this panel. "Keep this execution's
results"
is something users ask for on its own.
I have left it as it is rather than guessing at the shape of that flag, and
would rather agree it first.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01QqWWyuFAFLVswDYKXYyCwx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
