kz930 opened a new pull request, #8327: URL: https://github.com/apache/texera/pull/8327
### What changes were proposed in this PR? The standalone export added in #8326 claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in. `OpExecHarness` runs a `LogicalOp` the way the engine does, compiling it to a physical plan and driving the executor, but outside a workflow: its inputs are JSONL files rather than a live upstream, and its outputs are written back as files. `PyOpExecHarness` does the same for a Python operator, through the worker the engine itself uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, runs it, and keeps the script where it ran so a failing operator can be opened as generated rather than described second-hand. `TupleIO` is the format the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a plain number and the two paths appear to disagree over a difference neither operator made. Both runners produce files rather than verdicts. What to make of a difference between two files — which columns may be reordered, which values are close enough — is a separate concern and follows in a later change. This PR sits on top of #8326, which adds the `StandaloneCodeGenerator` trait the standalone runner reads. A cross-fork pull request cannot take another fork's branch as its base, so the diff below carries #8326's commit as well. The commit to review here is the second one, `test(workflow-compiling-service): run an operator both ways and compare the files`; the diff shrinks to it once #8326 merges. ### Any related issues, documentation, discussions? Part of #8325. Builds on #8326. ### How was this PR tested? `HarnessSpec` runs one operator through both paths and reads the files back. Distinct is the operator throughout, because what is under test is the harness rather than the operator: it takes one input, needs no configuration, and its answer on four rows with one repeat is short enough to state in full. The three cases are the round trip through `TupleIO`, including the sidecar schema; `OpExecHarness` producing one file per output port with the duplicate gone; and `StandaloneRunner` reaching the same three rows through the generated script. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
