andygrove opened a new pull request, #2102: URL: https://github.com/apache/datafusion-ballista/pull/2102
# Which issue does this PR close? Closes #2101. # Rationale for this change The `tpch` benchmark binary already writes a JSON summary with `--output`, but it is fragile as a basis for tracking performance across refs: - The summary is written only *after* the whole run, and a failing query aborts the loop (`?`/`.unwrap()` on `ctx.sql`/`collect`). One failure — or an OOM `SIGKILL` at a large scale factor — discards every result collected so far. - There is no in-tree way to compare two runs. # What changes are included in this PR? - **Resilient output.** The run summary is written after every query, atomically (temp file + rename), so a suite killed part way through keeps the results collected so far. - **Continue on error.** A query that fails is recorded with an `error` message and its completed iterations, and the run continues to the remaining queries. The process still exits non-zero if any query failed. `QueryRun` gains an `error: Option<String>` field (omitted on success); the run structs now also derive `Deserialize`. - **`compare` subcommand.** `tpch compare <baseline.json> <candidate.json>` diffs two result files query-by-query — fastest iteration per side, delta, delta %, suite totals, and row-count agreement (flags mismatches). - Refactors each run loop's per-query body into a helper so the error handling is shared and testable. - Unit tests for the min-of-iterations metric, the failure/serialization round-trip, and the comparison logic. README documents `--output`, the resilience behaviour, and `compare`. # Are there any user-facing changes? Yes, all additive: - `tpch compare` is a new subcommand. - The JSON summary is written incrementally and gains an optional per-query `error` field. - A benchmark run no longer aborts on the first failing query; it records the failure and continues, exiting non-zero at the end if any query failed. No breaking API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
