andygrove opened a new pull request, #2814: URL: https://github.com/apache/datafusion-ballista/pull/2814
# Which issue does this PR close? N/A # Rationale for this change The top-level README has fallen behind the project. The architecture diagram shows neither Arrow Flight nor how results get back to clients. Features such as Arrow Flight SQL and ADBC aren't mentioned at all, and the README has had no performance numbers since the previous ones went stale and were removed in #2015. A reader should be able to see what Ballista offers, how mature it is and how fast it is. To keep the performance numbers from going stale again, benchmark results now live in the repository as JSON, and the docs table and README charts are generated from them. # What changes are included in this PR? **README** - A Performance section with two charts comparing Ballista with Apache Spark: total runtime and time per query. The text below them carries the caveats from the benchmarking guide and links to it for the full results. - A Features section covering query interfaces, execution, operations and extensibility. - A paragraph positioning Ballista as an alternative to Spark, alongside the existing framing of making DataFusion applications distributed. - A Project Status section describing releases, adopters and how changes are tested. The note about the gap between DataFusion and Ballista moves here from the top of the page. - The two code samples become one diff, the Cargo feature tables are collapsed and gain the missing `flight-sql` row, and there are new badges and a logo that is visible in dark mode. **Architecture diagram** - A new diagram showing clients, the scheduler, executors, monitoring and storage, with control and data paths drawn separately. It replaces the existing SVG in place, because the published crate's README links to that path. - The architecture guide's text now matches it: tasks are pushed by default, and jobs are submitted as logical, Substrait or physical plans rather than SQL strings. **Benchmark results as JSON** - `benchmarks/cluster/published/tpch-sf1000/` holds one file per column of the results table, in the format `run.py` writes plus a `label`. A recorded `total` keeps the published totals, because the per-query means in these files are rounded. - `benchmarks/cluster/publish.py` regenerates the results table on the benchmarking page, the two README charts and the speedups quoted in the README. It shares its table logic with `compare.py`. - Regenerating reproduces the existing results table byte for byte. - `test_publish.py` covers the table and charts, and fails if a committed output no longer matches the JSON. Run it with `python3 -m unittest -v test_publish` from `benchmarks/cluster`. The README's fault tolerance claim rests on the chaos suite, which kills executors mid-query with AQE on and off. The tuning guide and #1986 still list executor failure under AQE as a limitation, and they look out of date. # Are there any user-facing changes? Documentation only, plus a new script for maintainers. No code, API or wire-format changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
