andygrove opened a new pull request, #326:
URL: https://github.com/apache/datafusion-spark/pull/326

   ## Which issue does this PR close?
   
   Part of #56. It covers the `datafusion-cli --spark` references in the 
"Update the imported docs" item, since those READMEs now point to the new CLI.
   
   ## Rationale for this change
   
   apache/datafusion#27337 removed `datafusion-cli --spark`, which enabled the 
Spark SQL dialect and registered the Spark-compatible functions. Since then 
there is no way to try the functions interactively, and 
`crates/spark/README.md` and `crates/sqllogictest/test_files/README.md` still 
tell people to run `datafusion-cli --spark`.
   
   ## What changes are included in this PR?
   
   - **`crates/cli`, a new `datafusion-spark-cli` binary.** It uses the 
`datafusion-cli` library for the REPL, for running files and commands, for 
output formats, and for querying files by path or URL (`SELECT * FROM 
'data.parquet'`). Its `main.rs` is about 200 lines, following 
`datafusion-cli`'s.
     - **Always Spark:** the Spark SQL dialect and the Spark functions are 
always enabled. The Spark functions override DataFusion's functions of the same 
name.
     - **ANSI mode on by default,** as in Spark 4 (`spark.sql.ansi.enabled = 
true`). `--ansi false` gives Spark's legacy behavior. The startup banner says 
which mode is on.
     - **Options kept from `datafusion-cli`:** `-c`, `-f`, `-r`, `-p`, `-b`, 
`--format`, `-q`, `--maxrows` and `--color`. The memory and disk tuning options 
are left out for now.
   - **Not published:** `publish = false`.
   - **Not a default workspace member:** `datafusion-cli` brings in the object 
store and AWS dependencies, so a plain `cargo build` or `cargo test` at the 
root skips this crate. The clippy job uses `--workspace`, so it still lints the 
crate.
   - **New workflow, `.github/workflows/cli.yml`.** It builds the CLI and runs 
a smoke test:
     - `sha2` returns Spark's result.
     - `abs(cast(-2147483648 AS int))` fails by default, because ANSI mode is 
on.
     - The same query returns `-2147483648` with `--ansi false`.
   - **Docs:** `crates/spark/README.md`, `test_files/README.md`, the root 
`README.md` and `AGENTS.md` describe the new CLI.
   
   `datafusion-cli` on DataFusion's `branch-56` still depends on that branch's 
own `datafusion-spark`, so the build compiles that copy too. Only this 
repository's functions are registered. The extra copy goes away with DataFusion 
57, where `datafusion-cli` no longer depends on `datafusion-spark`.
   
   ## Are these changes tested?
   
   - The smoke test's three queries pass locally.
   - I also tried the REPL reading from stdin, a backquoted identifier (Spark 
dialect), `--help` and the banner.
   - `cargo clippy --all-targets --all-features --workspace -- -D warnings` is 
clean.
   - `cargo test -p datafusion-spark` and `cargo test -p 
datafusion-spark-sqllogictest` still pass.
   - `actionlint` reports no problems with the new workflow.
   
   ## Are there any user-facing changes?
   
   There is a new, unpublished `datafusion-spark-cli` binary. You run it with 
`cargo run -p datafusion-spark-cli`.
   
   _This PR was written by an LLM (Claude) and reviewed before opening._
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to