kosiew commented on code in PR #57: URL: https://github.com/apache/datafusion-spark/pull/57#discussion_r4582599818
########## crates/spark/src/lib.rs: ########## @@ -11,22 +11,67 @@ #![deny(clippy::clone_on_ref_ptr)] #![cfg_attr(test, allow(clippy::needless_pass_by_value))] -//! Spark Expression packages for [DataFusion]. +//! [Apache Spark]-compatible expressions for [Apache DataFusion]. //! -//! This crate contains a collection of various Spark function packages for DataFusion, -//! implemented using the extension API. +//! DataFusion's built-in functions follow PostgreSQL / DuckDB conventions. When +//! the same function name exists in Apache Spark, the behavior often differs: +//! NULL handling, integer overflow, string indexing (1-based, negative +//! indices), rounding modes, date/time handling, error behavior in ANSI and +//! non-ANSI mode, and so on. This crate provides implementations of Spark +//! functions that follow Spark's semantics, so that engines built on DataFusion +//! (for example [Apache DataFusion Comet]) and applications migrating SQL from +//! Spark get the results Spark would produce. //! -//! [DataFusion]: https://crates.io/crates/datafusion +//! The crate depends only on the DataFusion building blocks (`datafusion-expr`, +//! `datafusion-common`, ...), not on the full `datafusion` crate, unless the +//! optional `core` feature is enabled. //! +//! # Getting started +//! +//! ```toml +//! [dependencies] +//! datafusion = "X" +//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark mode on +//! datafusion-spark = { version = "X", features = ["core"] } +//! ``` //! //! # Available Function Packages -//! See the list of [modules](#modules) in this crate for available packages. +//! +//! Functions are grouped into packages by category (for example [`function::string`], +//! [`function::datetime`], [`function::math`], [`function::hash`]). See the list of +//! [modules](#modules) in this crate for the available packages. Each package Review Comment: Consider linking "available packages" directly to [`function`] instead of `#modules`. The package descriptions are under `function/index.html`, so this would save readers a navigation step. The existing link works; this is only a discoverability suggestion. ########## crates/spark/src/lib.rs: ########## @@ -11,22 +11,67 @@ #![deny(clippy::clone_on_ref_ptr)] #![cfg_attr(test, allow(clippy::needless_pass_by_value))] -//! Spark Expression packages for [DataFusion]. +//! [Apache Spark]-compatible expressions for [Apache DataFusion]. //! -//! This crate contains a collection of various Spark function packages for DataFusion, -//! implemented using the extension API. +//! DataFusion's built-in functions follow PostgreSQL / DuckDB conventions. When +//! the same function name exists in Apache Spark, the behavior often differs: +//! NULL handling, integer overflow, string indexing (1-based, negative +//! indices), rounding modes, date/time handling, error behavior in ANSI and +//! non-ANSI mode, and so on. This crate provides implementations of Spark +//! functions that follow Spark's semantics, so that engines built on DataFusion +//! (for example [Apache DataFusion Comet]) and applications migrating SQL from +//! Spark get the results Spark would produce. //! -//! [DataFusion]: https://crates.io/crates/datafusion +//! The crate depends only on the DataFusion building blocks (`datafusion-expr`, +//! `datafusion-common`, ...), not on the full `datafusion` crate, unless the +//! optional `core` feature is enabled. //! +//! # Getting started +//! +//! ```toml +//! [dependencies] +//! datafusion = "X" +//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark mode on +//! datafusion-spark = { version = "X", features = ["core"] } +//! ``` //! //! # Available Function Packages -//! See the list of [modules](#modules) in this crate for available packages. +//! +//! Functions are grouped into packages by category (for example [`function::string`], +//! [`function::datetime`], [`function::math`], [`function::hash`]). See the list of +//! [modules](#modules) in this crate for the available packages. Each package +//! provides: +//! +//! * a `functions()` method returning all of its UDFs for registration +//! * an `expr_fn` module with helpers to build [`Expr`]s from Rust +//! * one accessor per function (for example `function::hash::sha2()`) returning the UDF +//! +//! Not every Spark function is implemented yet; coverage is growing. The +//! [`datafusion-spark` epic] tracks progress. +//! +//! # Semantics and caveats +//! +//! * **Registration overrides built-ins.** Spark functions are registered under +//! their Spark names. If DataFusion already has a function with that name, +//! the Spark one replaces it for the session. Register Spark functions after +//! DataFusion's defaults, and only when you want Spark behavior. +//! * **Input types.** Only types available in Spark are supported (no unsigned +//! integers, `Float16`, `Decimal64`, ...). +//! * **ANSI mode.** Functions whose Spark behavior depends on ANSI mode read +//! `datafusion.execution.enable_ansi_mode` from the session configuration. +//! * **Type inference.** DataFusion and Spark do not infer literal types in the +//! same way, so add explicit casts (`CAST(1 AS INT)`) when exact types matter. Review Comment: Consider clarifying that `with_spark_features()` does not select the Spark SQL parser dialect. Spark-specific syntax such as `CAST(1 AS LONG)` requires setting `config.options_mut().sql_parser.dialect = Dialect::Spark` and passing `config` to `SessionStateBuilder::with_config(config)`; the generic dialect rejects `LONG`. Not all supported queries require the Spark dialect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
