JingsongLi opened a new pull request, #1308: URL: https://github.com/apache/paimon-rust/pull/1308
### Purpose Closes #1307. SQLContext duplicates several functions already maintained by DataFusion's Spark package and lacks other Spark function families. Enable the matching upstream packages and expression planner by default, delegate overlapping implementations, and retain Paimon-specific SQL extensions. ### Brief change log - Register `datafusion-spark` 54.0.0 after default DataFusion features. Application registrations made afterward retain precedence, and qualified catalog functions retain REST resolution. - Remove local algorithms for mod, date_sub, sha2, Base64, hex/unhex, size, hour, collect_list and two-argument date_diff. Keep to_base64 as an upstream Base64 alias; delegate two-argument date_diff to Spark and preserve the three-argument Paimon overload. - Normalize supported Arrow dictionary/NULL inputs and ordinary SQL integer arguments before invoking upstream functions. Forward aliases, configuration, simplification and return fields. - Reuse upstream collection and sum accumulators, handling collection DISTINCT, empty-window defaults and typed NULL sum defaults. Pre-filter evaluated AVG batch arrays and group IDs to repair upstream 54's grouped FILTER behavior while retaining its vectorized accumulator, state merging and partial-aggregation bypass. - Update the dependency lock and reports; retain existing split/locate/typeof, regex extraction, Variant conversion, selection aggregates and metadata/schema operations. ### Tests - Added ten Spark integration regression groups and one persistent REST function/view regression. The initial three integration groups failed against the merged baseline. - Cover function families, ANSI remainder behavior, native input normalization and aliases, application overrides, DISTINCT, NULL/empty/overflow handling, grouped FILTER, empty window frames, multiple batches/partitions and forced partial-aggregation bypass. - Reproduced grouped AVG FILTER errors, incorrect empty collection frames and untyped TRY_SUM window NULLs before fixing them. Compare unsupported AVG suffix windows with an unmodified upstream Spark session. - Adapt existing compatibility and combined-feature assertions to the selected upstream semantics. - Passed `cargo test --locked -p paimon-datafusion --all-targets -- --test-threads=4` with UTC and a 16 MiB test-thread stack, including 377 library tests and all integration targets. - Passed the Vortex table integration test with `--features vortex --test vortex_tables`. - Passed Rust 1.98 workspace Clippy with `--locked --all-targets --workspace --features vortex -- -D warnings`, Python binding all-target compilation, formatting and diff checks. - Passed dependency-report verification, cargo-deny advisory/license checks and release-license validation for all 21 generated binary legal outputs. - Completed independent reviews of registration/REST semantics, aggregate boundaries and API/tests. Fixed the identified issues and revalidated the final implementation. ### API and Format SQLContext defaults now use upstream Spark function semantics for overlapping names. In particular: concat propagates NULL, size uses legacy NULL-to-minus-one behavior, mod/pmod return NULL on zero by default and error in ANSI mode, two-argument date_diff returns INT, and hour expects timestamps. Upstream 54 AVG returns DOUBLE. DISTINCT and retracting-window support follow the documented version limits, with unsupported plans reporting errors. Paimon-specific extensions remain available. There are no Paimon storage-format or public Rust API changes. ### Documentation Document the upstream package, registry precedence, migration differences, input adapters and version-specific limits in the SQL guide and integration README. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
