JingsongLi opened a new pull request, #1308:
URL: https://github.com/apache/paimon-rust/pull/1308

   ### Purpose
   
   Closes #1307.
   
   SQLContext duplicates several functions already maintained by DataFusion's 
Spark package and lacks other Spark function families. Enable the matching 
upstream packages and expression planner by default, delegate overlapping 
implementations, and retain Paimon-specific SQL extensions.
   
   ### Brief change log
   
   - Register `datafusion-spark` 54.0.0 after default DataFusion features. 
Application registrations made afterward retain precedence, and qualified 
catalog functions retain REST resolution.
   - Remove local algorithms for mod, date_sub, sha2, Base64, hex/unhex, size, 
hour, collect_list and two-argument date_diff. Keep to_base64 as an upstream 
Base64 alias; delegate two-argument date_diff to Spark and preserve the 
three-argument Paimon overload.
   - Normalize supported Arrow dictionary/NULL inputs and ordinary SQL integer 
arguments before invoking upstream functions. Forward aliases, configuration, 
simplification and return fields.
   - Reuse upstream collection and sum accumulators, handling collection 
DISTINCT, empty-window defaults and typed NULL sum defaults. Pre-filter 
evaluated AVG batch arrays and group IDs to repair upstream 54's grouped FILTER 
behavior while retaining its vectorized accumulator, state merging and 
partial-aggregation bypass.
   - Update the dependency lock and reports; retain existing 
split/locate/typeof, regex extraction, Variant conversion, selection aggregates 
and metadata/schema operations.
   
   ### Tests
   
   - Added ten Spark integration regression groups and one persistent REST 
function/view regression. The initial three integration groups failed against 
the merged baseline.
   - Cover function families, ANSI remainder behavior, native input 
normalization and aliases, application overrides, DISTINCT, NULL/empty/overflow 
handling, grouped FILTER, empty window frames, multiple batches/partitions and 
forced partial-aggregation bypass.
   - Reproduced grouped AVG FILTER errors, incorrect empty collection frames 
and untyped TRY_SUM window NULLs before fixing them. Compare unsupported AVG 
suffix windows with an unmodified upstream Spark session.
   - Adapt existing compatibility and combined-feature assertions to the 
selected upstream semantics.
   - Passed `cargo test --locked -p paimon-datafusion --all-targets -- 
--test-threads=4` with UTC and a 16 MiB test-thread stack, including 377 
library tests and all integration targets.
   - Passed the Vortex table integration test with `--features vortex --test 
vortex_tables`.
   - Passed Rust 1.98 workspace Clippy with `--locked --all-targets --workspace 
--features vortex -- -D warnings`, Python binding all-target compilation, 
formatting and diff checks.
   - Passed dependency-report verification, cargo-deny advisory/license checks 
and release-license validation for all 21 generated binary legal outputs.
   - Completed independent reviews of registration/REST semantics, aggregate 
boundaries and API/tests. Fixed the identified issues and revalidated the final 
implementation.
   
   ### API and Format
   
   SQLContext defaults now use upstream Spark function semantics for 
overlapping names. In particular: concat propagates NULL, size uses legacy 
NULL-to-minus-one behavior, mod/pmod return NULL on zero by default and error 
in ANSI mode, two-argument date_diff returns INT, and hour expects timestamps. 
Upstream 54 AVG returns DOUBLE. DISTINCT and retracting-window support follow 
the documented version limits, with unsupported plans reporting errors.
   
   Paimon-specific extensions remain available. There are no Paimon 
storage-format or public Rust API changes.
   
   ### Documentation
   
   Document the upstream package, registry precedence, migration differences, 
input adapters and version-specific limits in the SQL guide and integration 
README.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to