JingsongLi opened a new pull request, #1311: URL: https://github.com/apache/paimon-rust/pull/1311
### Purpose Linked issue: close #1310 Complete the remaining DataFusion SQL compatibility gaps after #1308. Correct `IS [NOT] DISTINCT FROM` boolean precedence, accept binary byte-length/slice inputs, and add the missing split-index extension and Spark approximate percentile aggregate without changing existing Spark split functions. ### Brief change log - Backport the sqlparser precedence fix while retaining DataFusion's pinned AST dependency; preserve the binding when SQL is serialized and reparsed for DML, functions and views. - Support binary and dictionary inputs for byte lengths, `length` and `substring`, including FixedSizeBinary. Keep Spark INT byte-length results and existing substring argument names. - Add literal, zero-based `split_index` with Flink-compatible NULL, empty-input and Java-whitespace behavior. - Port Spark's mergeable Greenwald–Khanna summaries for `approx_percentile` / `percentile_approx`; support scalar/array percentages, accuracy, numeric/decimal/date/timestamp/interval results, DISTINCT, FILTER and windows. - Validate foldable percentile parameters before empty-input optimization and before persisting views. Keep application-defined overrides intact. Use actual frame ranges for moving windows, including disjoint RANGE frames; growing windows retain incremental summaries. Moving windows recompute each frame. ### Tests - `cargo test --locked -p paimon-datafusion --all-targets`: all targets passed, including new SQL regressions and compressed, multi-batch/two-partition quantile states. - Spark 4.0.3 on Java 17 used as a differential oracle for percentile values and parameter/modifier rejection, decimal rounding, partial summaries and empty window frames. Quantile summary port follows Spark 4.1.2. - `cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- -D warnings` and `cargo +1.98.0 fmt --all -- --check`: passed. - Multiple independent review passes covering SQL semantics, boundaries and API/tests; findings fixed with regressions. ### API and Format Adds SQL functions and corrects SQL expression behavior. No new dependencies, public Rust API changes or Paimon storage-format changes. Quantile serialization is internal aggregate execution state. ### Documentation Updated the SQL guide and DataFusion integration README with the supported contracts, type/rounding behavior and moving-window cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
