JingsongLi opened a new pull request, #1435:
URL: https://github.com/apache/paimon-rust/pull/1435
### Purpose
Support Paimon/Spark SQL compatibility for maximum-partition lookup and
discrete approximate percentiles in SQLContext.
For example, `SELECT * FROM events WHERE dt = max_pt('events')` resolves the
current maximum first-level partition during planning, and
`approx_percentile(v, array(0.5, 0.4, 0.1), 100)` preserves Spark's percentile
ordering and input type.
### Brief change log
- Add `max_pt`, `sys.max_pt`, and catalog-qualified calls using the normal
session table resolution, snapshot/branch options, and scan authorization.
Compare partition values by their logical type and fold results into literals
for partition pruning. Reject grants with row filters or column masks before
reading raw partition values.
- Add `approx_percentile` and its `percentile_approx` alias using Spark's
mergeable Greenwald–Khanna summary, with constant accuracy and scalar/array
percentages, discrete rank selection, typed results, DISTINCT, FILTER, and
windows.
- Validate percentile parameters before optimizer elimination. Cover decimal
conversion and overflow, temporal/interval units, floating-point normalization,
partial merges, and bounded summary memory.
- Keep persistent views dynamic and reject metadata reads inside immutable
catalog SQL functions. Reuse the canonical Java-compatible partition formatting
helpers.
### Tests
- `cargo +1.95.0 test --locked -p paimon-datafusion --all-targets` with the
CI-required `RUST_MIN_STACK=16777216` — passed.
- `cargo +1.95.0 test --locked -p paimon-datafusion --lib
approx_percentile::quantile_summary` after the final Clippy repair — 6 passed.
- `cargo +1.98.0 clippy --locked -p paimon-datafusion --all-targets
--no-deps -- -D warnings` — passed.
- `cargo +1.98.0 fmt --all -- --check` and `git diff --check` — passed.
- Regression coverage includes restricted metadata grants through persistent
views, negative submicrosecond intervals, Float16 DISTINCT signed zeros and NaN
payloads, typed partition ordering, fresh snapshots, Spark percentile examples,
parameter errors, partial aggregation, and moving windows.
### API and Format
Adds SQL functions and public reexports of existing Java-compatible
UTF-8/double formatting helpers. No table storage or catalog metadata format
changes. Aggregate summary serialization is internal query state. Implicit SQL
casts retain DataFusion semantics; decimal percentile arrays expose nullable
elements to represent overflow safely.
### Documentation
Updated `docs/src/sql.md` with signatures, examples, compatibility
boundaries, authorization behavior, and resource characteristics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]