andygrove opened a new issue, #7399: URL: https://github.com/apache/datafusion-comet/issues/7399
## What is the problem the feature request solves? Part of #7397. Spark 4.1+ provides KLL quantile sketches in bigint, float and double variants: `kll_sketch_agg_*`, `kll_merge_agg_*`, `kll_sketch_merge_*`, `kll_sketch_get_quantile_*`, `kll_sketch_get_rank_*`, `kll_sketch_get_n_*` and `kll_sketch_to_string_*`. None of them run natively in Comet. The crate's `kll` feature is new in `datasketches` 0.6.0. ## Describe the potential solution 1. Score the family with `suggest-native-expression` first. 2. Audit the crate's `kll` feature against DataSketches-Java 6.2.0 using committed fixtures: serialized sketches of each item type, merge results, quantile and rank answers, and the `to_string` output. KLL compaction is randomized in Java, so work out which results are deterministic enough to compare exactly and which can only be compared within the rank error bound. The crate's `KllFloat` adapter rejects NaN, so check how Spark treats NaN and infinities. 3. Implement the serde and native functions once the audit shows the serialized form is read in both directions, opt-in until then. 4. Add SQL file tests, including a Spark-produced sketch column read by Comet and the reverse. ## Additional context The new crate feature is young (first release in 0.6.0), so read its open issues before relying on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
