andygrove opened a new issue, #7401: URL: https://github.com/apache/datafusion-comet/issues/7401
## What is the problem the feature request solves? Part of #7397. Spark 4.2 adds Tuple sketch functions: `tuple_sketch_agg`, `tuple_union_agg`, `tuple_intersection_agg`, the `tuple_difference` / `tuple_union` / `tuple_intersection` scalars, and `tuple_sketch_estimate`, `tuple_sketch_summary` and `tuple_sketch_theta`, each in integer and double summary variants. None of them run natively in Comet. The crate's `tuple` feature arrived in `datasketches` 0.4.0. ## Describe the potential solution 1. Score the family with `suggest-native-expression` first. This is the lowest priority of the four, because it only exists in the newest Spark version Comet supports. 2. Do the theta work first, since a tuple sketch is a theta sketch with a summary attached and shares its hashing and set-operation behavior. 3. Audit the crate's `tuple` feature against DataSketches-Java 6.2.0 using committed fixtures. The summary types and the policies that combine summaries on union and intersection need to match Spark's exactly, and the serialized summary format has to be readable in both directions. 4. Implement, test and document as for the other families, with a fallback on Spark versions before 4.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
