peterxcli opened a new pull request, #6595: URL: https://github.com/apache/datafusion-comet/pull/6595
## Which issue does this PR close? Part of #5424. ## Rationale for this change Evaluate `variant_get` and `try_variant_get` natively for foldable paths and scalar outputs. This can proceed independently of #6443 with `spark.sql.variant.pushVariantIntoScan=false`. ## What changes are included in this PR? Reuse Spark's path parser and Comet's casts for extraction over whole-value Variant input, preserving missing-value and strict/try behavior. Invalid paths retain Spark's error handling. Strict cast failures replay the failed row in Spark to preserve its exact error parameters. The `implement-comet-expression` workflow guided the implementation and compatibility tests. Boolean, integer, floating-point, and binary outputs are enabled. String and TIME outputs retain fallback; decimal, date, and timestamp outputs require `allowIncompatible` because of known cast differences. #5424 stays open. Selective Parquet reads remain under #3983. ## How are these changes tested? All 12 native tests and 20 Spark 4.1 SQL/Variant tests passed, covering both dictionary encodings, paths, nulls, exact errors, and canonical and shredded scans. Two reproduced decimal/date incompatibilities remain ignored under #5424. Full workspace Clippy, formatting, and Scalastyle passed. Broader CI is pending. Please apply `run-all-spark-profiles` and `run-spark-4.1-tests`; I do not have label permissions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
