andygrove opened a new issue, #69: URL: https://github.com/apache/datafusion-spark/issues/69
Follow-up to #68 (adds `hash`). Not started; to be done after #68 merges. ## Problem Spark's `HashExpression.checkInputDataTypes` (`sql/catalyst/.../expressions/hash.scala`, 4.2.0) rejects some inputs for `hash` and `xxhash64`: - Any argument whose type contains a map, including maps nested in arrays, structs or other maps, fails analysis with `HASH_MAP_TYPE`. The exception is the internal flag `spark.sql.legacy.allowHashOnMapType=true`. - Any argument whose type is a variant fails with `HASH_VARIANT_TYPE`. - Zero arguments fails with a wrong-number-of-arguments error. Here both functions use `Signature::variadic_any` and the shared macros in `hash/utils.rs` hash maps, so `hash(map(...))` and `xxhash64(map(...))` succeed instead of failing. Unsupported types only fail at execution (`utils.rs`, "Unsupported data type in hasher"). ## Proposed work - Add a recursive "contains a map" type check in `hash/utils.rs`, used from `return_field_from_args` of both functions, raising a Spark-style `HASH_MAP_TYPE` error. - Decide how to handle variant extension types. - Decide whether to support `spark.sql.legacy.allowHashOnMapType`. This needs a config mechanism, which the crate doesn't appear to have. Comet keeps map hashing because it supports the legacy flag. - Use a Spark-style error for the zero-argument case in `hash` (added in #68 as a plain planning error) and add the same check to `xxhash64`, which currently accepts zero arguments. - Add slt cases: a top-level map, a map nested in an array and in a struct, and zero arguments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
