andygrove opened a new issue, #90: URL: https://github.com/apache/datafusion-spark/issues/90
`size` returns -1 for a NULL array or map, and declares its result non-nullable (`crates/spark/src/function/collection/size.rs`). It ignores both ANSI mode and `spark.sql.legacy.sizeOfNull`. In Spark, `size` returns -1 for NULL only when `spark.sql.legacy.sizeOfNull` is true *and* ANSI mode is off. Otherwise it returns NULL and the result is nullable: - [`SQLConf.legacySizeOfNull`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L7595-L7598) is `getConf(LEGACY_SIZE_OF_NULL) && !getConf(ANSI_ENABLED)`. [`spark.sql.legacy.sizeOfNull`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4987-L4993) defaults to true. - [`Size`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L118-L146) receives `legacySizeOfNull` as a constructor argument. It uses it for both `nullable` and `eval`. ANSI mode is on by default in Spark 4.0 and later, so Spark 4's default result for `size(NULL)` is NULL. The crate returns -1 even with `enable_ansi_mode = true`: ```sql set datafusion.execution.enable_ansi_mode = true; SELECT size(CAST(NULL AS ARRAY<INT>)); -- datafusion-spark: -1 -- Spark 4.x default: NULL ``` The `legacySizeOfNull` expression is the same in Spark 3.4.3, 3.5.8, 4.0.1 and 4.1.1, which I read in source; I did not run these on PySpark. Spark's `cardinality` is the same expression. ### Fix The flag decides the result's nullability, and `return_field_from_args` has no access to the config. So `size` should take `legacy_size_of_null` when it is constructed, as Spark's `Size` does. In a DataFusion session it can be set in `with_updated_config` from `enable_ansi_mode` and a `spark.sql.legacy.sizeOfNull` option. That option needs `SparkOptions` to move into this crate. Part of the Spark-version and legacy-setting work in #16. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
