andygrove opened a new issue, #94: URL: https://github.com/apache/datafusion-spark/issues/94
These functions already match Spark's default for a setting, but can't follow it when a user changes the setting. Each one needs a flag (see #16), set from `SparkOptions` once it moves into this crate (#92). These are low priority. | Function | Spark setting (default) | Effect when the setting is changed | Decided at | |---|---|---|---| | `format_string` | [`spark.sql.legacy.allowZeroIndexInFormatString`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L2055-L2057) (false) | `%0$s` is accepted instead of rejected | construction | | `concat` | [`spark.sql.function.concatBinaryAsString`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala#L425-L438) (false) | all-binary input returns `STRING` instead of `BINARY` | type coercion | | `array()` with no arguments | `spark.sql.legacy.createEmptyCollectionUsingStringType` (false) | returns `array<string>` instead of `array<void>` | return type | I checked the current results of `concat(X'61', X'62')` (`Binary`) and `array()` (`List(Null)`) through the sqllogictest runner. The `format_string` result comes from reading the code: `crates/spark/src/function/string/format_string.rs` rejects argument index 0. Because all three change the return type or the accepted input, the flag has to be set when the function is constructed, and refreshed in `with_updated_config` in a DataFusion session. `elt` has the same kind of setting, `spark.sql.function.eltOutputAsString`, but its default is not handled correctly either. That is tracked in #91. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
