andygrove opened a new issue, #92: URL: https://github.com/apache/datafusion-spark/issues/92
This is step 1 of apache/datafusion#27342, done in this repository. `SparkOptions` (`datafusion.spark.*`) and `enable_ansi_mode` are defined in `datafusion-common`, so every new Spark option needs a DataFusion PR and then a DataFusion release. Several functions need new options (see #16): - `spark.sql.legacy.sizeOfNull` (#90) - `spark.sql.function.eltOutputAsString` (#91) - `spark.sql.legacy.javaCharsets` and `spark.sql.legacy.codingErrorAction` (#46) - `spark.sql.legacy.allowHashOnMapType` (#69) - `spark.sql.legacy.negativeIndexInArrayInsert` (#70) - a Spark version (#93) ### Proposal - Define `SparkOptions` here as a `ConfigExtension` with the prefix `spark`, so that keys mirror Spark's names (`spark.map_key_dedup_policy`, later `spark.sql.legacy.*`). - Register it in `SessionStateBuilderSpark::with_spark_features()`. - Have functions read it from `config_options.extensions`, falling back to the defaults when it isn't registered. This builds against both DataFusion 56 and `main`. - Move `map_from_arrays`, `map_from_entries` and `str_to_map` to the new `spark.map_key_dedup_policy` key. Core's `datafusion.spark.map_key_dedup_policy` stays on `branch-56` until apache/datafusion#27371 removes it for DataFusion 57. apache/datafusion#27371 implemented this in DataFusion before the crate moved here. Its author has been invited to port the `datafusion-spark` half to this repository. ### Notes for users of these options - A config extension reaches scalar functions through `ScalarFunctionArgs::config_options` at execution time. - It does not reach `return_field_from_args`, `coerce_types` or aggregate accumulators. Functions whose return type or input types depend on an option need it when they are constructed, refreshed in `with_updated_config`. - Comet builds `ScalarFunctionExpr` with `ConfigOptions::default()`, so it sees only the defaults. Every flag therefore also needs a direct way in, through a constructor argument or a public function. `enable_ansi_mode` stays in core until apache/datafusion#23929 decides whether core will read it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
