andygrove commented on PR #27371: URL: https://github.com/apache/datafusion/pull/27371#issuecomment-6854726390
Thanks for working on this, @rizzabhdwaj. apache/datafusion#27337 removed `datafusion/spark` from this repository, so the crate's half of this PR now belongs in [apache/datafusion-spark](https://github.com/apache/datafusion-spark). Would you be willing to open it there? That would be step 1 of #27342: add `SparkOptions` as a `ConfigExtension` in `datafusion-spark`, register it in `with_spark_features()`, and have the map functions read it from `config_options.extensions`, falling back to the defaults when the extension isn't registered. apache/datafusion-spark builds against `branch-56`, where `datafusion.spark.map_key_dedup_policy` still exists, so the crate can't remove or rename anything in core yet. The removal from `datafusion-common`, `information_schema.slt` and the configs docs can stay in this PR for DataFusion 57, once the crate no longer reads the old key. There is some urgency: I'd like to add more options to that extension, including Spark-version behavior for the `CAST` port (apache/datafusion-spark#88). apache/datafusion#27260 and apache/datafusion#27273 are also waiting on it to add `spark.sql.legacy.*` flags for `encode`. @AnuragRaut08 offered to help on #27342 too, so please coordinate there if you'd rather hand it off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
