alamb opened a new issue, #9575: URL: https://github.com/apache/arrow-datafusion/issues/9575
### Is your feature request related to a problem or challenge? Before https://github.com/apache/arrow-datafusion/pull/9382, DataFusion write configuration was inconsistent and different than the main configuration system and https://github.com/apache/arrow-datafusion/pull/9382 rationalized it ❤️ However, it also broke compatibility with old configuration and introduced some redundancy that can be annoying, as @devinjdangelo noted https://github.com/apache/arrow-datafusion/pull/9382#pullrequestreview-1908872706 For example, to specify the compression type when writing parquet, after https://github.com/apache/arrow-datafusion/pull/9382 one needs to repeat 'parquet' to namespace the configuration options (`format parquet` and `'parquet.compression'`) ```sql DataFusion CLI v36.0.0 ❯ COPY (values (1)) TO 'test_files/scratch/copy/table/' (format parquet, 'compression' 'zstd(10)'); Invalid or Unsupported Configuration: could not find config namespace for key "compression" ❯ COPY (values (1)) TO 'test_files/scratch/copy/table/' (format parquet, 'parquet.compression' 'zstd(10)'); +-------+ | count | +-------+ | 1 | +-------+ 1 row in set. Query took 0.024 seconds. ``` ### Describe the solution you'd like I would like to introduce some backwards compatibility so when I write ```sql COPY source_table TO 'test_files/scratch/copy/table/' (format parquet, 'compression' 'zstd(10)'); ``` It means the same as ```sql COPY source_table TO 'test_files/scratch/copy/table/' (format parquet, 'parquet.compression' 'zstd(10)'); ``` ### Describe alternatives you've considered I suggest we add some sort of special case code that tried to parse the option as normal, and if that failed, try with the different formats / prefixes For example, if `compression` wasn't a valid option, ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
