devinjdangelo commented on issue #8421: URL: https://github.com/apache/arrow-datafusion/issues/8421#issuecomment-1840729524
The reason for the strict checking is to help identify typos or logical errors easily in SQL queries rather than silently ignoring invalid options. Given that datafusion-cli is using the same data structure to store additional options, there are a few options: - Add the additional datafusion-cli valid options to datafusion proper so they can be silently ignored rather than error - Have datafusion-cli remove any valid datafusion-cli specific options in the create external table statement before executing the statement - Downgrade the configuration error in datafusion to a warning message so execution doesn't fail I favor one of the first two options, but the third may be the best if we expect this problem to crop up beyond datafusion-cli. The question would be if it is likely for many downstream projects to inject additional options passed to a ListingTable or CopyTo statement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
