mccheah commented on issue #23915: SPARK-24252: Add v2 catalog plugin system. URL: https://github.com/apache/spark/pull/23915#issuecomment-468363882 Could config files be implementation-specific? For example I'd want my Data Source implementation to be given a YML file but another data source implementation wants a properties file. Deserialization of the file doesn't make sense to be done on the Spark side, but in the internal implementation of the data source. Thus I'd imagine one of the properties passed to the data source implementation would be the path to the configuration to deserialize, and we leave it up to the implementation to decide how to handle that path in `initialize`. So I don't think we need to make configuration files as an explicit part of the API here.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
