devinjdangelo commented on PR #11060: URL: https://github.com/apache/datafusion/pull/11060#issuecomment-2186494278
Thank you for taking a look @berkaysynnada > We cannot merge FileType and FileFormatFactory due to dependency issues. That is correct. Logical planning cannot depend on FileFormatFactory or FileFormat to avoid a dependency on datafusion's execution engine during logical planning. > We cannot merge FileFormatFactory and FileFormat because of stateful format factories. Yes, `SessionState` stores a `HashMap<String, Arc<dyn FileFormatFactory>`. Each `FileFormatFactory` could be shared among multiple concurrent queries and must be able to produce many, differently configured, `Arc<dyn FileFormat>`s. If `SessionState` stored only a `Arc<dyn FileFormat>` it would have to have a `&mut self` method enabled by a `Mutex<>` and even with that it would possibly lead to unexpected behavior if multiple queries modified the format during execution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
