devinjdangelo commented on PR #8667: URL: https://github.com/apache/arrow-datafusion/pull/8667#issuecomment-1879836968
I've been thinking about this more, and I think we need a similar interface to how we register a custom `TableProvider` at the `SessionContext` level. The idea being we'd have a `SessionContext::register_file_type` which accepts an `Arc<dyn FileType>` and stores it in a `DashMap<String, Arc<dyn FileType>>`. At planning and execution time, all the places we are doing a `match FileType` currently, would be replaced with a lookup on the map of registered FileTypes + a method call on the trait object. The above is based on my understanding having studied a bit how we handle external `TableProviders`. @alamb, does the above sound reasonable to you? My main uncertainty to the above is how we would handle serialization. Perhaps the `FileType` trait could have a method which returns a `Vec<u8>` based serialization of itself. I'm not sure how that would work on the receiving end exactly... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
