parthchandra commented on issue #2341: URL: https://github.com/apache/datafusion-comet/issues/2341#issuecomment-3633872291
> I see. Thank you [@parthchandra](https://github.com/parthchandra) for the extra context. > > > To support this we will end up having to implement a JNI layer to access the entire FileSystem API. > > Btw, OpenDAL has its own Java binding. Would it be useful for the OpenDAL community to build a Hadoop FileSystem implementation using that Java binding? I believe so, because we could potentially eliminate fs-hdfs. > > > Neither this, nor custom auth is a hard requirement for OpenDAL to be supported by Comet. > > Happy to know this. So if we want to move forward, what steps can we take? We have support for hdfs via fs-hdfs and OpenDAL but I'm not sure we have adequately defined what more we need from OpenDAL. I would think that at a minimum the tests defined in `ParquetReadFromFakeHadoopFsSuite` and `ParquetReadFromS3Suite` should pass for both the implementations. I also feel that support for custom auth and custom file systems is required in the longer term. Also, the implementations are behind a feature flag, but it might be easier to test/compare if we could choose the implementation based on a configuration flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
