LadyForest opened a new pull request, #369: URL: https://github.com/apache/flink-table-store/pull/369
Currently, we rely on Flink's plugin mechanism to initialize Filesystem, which does not apply to other engines like Spark/Hive. This PR resolves - Invoke `Filesystem#initialize` to configure the target filesystem (including jar dependency and conf parameters) - For Spark, convert case insensitive conf to case sensitive conf (special handling for oss) - For Hive, let AK/endpoint can be configured via the `SET` command - Tested upon Spark3.3 & Hive3.1, found sorts of class conflicts then decided to shade oss to ease use. So introduce a submodule `flink-table-store-filesystem` to shade fs-oss/fs-s3(in the next pr) - Add readme Towards the E2E test, I'm unsure whether we can put AK info directly since the oss server is hard to mock. (maybe GitHub CI can support passing some secret values as env variable, then we can use credential class instead of AK text) For the next pr (which supports s3, there will be an E2E test using minio docker to serve like an s3 server) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
