## What is the purpose of the change We currently have have three bundled/shaded filesystem connectors that build on top of Hadoop's classes. More will probably come, when we add more bundles file system connector libraries, for example for GCS. Each of them re-builds the shaded Hadoop module, including creating the relocated config, adapting native code loading, etc.
Similarly, there is a lot of code coming for the S3 connectors that will be shared between the Hadoop- and Presto-based implementations. This PR creates common bases projects for shaded Hadoop and common S3 functionality to be reused. ## Brief change log - Create the `flink-fs-hadoop-shaded` module and factors out the shaded Hadoop FS classes from the shaded S3 file systems into that module. - Bumps the Hadoop dependency to 3.1 to get access to newer connectors and better/later utilities. Adjusts the shading of the Hadoop configuration. - Creates an S3 base module `flink-s3-fs-base` as the common denominator for the Hadoop- and Presto-based implementations - Adjusts the Hadoop-based s3 connector to use the common denominator module - Adjust Presto-based S3 adapter to use the common denominator module - Consolidates shared classes for S3 in `flink-s3-fs-base` module - Upgrades the build script shading checks to new patterns. I put each change in a separate commit, for easier reviews. ## Verifying this change The test reworks and upgrades dependencies, it does not change functionality. The existing integration test cases and end-to-end tests still the existing functionality. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **yes** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **yes** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** [ Full content available at: https://github.com/apache/flink/pull/6714 ] This message was relayed via gitbox.apache.org for [email protected]
