GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/4927
[FLINK-7778] [build] Shade ZooKeeper dependency
## What is the purpose of the change
This PR shades and relocates curator&zookeeper into flink-runtime to avoid
dependency conflicts, primarily with hadoop.
This PR is based on work done by @StephanEwen. I only resolved the
remaining issues:
* flink-mesos did not compile since it relies on the flink-runtime
zookeeper dependency, which was now relocated (the process of which is not
visible to flink-mesos). This was fixed by adding a relocation to flink-mesos
for zookeeper that must at all times be in sync. A cleaner solution would be to
create a dedicated flink-shaded-curator/zookeeper module that both
flink-runtime/mesos work against.
* flink-tests test execution failed since it couldn't find a relocated
curator test class. The relocation pattern for curator in flink-runtime was
also applied to curator-test; however the relocated version is not included in
the test jar and was thus not accessible to other modules. This was fixed by
adding an relocation exclusion for classes from `org.apache.curator.test`.
* flink-yarn test execution failed since it couldn't find the unshaded
guava Function class. Curator is indeed shading&relocating guava, unfortunately
not all of it. As such it was needed to re-add flink-shaded-curator-recipes
including the guava relocation. While it would have been possible to do the
relocation in flink-runtime I instead opted for the known working solution.
## Brief change log
* shade&relocate zookeeper and flink-shaded-curator recipes into
flink-runtime
* add a relocation for zookeeper to flink-mesos, in sync with flink-runtime
* remove flink-shaded-curator-test, since it is simply no longer needed
* add travis checks to ensure no unshaded curator/zk make it into flink-dist
* JobManagerHAJobGraphRecoveryITCase was moved to flink-runtime to resolve
a ClassCastException related to ZK or smth
## Verifying this change
I've only made sure that things compile and tests can be run successfully.
I have not tried this in an actual cluster setting due to time constraints.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink new_zkcur
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4927.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4927
----
commit b69c21c36443355be95246b3fda795df4fb6524c
Author: Stephan Ewen <[email protected]>
Date: 2017-10-13T20:14:09Z
[FLINK-7778] [build] Shade ZooKeeper dependency
Shading the ZooKeeper dependency makes sure that this specific version of
ZooKeeper is used by the Flink runtime module. The ZooKeeper version is
sensitive, because we depend on bug fixes in later ZooKeeper versions
for Flink's high availability.
This prevents situations where for example a set of added dependencies (for
example transtive dependencies of Hadoop) cause a different ZooKeeper
version
to be in the classpath and be loaded.
This commit also removes the 'flink-shaded-curator' module, which was
originally
created to shade guava within curator, but is now obsolete, because newer
versions of curator shade guava already.
commit af6dc28cfcd8856ea812d8443c464bb1dedb4315
Author: zentol <[email protected]>
Date: 2017-10-31T10:26:48Z
Resolve remaining issues
----
---
