Repository: spark Updated Branches: refs/heads/branch-1.0 14515b416 -> e1852812f
SPARK-1802. Audit dependency graph when Spark is built with -Phive This initial commit resolves the conflicts in the Hive profiles as noted in https://issues.apache.org/jira/browse/SPARK-1802 . Most of the fix was to note that Hive drags in Avro, and so if the hive module depends on Spark's version of the `avro-*` dependencies, it will pull in our exclusions as needed too. But I found we need to copy some exclusions between the two Avro dependencies to get this right. And then had to squash some commons-logging intrusions. This turned up another annoying find, that `hive-exec` is basically an "assembly" artifact that _also_ packages all of its transitive dependencies. This means the final assembly shows lots of collisions between itself and its dependencies, and even other project dependencies. I have a TODO to examine whether that is going to be a deal-breaker or not. In the meantime I'm going to tack on a second commit to this PR that will also fix some similar, last collisions in the YARN profile. Author: Sean Owen <[email protected]> Closes #744 from srowen/SPARK-1802 and squashes the following commits: a856604 [Sean Owen] Resolve JAR version conflicts specific to Hive profile (cherry picked from commit 8586bf564fe010dfc19ef26874472a6f85e355fb) Signed-off-by: Patrick Wendell <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e1852812 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e1852812 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e1852812 Branch: refs/heads/branch-1.0 Commit: e1852812ff9a8b0b9eb28990db0ad754ab80b08b Parents: 14515b4 Author: Sean Owen <[email protected]> Authored: Mon May 12 14:17:25 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Mon May 12 14:17:34 2014 -0700 ---------------------------------------------------------------------- pom.xml | 16 ++++++++++++++++ sql/hive/pom.xml | 21 +++++++++++++++++++++ 2 files changed, 37 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/e1852812/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index e8157c3..e366ad6 100644 --- a/pom.xml +++ b/pom.xml @@ -535,6 +535,22 @@ <groupId>io.netty</groupId> <artifactId>netty</artifactId> </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>jetty</artifactId> + </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>jetty-util</artifactId> + </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>servlet-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.velocity</groupId> + <artifactId>velocity</artifactId> + </exclusion> </exclusions> </dependency> <!-- See SPARK-1556 for info on this dependency: --> http://git-wip-us.apache.org/repos/asf/spark/blob/e1852812/sql/hive/pom.xml ---------------------------------------------------------------------- diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml index 1f05d0b..4fd3cb0 100644 --- a/sql/hive/pom.xml +++ b/sql/hive/pom.xml @@ -51,6 +51,12 @@ <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>${hive.version}</version> + <exclusions> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> <groupId>org.codehaus.jackson</groupId> @@ -60,6 +66,21 @@ <groupId>org.apache.hive</groupId> <artifactId>hive-serde</artifactId> <version>${hive.version}</version> + <exclusions> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging-api</artifactId> + </exclusion> + </exclusions> + </dependency> + <!-- hive-serde already depends on avro, but this brings in customized config of avro deps from parent --> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> </dependency> <dependency> <groupId>org.scalatest</groupId>
