Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/17665#discussion_r112337820
--- Diff: core/pom.xml ---
@@ -357,6 +357,29 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-crypto</artifactId>
</dependency>
+
+ <!--
+ Testing Hive reflection needs hive on the test classpath only.
+ It doesn't need the spark hive modules, so the -Phive flag is not
checked.
+ -->
+ <dependency>
+ <groupId>${hive.group}</groupId>
+ <artifactId>hive-exec</artifactId>
--- End diff --
I moved these from YARN's pom.xml, but had to remove the "test" scope.
Maven builds work fine with the "test" scope, but for some reason sbt doesn't.
`./build/sbt compile` doesn't recognize that dependencies in the "test" scope
aren't transitive, so it tries to pull in the `pentaho` dependency via
`calcite-core`, which was deleted from maven central at some point, which is
why we exclude it in the `spark-parent` pom:
https://github.com/apache/spark/blob/master/pom.xml#L1776
What I can't understand is how this wasn't a problem when these
dependencies were in YARN. For some reason `sbt dependencyTree` on `master`
shows that `calcite-core` is excluded from the `hive-exec` dependency tree, but
the same command on this PR tries to resolve `calcite-core`.
Let me know if you have any ideas here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
