On 29 Oct 2016, at 10:50, Sean Owen <so...@cloudera.com<mailto:so...@cloudera.com>> wrote:
Declare your scalatest dependency as test scope (which is correct anyway). That would override it I think as desired? not sure about that, but then mvn dependencies are one of those graph-theory problems. It may just add it to the test CP alongside the compile one, so: no change. Someone would need to do an experiment there. If the scalatags dependency is marked as provided, then maven won't put it on the transitive CP. And I assume, Ivy will work similarly On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu <shixi...@databricks.com<mailto:shixi...@databricks.com>> wrote: This is my test pom: <project> <modelVersion>4.0.0</modelVersion> <artifactId>foo</artifactId> <groupId>bar</groupId> <version>1.0</version> <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>2.0.1</version> </dependency> </dependencies> </project> scalatest is in the compile scope: [INFO] bar:foo:jar:1.0 [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile [INFO] +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile [INFO] | +- org.apache.avro:avro-ipc:jar:1.7.7:compile [INFO] | | \- org.apache.avro:avro:jar:1.7.7:compile [INFO] | +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] +- com.twitter:chill_2.10:jar:0.8.0:compile [INFO] | \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile [INFO] | +- com.esotericsoftware:minlog:jar:1.3.0:compile [INFO] | \- org.objenesis:objenesis:jar:2.1:compile [INFO] +- com.twitter:chill-java:jar:0.8.0:compile [INFO] +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | | +- org.apache.commons:commons-math:jar:2.1:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-io:commons-io:jar:2.1:compile [INFO] | | +- commons-lang:commons-lang:jar:2.5:compile [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile [INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | | \- org.tukaani:xz:jar:1.0:compile [INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile [INFO] | | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile [INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile [INFO] | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile [INFO] | | | | \- com.google.inject:guice:jar:3.0:compile [INFO] | | | | +- javax.inject:javax.inject:jar:1:compile [INFO] | | | | \- aopalliance:aopalliance:jar:1.0:compile [INFO] | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile [INFO] | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile [INFO] +- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile [INFO] +- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile [INFO] | +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile [INFO] | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile [INFO] +- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile [INFO] +- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile [INFO] +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile [INFO] | +- commons-codec:commons-codec:jar:1.3:compile [INFO] | \- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] +- org.apache.curator:curator-recipes:jar:2.4.0:compile [INFO] | +- org.apache.curator:curator-framework:jar:2.4.0:compile [INFO] | | \- org.apache.curator:curator-client:jar:2.4.0:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] | \- com.google.guava:guava:jar:14.0.1:compile [INFO] +- javax.servlet:javax.servlet-api:jar:3.1.0:compile [INFO] +- org.apache.commons:commons-lang3:jar:3.3.2:compile [INFO] +- org.apache.commons:commons-math3:jar:3.4.1:compile [INFO] +- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] +- org.slf4j:slf4j-api:jar:1.7.16:compile [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile [INFO] +- log4j:log4j:jar:1.2.17:compile [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile [INFO] +- com.ning:compress-lzf:jar:1.0.3:compile [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile [INFO] +- net.jpountz.lz4:lz4:jar:1.3.0:compile [INFO] +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile [INFO] +- commons-net:commons-net:jar:2.2:compile [INFO] +- org.scala-lang:scala-library:jar:2.10.6:compile [INFO] +- org.json4s:json4s-jackson_2.10:jar:3.2.11:compile [INFO] | \- org.json4s:json4s-core_2.10:jar:3.2.11:compile [INFO] | +- org.json4s:json4s-ast_2.10:jar:3.2.11:compile [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile [INFO] | \- org.scala-lang:scalap:jar:2.10.0:compile [INFO] | \- org.scala-lang:scala-compiler:jar:2.10.0:compile [INFO] +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile [INFO] | +- javax.ws.rs<http://javax.ws.rs>:javax.ws.rs<http://javax.ws.rs>-api:jar:2.0.1:compile [INFO] | +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile [INFO] | | +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile [INFO] | | \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:compile [INFO] | +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile [INFO] | \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile [INFO] | \- org.javassist:javassist:jar:3.18.1-GA:compile [INFO] +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile [INFO] | +- javax.annotation:javax.annotation-api:jar:1.2:compile [INFO] | +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:compile [INFO] | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile [INFO] +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile [INFO] | +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:compile [INFO] | \- javax.validation:validation-api:jar:1.1.0.Final:compile [INFO] +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:compile [INFO] +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:compile [INFO] +- org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1:compile [INFO] +- io.netty:netty-all:jar:4.0.29.Final:compile [INFO] +- io.netty:netty:jar:3.8.0.Final:compile [INFO] +- com.clearspring.analytics:stream:jar:2.7.0:compile [INFO] +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile [INFO] +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile [INFO] +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile [INFO] +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile [INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:compile [INFO] +- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.6.5:compile [INFO] | +- org.scala-lang:scala-reflect:jar:2.10.6:compile [INFO] | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:compile [INFO] +- org.apache.ivy:ivy:jar:2.4.0:compile [INFO] +- oro:oro:jar:2.0.8:compile [INFO] +- net.razorvine:pyrolite:jar:4.9:compile [INFO] +- net.sf.py4j:py4j:jar:0.10.3:compile [INFO] +- org.apache.spark:spark-tags_2.10:jar:2.0.1:compile [INFO] | \- org.scalatest:scalatest_2.10:jar:2.2.6:compile [INFO] \- org.spark-project.spark:unused:jar:1.0.0:compile On Fri, Oct 28, 2016 at 1:04 PM, Sean Owen <so...@cloudera.com<mailto:so...@cloudera.com>> wrote: Yes, but scalatest doesn't end up in compile scope, says Maven? ... [INFO] +- org.apache.spark:spark-tags_2.11:jar:2.1.0-SNAPSHOT:compile [INFO] | +- (org.scalatest:scalatest_2.11:jar:2.2.6:test - scope managed from compile; omitted for duplicate) [INFO] | \- (org.spark-project.spark:unused:jar:1.0.0:compile - omitted for duplicate) [INFO] +- org.apache.commons:commons-crypto:jar:1.0.0:compile [INFO] +- org.spark-project.spark:unused:jar:1.0.0:compile [INFO] +- org.scalatest:scalatest_2.11:jar:2.2.6:test ... On Fri, Oct 28, 2016 at 8:52 PM Jeremy Smith <jeremy.sm...@acorns.com<mailto:jeremy.sm...@acorns.com>> wrote: spark-core depends on spark-launcher (compile) spark-launcher depends on spark-tags (compile) spark-tags depends on scalatest (compile) To be honest I'm not all that familiar with the project structure - should I just exclude spark-launcher if I'm not using it? On Fri, Oct 28, 2016 at 12:27 PM, Sean Owen <so...@cloudera.com<mailto:so...@cloudera.com>> wrote: It's required because the tags module uses it to define annotations for tests. I don't see it in compile scope for anything but the tags module, which is then in test scope for other modules. What are you seeing that makes you say it's in compile scope? On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith <jeremy.sm...@acorns.com<mailto:jeremy.sm...@acorns.com>> wrote: Hey everybody, Just a heads up that currently Spark 2.0.1 has a compile dependency on Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher, which has a transitive dependency on spark-tags, which has a compile dependency on Scalatest. This makes it impossible to use any other version of Scalatest for testing your app if you declare a dependency on any Spark 2.0.1 module; you'll get a bunch of runtime errors during testing (unless you figure out the reason like I did and explicitly exclude Scalatest from the spark dependency). I think that dependency should probably be moved to a test dependency instead. Thanks, Jeremy
