Re: Spark has a compile dependency on scalatest

[Sean Owen]

SBT and Maven resolution rules do differ. I thought SBT was generally
latest-first though, which should make 3.0 take priority. Maven is more
like closest-first, which means you can pretty much always override things
in your own build. An exclusion is the right way to go in this case because
the deployed POM does look like it says what we all want it to say, at
least.

On Sun, Oct 30, 2016 at 11:40 PM Jeremy Smith 
wrote:

> Well, recent experience tells me that at least SBT does not behave this
> way. Given a `test` dependency on scalatest 3.0.0, my tests were still
> being compiled against 2.2.6, which caused a combination of compile errors
> and runtime errors (given that apparently 3.0.0 was still present at
> runtime during the test run.) Actually, it could be that they were being
> compiled against 3.0.0 but 2.2.6 was present at runtime. I'm not sure which
> is accurate, but adding an exclusion rule when bringing in Spark in order
> to exclude both scalatest and scalactic resolved the problem.
>
> This may not be a huge issue in real practice, since people will typically
> be marking spark in its entirety as `provided,test`. In any case, it
> doesn't seem like it's intentional that scalatest makes its way into
> compile scope when using any spark dependency in SBT. It's not clear to me
> whether this is an issue with SBT or with the spark build, but it's a thing
> that does happen when using SBT (which I imagine a great many people do
> when building Spark applications).
>
> Jeremy
>
> On Sat, Oct 29, 2016 at 2:50 AM, Sean Owen  wrote:
>
> Declare your scalatest dependency as test scope (which is correct anyway).
> That would override it I think as desired?
>
>
> On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu 
> wrote:
>
> This is my test pom:
>
> 
> 4.0.0
>   foo
> bar
> 1.0
> 
>
> 
> org.apache.spark
> spark-core_2.10
> 2.0.1
> 
>  
>
> 
>
> scalatest is in the compile scope:
>
> [INFO] bar:foo:jar:1.0
> [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
> [INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
> [INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
> [INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
> [INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
> [INFO]| \- org.objenesis:objenesis:jar:2.1:compile
> [INFO]+- com.twitter:chill-java:jar:0.8.0:compile
> [INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
> [INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> [INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
> [INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
> [INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
> [INFO]|  |  +-
> commons-configuration:commons-configuration:jar:1.6:compile
> [INFO]|  |  |  +-
> commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO]|  |  |  |  \-
> commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO]|  |  |  \-
> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO]|  | \- org.tukaani:xz:jar:1.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
> [INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
> [INFO]|  |  +-
> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
> [INFO]|  |  |  +-
> org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
> [INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
> [INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
> [INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
> [INFO]|  |  |  \-
> org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
> [INFO]|  |  \-
> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile

Re: Spark has a compile dependency on scalatest

[Jeremy Smith]

Well, recent experience tells me that at least SBT does not behave this
way. Given a `test` dependency on scalatest 3.0.0, my tests were still
being compiled against 2.2.6, which caused a combination of compile errors
and runtime errors (given that apparently 3.0.0 was still present at
runtime during the test run.) Actually, it could be that they were being
compiled against 3.0.0 but 2.2.6 was present at runtime. I'm not sure which
is accurate, but adding an exclusion rule when bringing in Spark in order
to exclude both scalatest and scalactic resolved the problem.

This may not be a huge issue in real practice, since people will typically
be marking spark in its entirety as `provided,test`. In any case, it
doesn't seem like it's intentional that scalatest makes its way into
compile scope when using any spark dependency in SBT. It's not clear to me
whether this is an issue with SBT or with the spark build, but it's a thing
that does happen when using SBT (which I imagine a great many people do
when building Spark applications).

Jeremy

On Sat, Oct 29, 2016 at 2:50 AM, Sean Owen  wrote:

> Declare your scalatest dependency as test scope (which is correct anyway).
> That would override it I think as desired?
>
>
> On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu 
> wrote:
>
>> This is my test pom:
>>
>> 
>> 4.0.0
>>   foo
>> bar
>> 1.0
>> 
>>
>> 
>> org.apache.spark
>> spark-core_2.10
>> 2.0.1
>> 
>>  
>>
>> 
>>
>> scalatest is in the compile scope:
>>
>> [INFO] bar:foo:jar:1.0
>> [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
>> [INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
>> [INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
>> [INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
>> [INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
>> [INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
>> [INFO]|  \- org.codehaus.jackson:jackson-
>> mapper-asl:jar:1.9.13:compile
>> [INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
>> [INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
>> [INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
>> [INFO]| \- org.objenesis:objenesis:jar:2.1:compile
>> [INFO]+- com.twitter:chill-java:jar:0.8.0:compile
>> [INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
>> [INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
>> [INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
>> [INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
>> [INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
>> [INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
>> [INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
>> [INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
>> [INFO]|  |  +- commons-configuration:commons-
>> configuration:jar:1.6:compile
>> [INFO]|  |  |  +- commons-collections:commons-
>> collections:jar:3.2.1:compile
>> [INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
>> [INFO]|  |  |  |  \- commons-beanutils:commons-
>> beanutils:jar:1.7.0:compile
>> [INFO]|  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:
>> compile
>> [INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
>> [INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
>> [INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
>> [INFO]|  | \- org.tukaani:xz:jar:1.0:compile
>> [INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
>> [INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
>> [INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.
>> 0:compile
>> [INFO]|  |  +- org.apache.hadoop:hadoop-
>> mapreduce-client-common:jar:2.2.0:compile
>> [INFO]|  |  |  +- org.apache.hadoop:hadoop-yarn-
>> client:jar:2.2.0:compile
>> [INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
>> [INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
>> [INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
>> [INFO]|  |  |  \- org.apache.hadoop:hadoop-yarn-
>> server-common:jar:2.2.0:compile
>> [INFO]|  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:
>> 2.2.0:compile
>> [INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
>> [INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.
>> 0:compile
>> [INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
>> [INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:
>> jar:2.2.0:compile
>> [INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
>> [INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
>> [INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
>> [INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
>> [INFO]

Re: Spark has a compile dependency on scalatest

[Steve Loughran]

On 29 Oct 2016, at 10:50, Sean Owen 
> wrote:

Declare your scalatest dependency as test scope (which is correct anyway). That 
would override it I think as desired?


not sure about that, but then mvn dependencies are one of those graph-theory 
problems. It may just add it to the test CP alongside the compile one, so: no 
change. Someone would need to do an experiment there.

If the scalatags dependency is marked as provided, then maven won't put it on 
the transitive CP. And I assume, Ivy will work similarly


On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu 
> wrote:
This is my test pom:


4.0.0
  foo
bar
1.0



org.apache.spark
spark-core_2.10
2.0.1

 



scalatest is in the compile scope:

[INFO] bar:foo:jar:1.0
[INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
[INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
[INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
[INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]| \- org.objenesis:objenesis:jar:2.1:compile
[INFO]+- com.twitter:chill-java:jar:0.8.0:compile
[INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
[INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
[INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO]|  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO]|  |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO]|  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO]|  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
[INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO]|  | \- org.tukaani:xz:jar:1.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO]|  |  +- 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO]|  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
[INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
[INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
[INFO]|  |  |  \- 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO]|  |  \- 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO]|  +- 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
[INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
[INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO]|  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
[INFO]+- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile
[INFO]+- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile
[INFO]+- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO]|  +- commons-codec:commons-codec:jar:1.3:compile
[INFO]|  \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO]+- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO]|  +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO]|  |  \- org.apache.curator:curator-client:jar:2.4.0:compile
[INFO]|  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO]|  \- com.google.guava:guava:jar:14.0.1:compile
[INFO]+- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO]+- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO]+- 

Re: Spark has a compile dependency on scalatest

[Sean Owen]

Declare your scalatest dependency as test scope (which is correct anyway).
That would override it I think as desired?

On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu 
wrote:

> This is my test pom:
>
> 
> 4.0.0
>   foo
> bar
> 1.0
> 
>
> 
> org.apache.spark
> spark-core_2.10
> 2.0.1
> 
>  
>
> 
>
> scalatest is in the compile scope:
>
> [INFO] bar:foo:jar:1.0
> [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
> [INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
> [INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
> [INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
> [INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
> [INFO]| \- org.objenesis:objenesis:jar:2.1:compile
> [INFO]+- com.twitter:chill-java:jar:0.8.0:compile
> [INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
> [INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> [INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
> [INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
> [INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
> [INFO]|  |  +-
> commons-configuration:commons-configuration:jar:1.6:compile
> [INFO]|  |  |  +-
> commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO]|  |  |  |  \-
> commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO]|  |  |  \-
> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO]|  | \- org.tukaani:xz:jar:1.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
> [INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
> [INFO]|  |  +-
> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
> [INFO]|  |  |  +-
> org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
> [INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
> [INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
> [INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
> [INFO]|  |  |  \-
> org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
> [INFO]|  |  \-
> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
> [INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
> [INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
> [INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
> [INFO]|  \-
> com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
> [INFO]+- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile
> [INFO]+- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO]|  +- commons-codec:commons-codec:jar:1.3:compile
> [INFO]|  \- commons-httpclient:commons-httpclient:jar:3.1:compile
> [INFO]+- org.apache.curator:curator-recipes:jar:2.4.0:compile
> [INFO]|  +- org.apache.curator:curator-framework:jar:2.4.0:compile
> [INFO]|  |  \- org.apache.curator:curator-client:jar:2.4.0:compile
> [INFO]|  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
> [INFO]|  \- com.google.guava:guava:jar:14.0.1:compile
> [INFO]+- javax.servlet:javax.servlet-api:jar:3.1.0:compile
> [INFO]+- org.apache.commons:commons-lang3:jar:3.3.2:compile
> [INFO]+- org.apache.commons:commons-math3:jar:3.4.1:compile
> [INFO]+- com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [INFO]+- org.slf4j:slf4j-api:jar:1.7.16:compile
> [INFO]+- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
> [INFO]+- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
> [INFO]+- log4j:log4j:jar:1.2.17:compile

Re: Spark has a compile dependency on scalatest

[Marcelo Vanzin]

Hmm. Yes, that makes sense. Spark's root pom does not affect your
application's pom, in which case it will pick compile over test if
there are conflicting dependencies.

Perhaps spark-tags should override it to provided instead of compile...

On Fri, Oct 28, 2016 at 1:22 PM, Shixiong(Ryan) Zhu
 wrote:
> This is my test pom:
>
> 
> 4.0.0
>   foo
> bar
> 1.0
> 
>
> 
> org.apache.spark
> spark-core_2.10
> 2.0.1
> 
>  
>
> 
>
> scalatest is in the compile scope:
>
> [INFO] bar:foo:jar:1.0
> [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
> [INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
> [INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
> [INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
> [INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
> [INFO]| \- org.objenesis:objenesis:jar:2.1:compile
> [INFO]+- com.twitter:chill-java:jar:0.8.0:compile
> [INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
> [INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> [INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
> [INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
> [INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
> [INFO]|  |  +-
> commons-configuration:commons-configuration:jar:1.6:compile
> [INFO]|  |  |  +-
> commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO]|  |  |  |  \-
> commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO]|  |  |  \-
> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO]|  | \- org.tukaani:xz:jar:1.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
> [INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
> [INFO]|  |  +-
> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
> [INFO]|  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
> [INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
> [INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
> [INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
> [INFO]|  |  |  \-
> org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
> [INFO]|  |  \-
> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
> [INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
> [INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
> [INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
> [INFO]|  \-
> com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
> [INFO]+- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile
> [INFO]+- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO]|  +- commons-codec:commons-codec:jar:1.3:compile
> [INFO]|  \- commons-httpclient:commons-httpclient:jar:3.1:compile
> [INFO]+- org.apache.curator:curator-recipes:jar:2.4.0:compile
> [INFO]|  +- org.apache.curator:curator-framework:jar:2.4.0:compile
> [INFO]|  |  \- org.apache.curator:curator-client:jar:2.4.0:compile
> [INFO]|  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
> [INFO]|  \- com.google.guava:guava:jar:14.0.1:compile
> [INFO]+- javax.servlet:javax.servlet-api:jar:3.1.0:compile
> [INFO]+- org.apache.commons:commons-lang3:jar:3.3.2:compile
> [INFO]+- org.apache.commons:commons-math3:jar:3.4.1:compile
> [INFO]+- com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [INFO]+- org.slf4j:slf4j-api:jar:1.7.16:compile
> [INFO]+- 

Re: Spark has a compile dependency on scalatest

[Shixiong(Ryan) Zhu]

This is my test pom:


4.0.0
  foo
bar
1.0



org.apache.spark
spark-core_2.10
2.0.1

 



scalatest is in the compile scope:

[INFO] bar:foo:jar:1.0
[INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
[INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
[INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
[INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]| \- org.objenesis:objenesis:jar:2.1:compile
[INFO]+- com.twitter:chill-java:jar:0.8.0:compile
[INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
[INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
[INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO]|  |  +-
commons-configuration:commons-configuration:jar:1.6:compile
[INFO]|  |  |  +-
commons-collections:commons-collections:jar:3.2.1:compile
[INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO]|  |  |  |  \-
commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO]|  |  |  \-
commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
[INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO]|  | \- org.tukaani:xz:jar:1.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO]|  +-
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO]|  |  +-
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO]|  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
[INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
[INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
[INFO]|  |  |  \-
org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO]|  |  \-
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO]|  +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO]|  +-
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
[INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
[INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO]|  \-
com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
[INFO]+- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile
[INFO]+- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile
[INFO]+- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO]|  +- commons-codec:commons-codec:jar:1.3:compile
[INFO]|  \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO]+- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO]|  +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO]|  |  \- org.apache.curator:curator-client:jar:2.4.0:compile
[INFO]|  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO]|  \- com.google.guava:guava:jar:14.0.1:compile
[INFO]+- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO]+- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO]+- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO]+- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO]+- org.slf4j:slf4j-api:jar:1.7.16:compile
[INFO]+- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
[INFO]+- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
[INFO]+- log4j:log4j:jar:1.2.17:compile
[INFO]+- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
[INFO]+- com.ning:compress-lzf:jar:1.0.3:compile
[INFO]+- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO]+- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO]+- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile
[INFO]+- commons-net:commons-net:jar:2.2:compile
[INFO]+- 

Re: Spark has a compile dependency on scalatest

[Marcelo Vanzin]

The root pom declares scalatest explicitly with test scope. It's added
by default to all sub-modules, so every one should get it in test
scope unless the module explicitly overrides that, like the tags
module does.

If you look at the "blessed" dependency list in dev/deps, there's no scalatest.

That being said, there are two things:
- sbt seems to get confused with that; if I look at the assembly jars
dir created by sbt, it includes scalatest.
- there's always a chance that the published pom did something wrong
and promoted things when it shouldn't because of a bug. But that
doesn't seem to be the case:

http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.0.1/spark-core_2.11-2.0.1.pom

Has:


org.scalatest
scalatest_2.11
2.2.6
test



On Fri, Oct 28, 2016 at 1:04 PM, Sean Owen  wrote:
> Yes, but scalatest doesn't end up in compile scope, says Maven?
>
> ...
>
> [INFO] +- org.apache.spark:spark-tags_2.11:jar:2.1.0-SNAPSHOT:compile
>
> [INFO] |  +- (org.scalatest:scalatest_2.11:jar:2.2.6:test - scope managed
> from compile; omitted for duplicate)
>
> [INFO] |  \- (org.spark-project.spark:unused:jar:1.0.0:compile - omitted for
> duplicate)
>
> [INFO] +- org.apache.commons:commons-crypto:jar:1.0.0:compile
>
> [INFO] +- org.spark-project.spark:unused:jar:1.0.0:compile
>
> [INFO] +- org.scalatest:scalatest_2.11:jar:2.2.6:test
>
> ...
>
>
> On Fri, Oct 28, 2016 at 8:52 PM Jeremy Smith 
> wrote:
>>
>> spark-core depends on spark-launcher (compile)
>> spark-launcher depends on spark-tags (compile)
>> spark-tags depends on scalatest (compile)
>>
>> To be honest I'm not all that familiar with the project structure - should
>> I just exclude spark-launcher if I'm not using it?
>>
>> On Fri, Oct 28, 2016 at 12:27 PM, Sean Owen  wrote:
>>>
>>> It's required because the tags module uses it to define annotations for
>>> tests. I don't see it in compile scope for anything but the tags module,
>>> which is then in test scope for other modules. What are you seeing that
>>> makes you say it's in compile scope?
>>>
>>> On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith 
>>> wrote:

 Hey everybody,

 Just a heads up that currently Spark 2.0.1 has a compile dependency on
 Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher,
 which has a transitive dependency on spark-tags, which has a compile
 dependency on Scalatest.

 This makes it impossible to use any other version of Scalatest for
 testing your app if you declare a dependency on any Spark 2.0.1 module;
 you'll get a bunch of runtime errors during testing (unless you figure out
 the reason like I did and explicitly exclude Scalatest from the spark
 dependency).

 I think that dependency should probably be moved to a test dependency
 instead.

 Thanks,
 Jeremy
>>
>>
>



-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Spark has a compile dependency on scalatest

[Sean Owen]

Yes, but scalatest doesn't end up in compile scope, says Maven?

...

[INFO] +- org.apache.spark:spark-tags_2.11:jar:2.1.0-SNAPSHOT:compile

[INFO] |  +- (org.scalatest:scalatest_2.11:jar:2.2.6:test - scope managed
from compile; omitted for duplicate)

[INFO] |  \- (org.spark-project.spark:unused:jar:1.0.0:compile - omitted
for duplicate)

[INFO] +- org.apache.commons:commons-crypto:jar:1.0.0:compile

[INFO] +- org.spark-project.spark:unused:jar:1.0.0:compile

[INFO] +- org.scalatest:scalatest_2.11:jar:2.2.6:test

...

On Fri, Oct 28, 2016 at 8:52 PM Jeremy Smith 
wrote:

> spark-core depends on spark-launcher (compile)
> spark-launcher depends on spark-tags (compile)
> spark-tags depends on scalatest (compile)
>
> To be honest I'm not all that familiar with the project structure - should
> I just exclude spark-launcher if I'm not using it?
>
> On Fri, Oct 28, 2016 at 12:27 PM, Sean Owen  wrote:
>
> It's required because the tags module uses it to define annotations for
> tests. I don't see it in compile scope for anything but the tags module,
> which is then in test scope for other modules. What are you seeing that
> makes you say it's in compile scope?
>
> On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith 
> wrote:
>
> Hey everybody,
>
> Just a heads up that currently Spark 2.0.1 has a compile dependency on
> Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher,
> which has a transitive dependency on spark-tags, which has a compile
> dependency on Scalatest.
>
> This makes it impossible to use any other version of Scalatest for testing
> your app if you declare a dependency on any Spark 2.0.1 module; you'll get
> a bunch of runtime errors during testing (unless you figure out the reason
> like I did and explicitly exclude Scalatest from the spark dependency).
>
> I think that dependency should probably be moved to a test dependency
> instead.
>
> Thanks,
> Jeremy
>
>
>

Re: Spark has a compile dependency on scalatest

[Shixiong(Ryan) Zhu]

You can just exclude scalatest from Spark.

On Fri, Oct 28, 2016 at 12:51 PM, Jeremy Smith 
wrote:

> spark-core depends on spark-launcher (compile)
> spark-launcher depends on spark-tags (compile)
> spark-tags depends on scalatest (compile)
>
> To be honest I'm not all that familiar with the project structure - should
> I just exclude spark-launcher if I'm not using it?
>
> On Fri, Oct 28, 2016 at 12:27 PM, Sean Owen  wrote:
>
>> It's required because the tags module uses it to define annotations for
>> tests. I don't see it in compile scope for anything but the tags module,
>> which is then in test scope for other modules. What are you seeing that
>> makes you say it's in compile scope?
>>
>> On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith 
>> wrote:
>>
>>> Hey everybody,
>>>
>>> Just a heads up that currently Spark 2.0.1 has a compile dependency on
>>> Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher,
>>> which has a transitive dependency on spark-tags, which has a compile
>>> dependency on Scalatest.
>>>
>>> This makes it impossible to use any other version of Scalatest for
>>> testing your app if you declare a dependency on any Spark 2.0.1 module;
>>> you'll get a bunch of runtime errors during testing (unless you figure out
>>> the reason like I did and explicitly exclude Scalatest from the spark
>>> dependency).
>>>
>>> I think that dependency should probably be moved to a test dependency
>>> instead.
>>>
>>> Thanks,
>>> Jeremy
>>>
>>
>

Re: Spark has a compile dependency on scalatest

[Shixiong(Ryan) Zhu]

spark-tags is in the compile scope of spark-core...

On Fri, Oct 28, 2016 at 12:27 PM, Sean Owen  wrote:

> It's required because the tags module uses it to define annotations for
> tests. I don't see it in compile scope for anything but the tags module,
> which is then in test scope for other modules. What are you seeing that
> makes you say it's in compile scope?
>
> On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith 
> wrote:
>
>> Hey everybody,
>>
>> Just a heads up that currently Spark 2.0.1 has a compile dependency on
>> Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher,
>> which has a transitive dependency on spark-tags, which has a compile
>> dependency on Scalatest.
>>
>> This makes it impossible to use any other version of Scalatest for
>> testing your app if you declare a dependency on any Spark 2.0.1 module;
>> you'll get a bunch of runtime errors during testing (unless you figure out
>> the reason like I did and explicitly exclude Scalatest from the spark
>> dependency).
>>
>> I think that dependency should probably be moved to a test dependency
>> instead.
>>
>> Thanks,
>> Jeremy
>>
>

Re: Spark has a compile dependency on scalatest

[Sean Owen]

It's required because the tags module uses it to define annotations for
tests. I don't see it in compile scope for anything but the tags module,
which is then in test scope for other modules. What are you seeing that
makes you say it's in compile scope?

On Fri, Oct 28, 2016 at 8:19 PM Jeremy Smith 
wrote:

> Hey everybody,
>
> Just a heads up that currently Spark 2.0.1 has a compile dependency on
> Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher,
> which has a transitive dependency on spark-tags, which has a compile
> dependency on Scalatest.
>
> This makes it impossible to use any other version of Scalatest for testing
> your app if you declare a dependency on any Spark 2.0.1 module; you'll get
> a bunch of runtime errors during testing (unless you figure out the reason
> like I did and explicitly exclude Scalatest from the spark dependency).
>
> I think that dependency should probably be moved to a test dependency
> instead.
>
> Thanks,
> Jeremy
>

Spark has a compile dependency on scalatest

[Jeremy Smith]

Hey everybody,

Just a heads up that currently Spark 2.0.1 has a compile dependency on
Scalatest 2.2.6. It comes from spark-core's dependency on spark-launcher,
which has a transitive dependency on spark-tags, which has a compile
dependency on Scalatest.

This makes it impossible to use any other version of Scalatest for testing
your app if you declare a dependency on any Spark 2.0.1 module; you'll get
a bunch of runtime errors during testing (unless you figure out the reason
like I did and explicitly exclude Scalatest from the spark dependency).

I think that dependency should probably be moved to a test dependency
instead.

Thanks,
Jeremy