The real fix is that the spark sink suite does not really need to use to the spark-streaming test jars. Removing that dependency altogether, and submitting a PR.
TD On Fri, Aug 22, 2014 at 6:34 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Figured it out. Fixing this ASAP. > > TD > > > On Fri, Aug 22, 2014 at 5:51 PM, Patrick Wendell <pwend...@gmail.com> > wrote: > >> Hey All, >> >> We can sort this out ASAP. Many of the Spark committers were at a company >> offsite for the last 72 hours, so sorry that it is broken. >> >> - Patrick >> >> >> On Fri, Aug 22, 2014 at 4:07 PM, Hari Shreedharan < >> hshreedha...@cloudera.com >> > wrote: >> >> > Sean - I think only the ones in 1726 are enough. It is weird that any >> > class that uses the test-jar actually requires the streaming jar to be >> > added explicitly. Shouldn't maven take care of this? >> > >> > I posted some comments on the PR. >> > >> > -- >> > >> > Thanks, >> > Hari >> > >> > >> > Sean Owen <mailto:so...@cloudera.com> >> >> August 22, 2014 at 3:58 PM >> >> >> >> Yes, master hasn't compiled for me for a few days. It's fixed in: >> >> >> >> https://github.com/apache/spark/pull/1726 >> >> https://github.com/apache/spark/pull/2075 >> >> >> >> Could a committer sort this out? >> >> >> >> Sean >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >> >> Ted Yu <mailto:yuzhih...@gmail.com> >> >> August 22, 2014 at 1:55 PM >> >> >> >> Hi, >> >> Using the following command on (refreshed) master branch: >> >> mvn clean package -DskipTests >> >> >> >> I got: >> >> >> >> constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/ >> >> --------------------------------------------------- >> >> java.lang.reflect.InvocationTargetException >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke( >> >> NativeMethodAccessorImpl.java:57) >> >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke( >> >> DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> >> org.codehaus.plexus.classworlds.launcher.Launcher. >> >> launchEnhanced(Launcher.java:289) >> >> at >> >> org.codehaus.plexus.classworlds.launcher.Launcher. >> >> launch(Launcher.java:229) >> >> at >> >> org.codehaus.plexus.classworlds.launcher.Launcher. >> >> mainWithExitCode(Launcher.java:415) >> >> at org.codehaus.plexus.classworlds.launcher.Launcher. >> >> main(Launcher.java:356) >> >> Caused by: scala.reflect.internal.Types$TypeError: bad symbolic >> >> reference. >> >> A signature in TestSuiteBase.class refers to term dstream >> >> in package org.apache.spark.streaming which is not available. >> >> It may be completely missing from the current classpath, or the >> version on >> >> the classpath might be incompatible with the version used when >> compiling >> >> TestSuiteBase.class. >> >> at >> >> scala.reflect.internal.pickling.UnPickler$Scan. >> >> toTypeError(UnPickler.scala:847) >> >> at >> >> scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete( >> >> UnPickler.scala:854) >> >> at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231) >> >> at >> >> >> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply( >> >> Types.scala:4280) >> >> at >> >> >> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply( >> >> Types.scala:4280) >> >> at >> >> scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized. >> >> scala:70) >> >> at scala.collection.immutable.List.forall(List.scala:84) >> >> at scala.reflect.internal.Types$TypeMap.noChangeToSymbols( >> >> Types.scala:4280) >> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293) >> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196) >> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638) >> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202) >> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638) >> >> at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754) >> >> at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773) >> >> at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224) >> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315) >> >> at >> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply( >> >> ExtractAPI.scala:296) >> >> at >> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply( >> >> ExtractAPI.scala:296) >> >> at >> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply( >> >> TraversableLike.scala:251) >> >> at >> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply( >> >> TraversableLike.scala:251) >> >> at >> >> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized. >> >> scala:33) >> >> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) >> >> at scala.collection.TraversableLike$class.flatMap( >> >> TraversableLike.scala:251) >> >> at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108) >> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI. >> >> scala:296) >> >> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293) >> >> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293) >> >> at xsbt.Message$$anon$1.apply(Message.scala:8) >> >> at xsbti.SafeLazy$$anonfun$apply$1.apply(SafeLazy.scala:8) >> >> at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:20) >> >> at xsbti.SafeLazy$Impl._t(SafeLazy.scala:18) >> >> at xsbti.SafeLazy$Impl.get(SafeLazy.scala:24) >> >> at >> xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138) >> >> at >> xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138) >> >> at scala.collection.immutable.List.foreach(List.scala:318) >> >> at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:138) >> >> at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:139) >> >> at xsbt.API$ApiPhase.processScalaUnit(API.scala:54) >> >> at xsbt.API$ApiPhase.processUnit(API.scala:38) >> >> at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34) >> >> at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34) >> >> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >> >> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) >> >> at xsbt.API$ApiPhase.run(API.scala:34) >> >> at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1583) >> >> at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1557) >> >> at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553) >> >> at scala.tools.nsc.Global$Run.compile(Global.scala:1662) >> >> at xsbt.CachedCompiler0.run(CompilerInterface.scala:123) >> >> at xsbt.CachedCompiler0.run(CompilerInterface.scala:99) >> >> at xsbt.CompilerInterface.run(CompilerInterface.scala:27) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke( >> >> NativeMethodAccessorImpl.java:57) >> >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke( >> >> DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:102) >> >> at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:48) >> >> at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41) >> >> at >> >> sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1. >> >> apply$mcV$sp(AggressiveCompile.scala:99) >> >> at >> >> sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1. >> >> apply(AggressiveCompile.scala:99) >> >> at >> >> sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1. >> >> apply(AggressiveCompile.scala:99) >> >> at >> >> sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$ >> >> timed(AggressiveCompile.scala:166) >> >> at >> >> sbt.compiler.AggressiveCompile$$anonfun$3.compileScala$1( >> >> AggressiveCompile.scala:98) >> >> at >> >> >> sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala: >> >> 143) >> >> at >> >> >> sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala: >> >> 87) >> >> at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply( >> >> Compile.scala:39) >> >> at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply( >> >> Compile.scala:37) >> >> at sbt.inc.IncrementalCommon.cycle(Incremental.scala:99) >> >> at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:38) >> >> at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:37) >> >> at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:65) >> >> at sbt.inc.Incremental$.compile(Incremental.scala:37) >> >> at sbt.inc.IncrementalCompile$.apply(Compile.scala:27) >> >> at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:157) >> >> at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:71) >> >> at com.typesafe.zinc.Compiler.compile(Compiler.scala:184) >> >> at com.typesafe.zinc.Compiler.compile(Compiler.scala:164) >> >> at >> sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:92) >> >> at >> >> scala_maven.ScalaCompilerSupport.incrementalCompile( >> >> ScalaCompilerSupport.java:303) >> >> at scala_maven.ScalaCompilerSupport.compile( >> >> ScalaCompilerSupport.java:119) >> >> at scala_maven.ScalaCompilerSupport.doExecute( >> >> ScalaCompilerSupport.java:99) >> >> at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482) >> >> at >> scala_maven.ScalaTestCompileMojo.execute(ScalaTestCompileMojo.java:48) >> >> at >> >> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( >> >> DefaultBuildPluginManager.java:106) >> >> at >> >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( >> >> MojoExecutor.java:208) >> >> at >> >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( >> >> MojoExecutor.java:153) >> >> at >> >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( >> >> MojoExecutor.java:145) >> >> at >> >> >> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject( >> >> LifecycleModuleBuilder.java:84) >> >> at >> >> >> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject( >> >> LifecycleModuleBuilder.java:59) >> >> at >> >> >> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild( >> >> LifecycleStarter.java:183) >> >> at >> >> org.apache.maven.lifecycle.internal.LifecycleStarter. >> >> execute(LifecycleStarter.java:161) >> >> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317) >> >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152) >> >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555) >> >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214) >> >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:158) >> >> ... 8 more >> >> >> >> Has anyone seen similar error ? >> >> >> >> Cheers >> >> >> >> >> > >
