Is it possible to come up with code snippet which reproduces the following ?
Thanks On Fri, May 13, 2016 at 8:13 AM, Raghava Mutharaju < m.vijayaragh...@gmail.com> wrote: > I am able to run my application after I compiled Spark source in the > following way > > ./dev/change-scala-version.sh 2.11 > > ./dev/make-distribution.sh --name spark-2.0.0-snapshot-bin-hadoop2.6 --tgz > -Phadoop-2.6 -DskipTests > > But while the application is running I get the following exception, which > I was not getting with Spark 1.6.1. Any idea why this might be happening? > > java.lang.IllegalArgumentException: requirement failed: chunks must be > non-empty > > at scala.Predef$.require(Predef.scala:224) > > at > org.apache.spark.util.io.ChunkedByteBuffer.<init>(ChunkedByteBuffer.scala:41) > > at > org.apache.spark.util.io.ChunkedByteBuffer.<init>(ChunkedByteBuffer.scala:52) > > at > org.apache.spark.storage.BlockManager.getRemoteBytes(BlockManager.scala:580) > > at > org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:514) > > at org.apache.spark.storage.BlockManager.get(BlockManager.scala:601) > > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:653) > > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:329) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:280) > > at > org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100) > > at > org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99) > > at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) > > at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) > > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > > at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > > at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > > at scala.collection.mutable.SetBuilder.$plus$plus$eq(SetBuilder.scala:20) > > at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) > > at scala.collection.AbstractIterator.to(Iterator.scala:1336) > > at scala.collection.TraversableOnce$class.toSet(TraversableOnce.scala:304) > > at scala.collection.AbstractIterator.toSet(Iterator.scala:1336) > > at > org.daselab.sparkel.SparkELHDFSTestCopy$$anonfun$45.apply(SparkELHDFSTestCopy.scala:392) > > at > org.daselab.sparkel.SparkELHDFSTestCopy$$anonfun$45.apply(SparkELHDFSTestCopy.scala:391) > > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$22.apply(RDD.scala:756) > > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$22.apply(RDD.scala:756) > > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:282) > > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:282) > > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:282) > > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > > at org.apache.spark.scheduler.Task.run(Task.scala:85) > > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > On Fri, May 13, 2016 at 6:33 AM, Raghava Mutharaju < > m.vijayaragh...@gmail.com> wrote: > >> Thank you for the response. >> >> I used the following command to build from source >> >> build/mvn -Dhadoop.version=2.6.4 -Phadoop-2.6 -DskipTests clean package >> >> Would this put in the required jars in .ivy2 during the build process? If >> so, how can I make the spark distribution runnable, so that I can use it on >> other machines as well (make-distribution.sh no longer exists in Spark root >> folder)? >> >> For compiling my application, I put in the following lines in the >> build.sbt >> >> packAutoSettings >> val spark = "org.apache.spark" %% "spark-core" % "2.0.0-SNAPSHOT" >> val sparksql = "org.apache.spark" % "spark-sql_2.11" % "2.0.0-SNAPSHOT" >> >> lazy val root = (project in file(".")). >> settings( >> name := "sparkel", >> version := "0.1.0", >> scalaVersion := "2.11.8", >> libraryDependencies += spark, >> libraryDependencies += sparksql >> ) >> >> >> Regards, >> Raghava. >> >> >> On Fri, May 13, 2016 at 12:23 AM, Luciano Resende <luckbr1...@gmail.com> >> wrote: >> >>> Spark has moved to build using Scala 2.11 by default in master/trunk. >>> >>> As for the 2.0.0-SNAPSHOT, it is actually the version of master/trunk >>> and you might be missing some modules/profiles for your build. What command >>> did you use to build ? >>> >>> On Thu, May 12, 2016 at 9:01 PM, Raghava Mutharaju < >>> m.vijayaragh...@gmail.com> wrote: >>> >>>> Hello All, >>>> >>>> I built Spark from the source code available at >>>> https://github.com/apache/spark/. Although I haven't specified the >>>> "-Dscala-2.11" option (to build with Scala 2.11), from the build messages I >>>> see that it ended up using Scala 2.11. Now, for my application sbt, what >>>> should be the spark version? I tried the following >>>> >>>> val spark = "org.apache.spark" %% "spark-core" % "2.0.0-SNAPSHOT" >>>> val sparksql = "org.apache.spark" % "spark-sql_2.11" % "2.0.0-SNAPSHOT" >>>> >>>> and scalaVersion := "2.11.8" >>>> >>>> But this setting of spark version gives sbt error >>>> >>>> unresolved dependency: org.apache.spark#spark-core_2.11;2.0.0-SNAPSHOT >>>> >>>> I guess this is because the repository doesn't contain 2.0.0-SNAPSHOT. >>>> Does this mean, the only option is to put all the required jars in the lib >>>> folder (unmanaged dependencies)? >>>> >>>> Regards, >>>> Raghava. >>>> >>> >>> >>> >>> -- >>> Luciano Resende >>> http://twitter.com/lresende1975 >>> http://lresende.blogspot.com/ >>> >> >> >> >> -- >> Regards, >> Raghava >> http://raghavam.github.io >> > > > > -- > Regards, > Raghava > http://raghavam.github.io >