no i havent used it with anything but 1.0.1 and 0.9.x . on a side note, I just have changed my employer. It is one of these big guys that make it very difficult to do any contributions. So I am not sure how much of anything i will be able to share/contribute.
On Tue, Oct 21, 2014 at 9:43 AM, Pat Ferrel <[email protected]> wrote: > But unless you have the time to devote to errors avoid it. I’ve built > everything from scratch using 1.0.2 and 1.1.0 and am getting these and > missing class errors. The 1.x branch seems to have some kind of peculiar > build order dependencies. The errors sometimes don’t show up until runtime, > passing all build tests. > > Dmitriy, have you successfully used any Spark version other than 1.0.1 on > a cluster? If so do you recall the exact order and from what sources you > built? > > > On Oct 21, 2014, at 9:35 AM, Dmitriy Lyubimov <[email protected]> wrote: > > You can't use spark client of one version and have the backend of another. > You can try to change spark dependency in mahout poms to match your backend > (or vice versa, you can change your backend to match what's on the client). > > On Tue, Oct 21, 2014 at 7:12 AM, Mahesh Balija <[email protected] > > > wrote: > > > Hi All, > > > > Here are the errors I get which I run in a pseudo distributed mode, > > > > Spark 1.0.2 and Mahout latest code (Clone) > > > > When I run the command in page, > > https://mahout.apache.org/users/sparkbindings/play-with-shell.html > > > > val drmX = drmData(::, 0 until 4) > > > > java.io.InvalidClassException: org.apache.spark.rdd.RDD; local class > > incompatible: stream classdesc serialVersionUID = 385418487991259089, > > local class serialVersionUID = -6766554341038829528 > > at > > java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:592) > > at > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1621) > > at > > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1516) > > at > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1621) > > at > > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1516) > > at > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1770) > > at > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349) > > at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) > > at > > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) > > at > > > org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61) > > at > > org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141) > > at > > java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1836) > > at > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795) > > at > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349) > > at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) > > at > > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) > > at > > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85) > > at > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:165) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:701) > > 14/10/21 19:35:37 WARN TaskSetManager: Lost TID 1 (task 0.0:1) > > 14/10/21 19:35:37 WARN TaskSetManager: Lost TID 2 (task 0.0:0) > > 14/10/21 19:35:37 WARN TaskSetManager: Lost TID 3 (task 0.0:1) > > 14/10/21 19:35:38 WARN TaskSetManager: Lost TID 4 (task 0.0:0) > > 14/10/21 19:35:38 WARN TaskSetManager: Lost TID 5 (task 0.0:1) > > 14/10/21 19:35:38 WARN TaskSetManager: Lost TID 6 (task 0.0:0) > > org.apache.spark.SparkException: Job aborted due to stage failure: > > Task 0.0:0 failed 4 times, most recent failure: Exception failure in > > TID 6 on host mahesh-VirtualBox.local: java.io.InvalidClassException: > > org.apache.spark.rdd.RDD; local class incompatible: stream classdesc > > serialVersionUID = 385418487991259089, local class serialVersionUID = > > -6766554341038829528 > > java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:592) > > > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1621) > > > > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1516) > > > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1621) > > > > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1516) > > > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1770) > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349) > > java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) > > > > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) > > > > > org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61) > > > > org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141) > > > > java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1836) > > > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795) > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349) > > java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) > > > > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) > > > > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85) > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:165) > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > java.lang.Thread.run(Thread.java:701) > > Driver stacktrace: > > at org.apache.spark.scheduler.DAGScheduler.org > > > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044) > > at > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028) > > at > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026) > > at > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > > at > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > > at > > > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026) > > at > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634) > > at > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634) > > at scala.Option.foreach(Option.scala:236) > > at > > > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634) > > at > > > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) > > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > > at > > > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) > > at > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > > > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at > > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > Best, > > Mahesh Balija. > > > > > > > > > > > > > > > > On Tue, Oct 21, 2014 at 2:38 AM, Dmitriy Lyubimov <[email protected]> > > wrote: > > > >> On Mon, Oct 20, 2014 at 1:51 PM, Pat Ferrel <[email protected]> > > wrote: > >> > >>> Is anyone else nervous about ignoring this issue or relying on > > non-build > >>> (hand run) test driven transitive dependency checking. I hope someone > >> else > >>> will chime in. > >>> > >>> As to running unit tests on a TEST_MASTER I’ll look into it. Can we set > >> up > >>> the build machine to do this? I’d feel better about eyeballing deps if > > we > >>> could have a TEST_MASTER automatically run during builds at Apache. > > Maybe > >>> the regular unit tests are OK for building locally ourselves. > >>> > >>>> > >>>> On Oct 20, 2014, at 12:23 PM, Dmitriy Lyubimov <[email protected]> > >>> wrote: > >>>> > >>>> On Mon, Oct 20, 2014 at 11:44 AM, Pat Ferrel <[email protected]> > >>> wrote: > >>>> > >>>>> Maybe a more fundamental issue is that we don’t know for sure > > whether > >> we > >>>>> have missing classes or not. The job.jar at least used the pom > >>> dependencies > >>>>> to guarantee every needed class was present. So the job.jar seems to > >>> solve > >>>>> the problem but may ship some unnecessary duplicate code, right? > >>>>> > >>>> > >>>> No, as i wrote spark doesn't work with job jar format. Neither as it > >>> turns > >>>> out more recent hadoop MR btw. > >>> > >>> Not speaking literally of the format. Spark understands jars and maven > >> can > >>> build one from transitive dependencies. > >>> > >>>> > >>>> Yes, this is A LOT of duplicate code (will take normally MINUTES to > >>> startup > >>>> tasks with all of it just on copy time). This is absolutely not the > > way > >>> to > >>>> go with this. > >>>> > >>> > >>> Lack of guarantee to load seems like a bigger problem than startup > > time. > >>> Clearly we can’t just ignore this. > >>> > >> > >> Nope. given highly iterative nature and dynamic task allocation in this > >> environment, one is looking to effects similar to Map Reduce. This is > not > >> the only reason why I never go to MR anymore, but that's one of main > > ones. > >> > >> How about experiment: why don't you create assembly that copies ALL > >> transitive dependencies in one folder, and then try to broadcast it from > >> single point (front end) to well... let's start with 20 machines. (of > >> course we ideally want to into 10^3 ..10^4 range -- but why bother if we > >> can't do it for 20). > >> > >> Or, heck, let's try to simply parallel-copy it between too machines 20 > >> times that are not collocated on the same subnet. > >> > >> > >>>> > >>>>> There may be any number of bugs waiting for the time we try running > >> on a > >>>>> node machine that doesn’t have some class in it’s classpath. > >>>> > >>>> > >>>> No. Assuming any given method is tested on all its execution paths, > >> there > >>>> will be no bugs. The bugs of that sort will only appear if the user > > is > >>>> using algebra directly and calls something that is not on the path, > >> from > >>>> the closure. In which case our answer to this is the same as for the > >>> solver > >>>> methodology developers -- use customized SparkConf while creating > >> context > >>>> to include stuff you really want. > >>>> > >>>> Also another right answer to this is that we probably should > > reasonably > >>>> provide the toolset here. For example, all the stats stuff found in R > >>> base > >>>> and R stat packages so the user is not compelled to go non-native. > >>>> > >>>> > >>> > >>> Huh? this is not true. The one I ran into was found by calling > > something > >>> in math from something in math-scala. It led outside and you can > >> encounter > >>> such things even in algebra. In fact you have no idea if these > > problems > >>> exists except for the fact you have used it a lot personally. > >>> > >> > >> > >> You ran it with your own code that never existed before. > >> > >> But there's difference between released Mahout code (which is what you > > are > >> working on) and the user code. Released code must run thru remote tests > > as > >> you suggested and thus guarantee there are no such problems with post > >> release code. > >> > >> For users, we only can provide a way for them to load stuff that they > >> decide to use. We don't have apriori knowledge what they will use. It is > >> the same thing that spark does, and the same thing that MR does, doesn't > >> it? > >> > >> Of course mahout should drop rigorously the stuff it doesn't load, from > > the > >> scala scope. No argue about that. In fact that's what i suggested as #1 > >> solution. But there's nothing much to do here but to go dependency > >> cleansing for math and spark code. Part of the reason there's so much is > >> because newer modules still bring in everything from mrLegacy. > >> > >> You are right in saying it is hard to guess what else dependencies are > in > >> the util/legacy code that are actually used. but that's not a > > justification > >> for brute force "copy them all" approach that virtually guarantees > > ruining > >> one of the foremost legacy issues this work intended to address. > >> > > > >
