Just ran tutorial on distributed (standalone) spark with 2 workers. seems to work like a charm. Can't reproduce any of the problems.
On Thu, Aug 14, 2014 at 7:35 PM, Dmitriy Lyubimov <[email protected]> wrote: > so as i suspected, picking up proper jars was broken in the head. > I did a quick patch (and a unit test) to assert proper jars are being > picked up to be shipped with the job, but i haven't tested with fully > distributed setup on 1.0.1 (haven't had time to set it for 1.0.1 yet). > > Spark people are releasing too fast. we barely have migrated to 1.0.1 and > they have already released 1.0.2. Well technically minor version should > not matter, users can change the pom and recompile with 1.0.2, my guess is > it should work. > > > On Thu, Aug 14, 2014 at 5:02 PM, Dmitriy Lyubimov <[email protected]> > wrote: > >> if errors appear in MASTER=local, then it is something wrong with spark >> binaries IMO. What i usually do is i do my own compilation of Spark with >> CDH4 (right now, it happens to be 4.3.1) , set SPARK_HOME and MAHOUT_HOME, >> compile mahout HEAD with maven install -Dskip.tests=true, and then just >> running `bin/mahout spark-shell` from the MAHOUT_HOME. >> >> >> >> >> On Thu, Aug 14, 2014 at 4:55 PM, Dmitriy Lyubimov <[email protected]> >> wrote: >> >>> just spend last 5 minutes to cut-and-paste the tutorial on HEAD with >>> 1.0.1 in local mode. Everything works without problem in local mode. What >>> was used for MASTER setting with this problem? >>> >>> >>> On Thu, Aug 14, 2014 at 11:29 AM, Dmitriy Lyubimov <[email protected]> >>> wrote: >>> >>>> for the same reason it may have screwed mahout context creation so that >>>> mahout jars are now not shpped to the backend properly. >>>> >>>> >>>> if the sole purpose of exercise is to get the totorial working, i'd >>>> suggest to just roll back to commit level before Anand's change and Spark >>>> 0.9.1 dependency, I am pretty sure it should work then. e.g. this one >>>> should be the last good commit (this requires Spark 0.9.1) >>>> >>>> commit 7a50a291b4598e9809f9acf609b92175ce7f953b >>>> Author: Dmitriy Lyubimov <[email protected]> >>>> Date: Wed Aug 6 12:30:51 2014 -0700 >>>> >>>> MAHOUT-1597: A + 1.0 (fixes) >>>> >>>> >>>> (use >>>> >>>> git reset 7a50a291 --hard >>>> >>>> to sync to this one) >>>> >>>> >>>> >>>> On Thu, Aug 14, 2014 at 11:20 AM, Dmitriy Lyubimov <[email protected]> >>>> wrote: >>>> >>>>> not sure either at this point. I guess PR from Anand renaming >>>>> artifacts created classpath problems but somehow it did not necessarily >>>>> manifest in my local tests since my maven repo holds the old ones as well. >>>>> >>>>> >>>>> On Thu, Aug 14, 2014 at 9:55 AM, Pat Ferrel <[email protected]> >>>>> wrote: >>>>> >>>>>> There are two problems here: >>>>>> >>>>>> 1) a bug in the mahout script. Just pushed your fix, thx. The jars >>>>>> got renamed is seems. >>>>>> >>>>>> 2) not sure what’s happening with the array serializer, maybe Dmitriy >>>>>> has an idea? >>>>>> >>>>>> >>>>>> On Aug 14, 2014, at 8:13 AM, Andrea Abelli < >>>>>> [email protected]> wrote: >>>>>> >>>>>> Hi Again >>>>>> >>>>>> new version of spark, new stack trace: >>>>>> http://pastebin.com/KPNZ3rYQ >>>>>> >>>>>> I'm going to have a look at it tomorrow. >>>>>> >>>>>> Good evening >>>>>> Andrea >>>>>> >>>>>> >>>>> >>>> >>> >> >
