On Wed, Jun 4, 2014 at 12:31 PM, Matei Zaharia <matei.zaha...@gmail.com> wrote:
> Ah, sorry to hear you had more problems. Some thoughts on them: > There will always be more problems, 'tis the nature of coding. :-) I try not to bother the list until I've smacked my head against them for a few hours, so it's only the "most confusing" stuff I pour out here. I'm actually progressing pretty well. > (the streaming.Twitter ones especially) depend on there being a > "/mnt/spark" and "/mnt2/spark" directory (I think for java tempfiles?) and > those don't seem to exist out-of-the-box. > > I think this is a side-effect of the r3 instances not having those drives > mounted. Our setup script would normally create these directories. What was > the error? > > Oh, I went back to m1.large while those issues get sorted out. I decided I had enough problems without messing with that too. (seriously, why does Amazon do these things? It's like they _try_ to make the instances incompatible.) I forget the exact error, but it traced through createTempFile and it was fairly clear about the directory being missing. Things like "bin/run-example SparkPi" worked fine, but I'll bet twitter4j creates temp files, so "bin/run-example streaming.TwitterPopularTags" broke. What did you change log4j.properties to? It should be changed to say > log4j.rootCategory=WARN, console but maybe another log4j.properties is > somehow arriving on the classpath. This is definitely a common problem so > we need to add some explicit docs on it. > I seem to have this sorted out, don't ask me how. Once again I was probably editing things on the cluster master when I should have been editing the cluster controller, or vice versa. But, yeah, many of the examples just get lost in a sea of DAG INFO messages. > Are you going through http://spark.apache.org/docs/latest/quick-start.html? > You should be able to do just sbt package. Once you do that you don’t need > to deploy your application’s JAR to the cluster, just pass it to > spark-submit and it will automatically be sent over. > Ah, that answers another question I just asked elsewhere... Yup, I re-read pretty much every documentation page daily. And I'm making my way through every video. > > Meanwhile I'm learning scala... Great Turing's Ghost, it's the dream > language we've theorized about for years! I hadn't realized! > > Indeed, glad you’re enjoying it. > "Enjoying", not yet alas, I'm sure I'll get there. But I do understand the implications of a mixed functional-imperative language with closures and lambdas. That is serious voodoo. -- Jeremy Lee BCompSci(Hons) The Unorthodox Engineers
