Yeah, that's exactly what I did. Unfortunately it doesn't work: $SPARK_HOME/sbt/sbt package awk: cmd. line:1: fatal: cannot open file `./project/build.properties' for reading (No such file or directory) Attempting to fetch sbt /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or directory /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or directory Our attempt to download sbt locally to sbt/sbt-launch-.jar failed. Please install sbt manually from http://www.scala-sbt.org/
On Mon, Mar 24, 2014 at 4:25 PM, Ognen Duzlevski < og...@plainvanillagames.com> wrote: > You can use any sbt on your machine, including the one that comes with > spark. For example, try: > > ~/path_to_spark/sbt/sbt compile > ~/path_to_spark/sbt/sbt run <arguments> > > Or you can just add that to your PATH by: > > export $PATH=$PATH:~/path_to_spark/sbt > > To make it permanent, you can add it to your ~/.bashrc or ~/.bash_profile > or ??? depending on the system you are using. If you are on Windows, sorry, > I can't offer any help there ;) > > Ognen > > > On 3/24/14, 3:16 PM, Diana Carroll wrote: > > Thanks Ongen. > > Unfortunately I'm not able to follow your instructions either. In > particular: > >> >> sbt compile >> sbt run <arguments if any> > > > This doesn't work for me because there's no program on my path called > "sbt". The instructions in the Quick Start guide are specific that I > should call "$SPARK_HOME/sbt/sbt". I don't have any other executable on my > system called "sbt". > > Did you download and install sbt separately? In following the Quick > Start guide, that was not stated as a requirement, and I'm trying to run > through the guide word for word. > > Diana > > > On Mon, Mar 24, 2014 at 4:12 PM, Ognen Duzlevski < > og...@plainvanillagames.com> wrote: > >> Diana, >> >> Anywhere on the filesystem you have read/write access (you need not be in >> your spark home directory): >> >> mkdir myproject >> cd myproject >> mkdir project >> mkdir target >> mkdir -p src/main/scala >> cp $mypath/$mymysource.scala src/main/scala/ >> cp $mypath/myproject.sbt . >> >> Make sure that myproject.sbt has the following in it: >> >> name := "I NEED A NAME!" >> >> version := "I NEED A VERSION!" >> >> scalaVersion := "2.10.3" >> >> libraryDependencies += "org.apache.spark" % "spark-core_2.10" % >> "0.9.0-incubating" >> >> If you will be using Hadoop/HDFS functionality you will need the below >> line also >> >> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0" >> >> The above assumes you are using Spark 0.9 and Scala 2.10.3. If you are >> using 0.8.1 - adjust appropriately. >> >> That's it. Now you can do >> >> sbt compile >> sbt run <arguments if any> >> >> You can also do >> sbt package to produce a jar file of your code which you can then add to >> the SparkContext at runtime. >> >> In a more complicated project you may need to have a bit more involved >> hierarchy like com.github.dianacarroll which will then translate to >> src/main/scala/com/github/dianacarroll/ where you can put your multiple >> .scala files which will then have to be a part of a package >> com.github.dianacarroll (you can just put that as your first line in each >> of these scala files). I am new to Java/Scala so this is how I do it. More >> educated Java/Scala programmers may tell you otherwise ;) >> >> You can get more complicated with the sbt project subrirectory but you >> can read independently about sbt and what it can do, above is the bare >> minimum. >> >> Let me know if that helped. >> Ognen >> >> >> On 3/24/14, 2:44 PM, Diana Carroll wrote: >> >>> Has anyone successfully followed the instructions on the Quick Start >>> page of the Spark home page to run a "standalone" Scala application? I >>> can't, and I figure I must be missing something obvious! >>> >>> I'm trying to follow the instructions here as close to "word for word" >>> as possible: >>> >>> http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala >>> >>> 1. The instructions don't say what directory to create my test >>> application in, but later I'm instructed to run "sbt/sbt" so I conclude >>> that my working directory must be $SPARK_HOME. (Temporarily ignoring that >>> it is a little weird to be working directly in the Spark distro.) >>> >>> 2. Create $SPARK_HOME/mysparktest/src/main/scala/SimpleApp.scala. >>> Copy&paste in the code from the instructions exactly, replacing >>> YOUR_SPARK_HOME with my spark home path. >>> >>> 3. Create $SPARK_HOME/mysparktest/simple.sbt. Copy&paste in the sbt >>> file from the instructions >>> >>> 4. From the $SPARK_HOME I run "sbt/sbt package". It runs through the >>> ENTIRE Spark project! This takes several minutes, and at the end, it says >>> "Done packaging". unfortunately, there's nothing in the >>> $SPARK_HOME/mysparktest/ folder other than what I already had there. >>> >>> (Just for fun, I also did what I thought was more logical, which is set >>> my working directory to $SPARK_HOME/mysparktest, and but >>> $SPARK_HOME/sbt/sbt package, but that was even less successful: I got an >>> error: >>> awk: cmd. line:1: fatal: cannot open file `./project/build.properties' >>> for reading (No such file or directory) >>> Attempting to fetch sbt >>> /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or >>> directory >>> /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or >>> directory >>> Our attempt to download sbt locally to sbt/sbt-launch-.jar failed. >>> Please install sbt manually from http://www.scala-sbt.org/ >>> >>> >>> So, help? I'm sure these instructions work because people are following >>> them every day, but I can't tell what they are supposed to do. >>> >>> Thanks! >>> Diana >>> >> >> > > -- > "A distributed system is one in which the failure of a computer you didn't > even know existed can render your own computer unusable" > -- Leslie Lamport > >
