To set up Eclipse for Spark you should install the Scala IDE plugins: http://scala-ide.org/download/current.html
Define your project in Maven with Scala plugins configured (you should be able to find documentation online) and import as an existing Maven project. The source code should be in src/main/scala but otherwise the project structure will be the same as you'd expect in Java. Nothing special is needed for Spark. Just define the desired Spark jars ( spark-core and possibly others, such as spark-sql) in your Maven POM as dependencies. You should scope these dependencies as provided, since they will automatically be on the classpath when you deploy your project to a Spark cluster. One thing to keep in mind is that Scala dependencies require separate jars for different versions of Scala, and it is convention to append the Scala version to the artifact ID. For example, if you are using Scala 2.11.x, your dependency will be spark-core_2.11 (look on search.maven.org if you're not sure). I think you can omit the Scala version if you're using SBT (not sure why you would, but some people seem to prefer it). Unit testing Spark is briefly explained in the programming guide <https://spark.apache.org/docs/latest/programming-guide.html#unit-testing>. To deploy using spark-submit you can build the jar using mvn package if and only if you don't have any non-Spark dependencies. Otherwise, the simplest thing is to build a jar with dependencies (typically using the assembly <http://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html> or shade <https://maven.apache.org/plugins/maven-shade-plugin/> plugins). On Wed, Jul 8, 2015 at 9:38 AM, Prateek . <prat...@aricent.com> wrote: > Hi > > > > I am beginner to scala and spark. I am trying to set up eclipse > environment to develop spark program in scala, then take it’s jar for > spark-submit. > > How shall I start? To start my task includes, setting up eclipse for > scala and spark, getting dependencies resolved, building project using > maven/sbt. > > Is there any good blog or documentation that is can follow. > > > > Thanks > > "DISCLAIMER: This message is proprietary to Aricent and is intended > solely for the use of the individual to whom it is addressed. It may > contain privileged or confidential information and should not be circulated > or used for any purpose other than for what it is intended. If you have > received this message in error, please notify the originator immediately. > If you are not the intended recipient, you are notified that you are > strictly prohibited from using, copying, altering, or disclosing the > contents of this message. Aricent accepts no responsibility for loss or > damage arising from the use of the information transmitted by this email > including damage from virus." >
