Getting started with spark-scala developemnt in eclipse.
Hi I am beginner to scala and spark. I am trying to set up eclipse environment to develop spark program in scala, then take it's jar for spark-submit. How shall I start? To start my task includes, setting up eclipse for scala and spark, getting dependencies resolved, building project using maven/sbt. Is there any good blog or documentation that is can follow. Thanks DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.
Re: Getting started with spark-scala developemnt in eclipse.
Hello Prateek, I started with getting the pre built binaries so as to skip the hassle of building them from scratch. I am not familiar with scala so can't comment on it. I have documented my experiences on my blog www.edumine.wordpress.com Perhaps it might be useful to you. On 08-Jul-2015 9:39 PM, Prateek . prat...@aricent.com wrote: Hi I am beginner to scala and spark. I am trying to set up eclipse environment to develop spark program in scala, then take it’s jar for spark-submit. How shall I start? To start my task includes, setting up eclipse for scala and spark, getting dependencies resolved, building project using maven/sbt. Is there any good blog or documentation that is can follow. Thanks DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.
Re: Getting started with spark-scala developemnt in eclipse.
To set up Eclipse for Spark you should install the Scala IDE plugins: http://scala-ide.org/download/current.html Define your project in Maven with Scala plugins configured (you should be able to find documentation online) and import as an existing Maven project. The source code should be in src/main/scala but otherwise the project structure will be the same as you'd expect in Java. Nothing special is needed for Spark. Just define the desired Spark jars ( spark-core and possibly others, such as spark-sql) in your Maven POM as dependencies. You should scope these dependencies as provided, since they will automatically be on the classpath when you deploy your project to a Spark cluster. One thing to keep in mind is that Scala dependencies require separate jars for different versions of Scala, and it is convention to append the Scala version to the artifact ID. For example, if you are using Scala 2.11.x, your dependency will be spark-core_2.11 (look on search.maven.org if you're not sure). I think you can omit the Scala version if you're using SBT (not sure why you would, but some people seem to prefer it). Unit testing Spark is briefly explained in the programming guide https://spark.apache.org/docs/latest/programming-guide.html#unit-testing. To deploy using spark-submit you can build the jar using mvn package if and only if you don't have any non-Spark dependencies. Otherwise, the simplest thing is to build a jar with dependencies (typically using the assembly http://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html or shade https://maven.apache.org/plugins/maven-shade-plugin/ plugins). On Wed, Jul 8, 2015 at 9:38 AM, Prateek . prat...@aricent.com wrote: Hi I am beginner to scala and spark. I am trying to set up eclipse environment to develop spark program in scala, then take it’s jar for spark-submit. How shall I start? To start my task includes, setting up eclipse for scala and spark, getting dependencies resolved, building project using maven/sbt. Is there any good blog or documentation that is can follow. Thanks DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.
Re: Getting started with spark-scala developemnt in eclipse.
Take a look at https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-Eclipse On Wed, Jul 8, 2015 at 7:47 AM, Daniel Siegmann daniel.siegm...@teamaol.com wrote: To set up Eclipse for Spark you should install the Scala IDE plugins: http://scala-ide.org/download/current.html Define your project in Maven with Scala plugins configured (you should be able to find documentation online) and import as an existing Maven project. The source code should be in src/main/scala but otherwise the project structure will be the same as you'd expect in Java. Nothing special is needed for Spark. Just define the desired Spark jars ( spark-core and possibly others, such as spark-sql) in your Maven POM as dependencies. You should scope these dependencies as provided, since they will automatically be on the classpath when you deploy your project to a Spark cluster. One thing to keep in mind is that Scala dependencies require separate jars for different versions of Scala, and it is convention to append the Scala version to the artifact ID. For example, if you are using Scala 2.11.x, your dependency will be spark-core_2.11 (look on search.maven.org if you're not sure). I think you can omit the Scala version if you're using SBT (not sure why you would, but some people seem to prefer it). Unit testing Spark is briefly explained in the programming guide https://spark.apache.org/docs/latest/programming-guide.html#unit-testing . To deploy using spark-submit you can build the jar using mvn package if and only if you don't have any non-Spark dependencies. Otherwise, the simplest thing is to build a jar with dependencies (typically using the assembly http://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html or shade https://maven.apache.org/plugins/maven-shade-plugin/ plugins). On Wed, Jul 8, 2015 at 9:38 AM, Prateek . prat...@aricent.com wrote: Hi I am beginner to scala and spark. I am trying to set up eclipse environment to develop spark program in scala, then take it’s jar for spark-submit. How shall I start? To start my task includes, setting up eclipse for scala and spark, getting dependencies resolved, building project using maven/sbt. Is there any good blog or documentation that is can follow. Thanks DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.
