<version>1.0</version> ... <artifactId>scala</artifactId>
On Tue, Mar 15, 2016 at 4:14 PM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > An observation > > Once compiled with MVN the job submit works as follows: > > + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages > com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master spark:// > 50.140.197.217:7077 --executor-memory=12G --executor-cores=12 > --num-executors=2 *target/scala-1.0.jar* > > With sbt it takes this form > > + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages > com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master spark:// > 50.140.197.217:7077 --executor-memory=12G --executor-cores=12 > --num-executors=2 > *target/scala-2.10/importcsv_2.10-1.0.jar* > > They both return the same results. However, why mvnjar file name is > different (may be a naive question!)? > > thanks > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 15 March 2016 at 22:43, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Many thanks Ted and thanks for heads up Jakob >> >> Just these two changes to dependencies >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-core*_2.10*</artifactId> >> <version>1.5.1</version> >> </dependency> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-sql*_2.10*</artifactId> >> <version>1.5.1</version> >> </dependency> >> >> >> [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0 >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD SUCCESS >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 01:04 min >> [INFO] Finished at: 2016-03-15T22:55:08+00:00 >> [INFO] Final Memory: 32M/1089M >> [INFO] >> ------------------------------------------------------------------------ >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 15 March 2016 at 22:18, Jakob Odersky <ja...@odersky.com> wrote: >> >>> Hi Mich, >>> probably unrelated to the current error you're seeing, however the >>> following dependencies will bite you later: >>> <artifactId>spark-hive_2.10</artifactId> >>> <artifactId>spark-csv_2.11</artifactId> >>> the problem here is that you're using libraries built for different >>> Scala binary versions (the numbers after the underscore). The simple >>> fix here is to specify the Scala binary version you're project builds >>> for (2.10 in your case, however note that version is EOL, you should >>> upgrade to scala 2.11.8 if possible). >>> >>> On a side note, sbt takes care of handling correct scala versions for >>> you (the double %% actually is a shorthand for appending >>> "_scalaBinaryVersion" to your dependency). It also enables you to >>> build and publish your project seamlessly against multiple versions. I >>> would strongly recommend to use it in Scala projects. >>> >>> cheers, >>> --Jakob >>> >>> >>> >>> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh >>> <mich.talebza...@gmail.com> wrote: >>> > Hi, >>> > >>> > I normally use sbt and using this sbt file works fine for me >>> > >>> > cat ImportCSV.sbt >>> > name := "ImportCSV" >>> > version := "1.0" >>> > scalaVersion := "2.10.4" >>> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" >>> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" >>> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >>> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % "1.3.0" >>> > >>> > This is my first trial using Mavan and pom >>> > >>> > >>> > my pom.xml file looks like this but throws error at build >>> > >>> > >>> > [DEBUG] com.univocity:univocity-parsers:jar:1.5.1:compile >>> > [INFO] >>> > >>> ------------------------------------------------------------------------ >>> > [INFO] BUILD FAILURE >>> > [INFO] >>> > >>> ------------------------------------------------------------------------ >>> > [INFO] Total time: 1.326 s >>> > [INFO] Finished at: 2016-03-15T22:17:29+00:00 >>> > [INFO] Final Memory: 14M/455M >>> > [INFO] >>> > >>> ------------------------------------------------------------------------ >>> > [ERROR] Failed to execute goal on project scala: Could not resolve >>> > dependencies for project spark:scala:jar:1.0: The following artifacts >>> could >>> > not be resolved: org.apache.spark:spark-core:jar:1.5.1, >>> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find >>> > org.apache.spark:spark-core:jar:1.5.1 in >>> > https://repo.maven.apache.org/maven2 was cached in the local >>> repository, >>> > resolution will not be reattempted until the update interval of >>> central has >>> > elapsed or updates are forced -> [Help 1] >>> > >>> > >>> > My pom file is >>> > >>> > >>> > cat pom.xml >>> > <project xmlns="http://maven.apache.org/POM/4.0.0"; >>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>> > http://maven.apache.org/maven-v4_0_0.xsd";> >>> > <modelVersion>4.0.0</modelVersion> >>> > <groupId>spark</groupId> >>> > <version>1.0</version> >>> > <name>${project.artifactId}</name> >>> > >>> > <properties> >>> > <maven.compiler.source>1.7</maven.compiler.source> >>> > <maven.compiler.target>1.7</maven.compiler.target> >>> > <encoding>UTF-8</encoding> >>> > <scala.version>2.10.4</scala.version> >>> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>> > </properties> >>> > >>> > <dependencies> >>> > <dependency> >>> > <groupId>org.scala-lang</groupId> >>> > <artifactId>scala-library</artifactId> >>> > <version>2.10.2</version> >>> > </dependency> >>> > <dependency> >>> > <groupId>org.apache.spark</groupId> >>> > <artifactId>spark-core</artifactId> >>> > <version>1.5.1</version> >>> > </dependency> >>> > <dependency> >>> > <groupId>org.apache.spark</groupId> >>> > <artifactId>spark-sql</artifactId> >>> > <version>1.5.1</version> >>> > </dependency> >>> > <dependency> >>> > <groupId>org.apache.spark</groupId> >>> > <artifactId>spark-hive_2.10</artifactId> >>> > <version>1.5.0</version> >>> > </dependency> >>> > <dependency> >>> > <groupId>com.databricks</groupId> >>> > <artifactId>spark-csv_2.11</artifactId> >>> > <version>1.3.0</version> >>> > </dependency> >>> > </dependencies> >>> > >>> > <build> >>> > <sourceDirectory>src/main/scala</sourceDirectory> >>> > <plugins> >>> > <plugin> >>> > <groupId>org.scala-tools</groupId> >>> > <artifactId>maven-scala-plugin</artifactId> >>> > <version>${maven-scala-plugin.version}</version> >>> > <executions> >>> > <execution> >>> > <goals> >>> > <goal>compile</goal> >>> > </goals> >>> > </execution> >>> > </executions> >>> > <configuration> >>> > <jvmArgs> >>> > <jvmArg>-Xms64m</jvmArg> >>> > <jvmArg>-Xmx1024m</jvmArg> >>> > </jvmArgs> >>> > </configuration> >>> > </plugin> >>> > <plugin> >>> > <groupId>org.apache.maven.plugins</groupId> >>> > <artifactId>maven-shade-plugin</artifactId> >>> > <version>1.6</version> >>> > <executions> >>> > <execution> >>> > <phase>package</phase> >>> > <goals> >>> > <goal>shade</goal> >>> > </goals> >>> > <configuration> >>> > <filters> >>> > <filter> >>> > <artifact>*:*</artifact> >>> > <excludes> >>> > <exclude>META-INF/*.SF</exclude> >>> > <exclude>META-INF/*.DSA</exclude> >>> > <exclude>META-INF/*.RSA</exclude> >>> > </excludes> >>> > </filter> >>> > </filters> >>> > <transformers> >>> > <transformer >>> > >>> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> >>> > <mainClass>com.group.id.Launcher1</mainClass> >>> > </transformer> >>> > </transformers> >>> > </configuration> >>> > </execution> >>> > </executions> >>> > </plugin> >>> > </plugins> >>> > </build> >>> > >>> > <artifactId>scala</artifactId> >>> > </project> >>> > >>> > >>> > I am sure I have omitted something? >>> > >>> > >>> > Thanks >>> > >>> > >>> > Dr Mich Talebzadeh >>> > >>> > >>> > >>> > LinkedIn >>> > >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> > >>> > >>> > >>> > http://talebzadehmich.wordpress.com >>> > >>> > >>> >> >> >
