ok Ted In sbt I have
name := "ImportCSV" version := "1.0" scalaVersion := "2.10.4" which ends up in importcsv_2.10-1.0.jar as part of *target/scala-2.10/importcsv_2.**10-1.0.jar* In mvn I have <version>1.0</version> <artifactId>scala</artifactId> Does it matter? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 15 March 2016 at 23:17, Ted Yu <yuzhih...@gmail.com> wrote: > <version>1.0</version> > ... > <artifactId>scala</artifactId> > > On Tue, Mar 15, 2016 at 4:14 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> An observation >> >> Once compiled with MVN the job submit works as follows: >> >> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages >> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master spark:// >> 50.140.197.217:7077 --executor-memory=12G --executor-cores=12 >> --num-executors=2 *target/scala-1.0.jar* >> >> With sbt it takes this form >> >> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages >> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master spark:// >> 50.140.197.217:7077 --executor-memory=12G --executor-cores=12 >> --num-executors=2 >> *target/scala-2.10/importcsv_2.10-1.0.jar* >> >> They both return the same results. However, why mvnjar file name is >> different (may be a naive question!)? >> >> thanks >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 15 March 2016 at 22:43, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >>> Many thanks Ted and thanks for heads up Jakob >>> >>> Just these two changes to dependencies >>> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-core*_2.10*</artifactId> >>> <version>1.5.1</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-sql*_2.10*</artifactId> >>> <version>1.5.1</version> >>> </dependency> >>> >>> >>> [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0 >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] BUILD SUCCESS >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] Total time: 01:04 min >>> [INFO] Finished at: 2016-03-15T22:55:08+00:00 >>> [INFO] Final Memory: 32M/1089M >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> >>> On 15 March 2016 at 22:18, Jakob Odersky <ja...@odersky.com> wrote: >>> >>>> Hi Mich, >>>> probably unrelated to the current error you're seeing, however the >>>> following dependencies will bite you later: >>>> <artifactId>spark-hive_2.10</artifactId> >>>> <artifactId>spark-csv_2.11</artifactId> >>>> the problem here is that you're using libraries built for different >>>> Scala binary versions (the numbers after the underscore). The simple >>>> fix here is to specify the Scala binary version you're project builds >>>> for (2.10 in your case, however note that version is EOL, you should >>>> upgrade to scala 2.11.8 if possible). >>>> >>>> On a side note, sbt takes care of handling correct scala versions for >>>> you (the double %% actually is a shorthand for appending >>>> "_scalaBinaryVersion" to your dependency). It also enables you to >>>> build and publish your project seamlessly against multiple versions. I >>>> would strongly recommend to use it in Scala projects. >>>> >>>> cheers, >>>> --Jakob >>>> >>>> >>>> >>>> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh >>>> <mich.talebza...@gmail.com> wrote: >>>> > Hi, >>>> > >>>> > I normally use sbt and using this sbt file works fine for me >>>> > >>>> > cat ImportCSV.sbt >>>> > name := "ImportCSV" >>>> > version := "1.0" >>>> > scalaVersion := "2.10.4" >>>> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" >>>> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" >>>> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >>>> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % "1.3.0" >>>> > >>>> > This is my first trial using Mavan and pom >>>> > >>>> > >>>> > my pom.xml file looks like this but throws error at build >>>> > >>>> > >>>> > [DEBUG] com.univocity:univocity-parsers:jar:1.5.1:compile >>>> > [INFO] >>>> > >>>> ------------------------------------------------------------------------ >>>> > [INFO] BUILD FAILURE >>>> > [INFO] >>>> > >>>> ------------------------------------------------------------------------ >>>> > [INFO] Total time: 1.326 s >>>> > [INFO] Finished at: 2016-03-15T22:17:29+00:00 >>>> > [INFO] Final Memory: 14M/455M >>>> > [INFO] >>>> > >>>> ------------------------------------------------------------------------ >>>> > [ERROR] Failed to execute goal on project scala: Could not resolve >>>> > dependencies for project spark:scala:jar:1.0: The following artifacts >>>> could >>>> > not be resolved: org.apache.spark:spark-core:jar:1.5.1, >>>> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find >>>> > org.apache.spark:spark-core:jar:1.5.1 in >>>> > https://repo.maven.apache.org/maven2 was cached in the local >>>> repository, >>>> > resolution will not be reattempted until the update interval of >>>> central has >>>> > elapsed or updates are forced -> [Help 1] >>>> > >>>> > >>>> > My pom file is >>>> > >>>> > >>>> > cat pom.xml >>>> > <project xmlns="http://maven.apache.org/POM/4.0.0"; >>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>>> > http://maven.apache.org/maven-v4_0_0.xsd";> >>>> > <modelVersion>4.0.0</modelVersion> >>>> > <groupId>spark</groupId> >>>> > <version>1.0</version> >>>> > <name>${project.artifactId}</name> >>>> > >>>> > <properties> >>>> > <maven.compiler.source>1.7</maven.compiler.source> >>>> > <maven.compiler.target>1.7</maven.compiler.target> >>>> > <encoding>UTF-8</encoding> >>>> > <scala.version>2.10.4</scala.version> >>>> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>>> > </properties> >>>> > >>>> > <dependencies> >>>> > <dependency> >>>> > <groupId>org.scala-lang</groupId> >>>> > <artifactId>scala-library</artifactId> >>>> > <version>2.10.2</version> >>>> > </dependency> >>>> > <dependency> >>>> > <groupId>org.apache.spark</groupId> >>>> > <artifactId>spark-core</artifactId> >>>> > <version>1.5.1</version> >>>> > </dependency> >>>> > <dependency> >>>> > <groupId>org.apache.spark</groupId> >>>> > <artifactId>spark-sql</artifactId> >>>> > <version>1.5.1</version> >>>> > </dependency> >>>> > <dependency> >>>> > <groupId>org.apache.spark</groupId> >>>> > <artifactId>spark-hive_2.10</artifactId> >>>> > <version>1.5.0</version> >>>> > </dependency> >>>> > <dependency> >>>> > <groupId>com.databricks</groupId> >>>> > <artifactId>spark-csv_2.11</artifactId> >>>> > <version>1.3.0</version> >>>> > </dependency> >>>> > </dependencies> >>>> > >>>> > <build> >>>> > <sourceDirectory>src/main/scala</sourceDirectory> >>>> > <plugins> >>>> > <plugin> >>>> > <groupId>org.scala-tools</groupId> >>>> > <artifactId>maven-scala-plugin</artifactId> >>>> > <version>${maven-scala-plugin.version}</version> >>>> > <executions> >>>> > <execution> >>>> > <goals> >>>> > <goal>compile</goal> >>>> > </goals> >>>> > </execution> >>>> > </executions> >>>> > <configuration> >>>> > <jvmArgs> >>>> > <jvmArg>-Xms64m</jvmArg> >>>> > <jvmArg>-Xmx1024m</jvmArg> >>>> > </jvmArgs> >>>> > </configuration> >>>> > </plugin> >>>> > <plugin> >>>> > <groupId>org.apache.maven.plugins</groupId> >>>> > <artifactId>maven-shade-plugin</artifactId> >>>> > <version>1.6</version> >>>> > <executions> >>>> > <execution> >>>> > <phase>package</phase> >>>> > <goals> >>>> > <goal>shade</goal> >>>> > </goals> >>>> > <configuration> >>>> > <filters> >>>> > <filter> >>>> > <artifact>*:*</artifact> >>>> > <excludes> >>>> > <exclude>META-INF/*.SF</exclude> >>>> > <exclude>META-INF/*.DSA</exclude> >>>> > <exclude>META-INF/*.RSA</exclude> >>>> > </excludes> >>>> > </filter> >>>> > </filters> >>>> > <transformers> >>>> > <transformer >>>> > >>>> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> >>>> > <mainClass>com.group.id.Launcher1</mainClass> >>>> > </transformer> >>>> > </transformers> >>>> > </configuration> >>>> > </execution> >>>> > </executions> >>>> > </plugin> >>>> > </plugins> >>>> > </build> >>>> > >>>> > <artifactId>scala</artifactId> >>>> > </project> >>>> > >>>> > >>>> > I am sure I have omitted something? >>>> > >>>> > >>>> > Thanks >>>> > >>>> > >>>> > Dr Mich Talebzadeh >>>> > >>>> > >>>> > >>>> > LinkedIn >>>> > >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> > >>>> > >>>> > >>>> > http://talebzadehmich.wordpress.com >>>> > >>>> > >>>> >>> >>> >> >
