Feel free to adjust artifact Id and version in maven. They're under your control.
> On Mar 15, 2016, at 4:27 PM, Mich Talebzadeh <[email protected]> > wrote: > > ok Ted > > In sbt I have > > name := "ImportCSV" > version := "1.0" > scalaVersion := "2.10.4" > > which ends up in importcsv_2.10-1.0.jar as part of > target/scala-2.10/importcsv_2.10-1.0.jar > > In mvn I have > > <version>1.0</version> > <artifactId>scala</artifactId> > > > Does it matter? > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > http://talebzadehmich.wordpress.com > > >> On 15 March 2016 at 23:17, Ted Yu <[email protected]> wrote: >> <version>1.0</version> >> ... >> <artifactId>scala</artifactId> >> >>> On Tue, Mar 15, 2016 at 4:14 PM, Mich Talebzadeh >>> <[email protected]> wrote: >>> An observation >>> >>> Once compiled with MVN the job submit works as follows: >>> >>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages >>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master >>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12 >>> --num-executors=2 target/scala-1.0.jar >>> >>> With sbt it takes this form >>> >>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages >>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master >>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12 >>> --num-executors=2 target/scala-2.10/importcsv_2.10-1.0.jar >>> >>> They both return the same results. However, why mvnjar file name is >>> different (may be a naive question!)? >>> >>> thanks >>> >>> >>> Dr Mich Talebzadeh >>> >>> LinkedIn >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>>> On 15 March 2016 at 22:43, Mich Talebzadeh <[email protected]> >>>> wrote: >>>> Many thanks Ted and thanks for heads up Jakob >>>> >>>> Just these two changes to dependencies >>>> >>>> <dependency> >>>> <groupId>org.apache.spark</groupId> >>>> <artifactId>spark-core_2.10</artifactId> >>>> <version>1.5.1</version> >>>> </dependency> >>>> <dependency> >>>> <groupId>org.apache.spark</groupId> >>>> <artifactId>spark-sql_2.10</artifactId> >>>> <version>1.5.1</version> >>>> </dependency> >>>> >>>> >>>> [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0 >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] BUILD SUCCESS >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] Total time: 01:04 min >>>> [INFO] Finished at: 2016-03-15T22:55:08+00:00 >>>> [INFO] Final Memory: 32M/1089M >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> >>>> Dr Mich Talebzadeh >>>> >>>> LinkedIn >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>>> On 15 March 2016 at 22:18, Jakob Odersky <[email protected]> wrote: >>>>> Hi Mich, >>>>> probably unrelated to the current error you're seeing, however the >>>>> following dependencies will bite you later: >>>>> <artifactId>spark-hive_2.10</artifactId> >>>>> <artifactId>spark-csv_2.11</artifactId> >>>>> the problem here is that you're using libraries built for different >>>>> Scala binary versions (the numbers after the underscore). The simple >>>>> fix here is to specify the Scala binary version you're project builds >>>>> for (2.10 in your case, however note that version is EOL, you should >>>>> upgrade to scala 2.11.8 if possible). >>>>> >>>>> On a side note, sbt takes care of handling correct scala versions for >>>>> you (the double %% actually is a shorthand for appending >>>>> "_scalaBinaryVersion" to your dependency). It also enables you to >>>>> build and publish your project seamlessly against multiple versions. I >>>>> would strongly recommend to use it in Scala projects. >>>>> >>>>> cheers, >>>>> --Jakob >>>>> >>>>> >>>>> >>>>> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh >>>>> <[email protected]> wrote: >>>>> > Hi, >>>>> > >>>>> > I normally use sbt and using this sbt file works fine for me >>>>> > >>>>> > cat ImportCSV.sbt >>>>> > name := "ImportCSV" >>>>> > version := "1.0" >>>>> > scalaVersion := "2.10.4" >>>>> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" >>>>> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" >>>>> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >>>>> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % "1.3.0" >>>>> > >>>>> > This is my first trial using Mavan and pom >>>>> > >>>>> > >>>>> > my pom.xml file looks like this but throws error at build >>>>> > >>>>> > >>>>> > [DEBUG] com.univocity:univocity-parsers:jar:1.5.1:compile >>>>> > [INFO] >>>>> > ------------------------------------------------------------------------ >>>>> > [INFO] BUILD FAILURE >>>>> > [INFO] >>>>> > ------------------------------------------------------------------------ >>>>> > [INFO] Total time: 1.326 s >>>>> > [INFO] Finished at: 2016-03-15T22:17:29+00:00 >>>>> > [INFO] Final Memory: 14M/455M >>>>> > [INFO] >>>>> > ------------------------------------------------------------------------ >>>>> > [ERROR] Failed to execute goal on project scala: Could not resolve >>>>> > dependencies for project spark:scala:jar:1.0: The following artifacts >>>>> > could >>>>> > not be resolved: org.apache.spark:spark-core:jar:1.5.1, >>>>> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find >>>>> > org.apache.spark:spark-core:jar:1.5.1 in >>>>> > https://repo.maven.apache.org/maven2 was cached in the local repository, >>>>> > resolution will not be reattempted until the update interval of central >>>>> > has >>>>> > elapsed or updates are forced -> [Help 1] >>>>> > >>>>> > >>>>> > My pom file is >>>>> > >>>>> > >>>>> > cat pom.xml >>>>> > <project xmlns="http://maven.apache.org/POM/4.0.0"; >>>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>>>> > http://maven.apache.org/maven-v4_0_0.xsd";> >>>>> > <modelVersion>4.0.0</modelVersion> >>>>> > <groupId>spark</groupId> >>>>> > <version>1.0</version> >>>>> > <name>${project.artifactId}</name> >>>>> > >>>>> > <properties> >>>>> > <maven.compiler.source>1.7</maven.compiler.source> >>>>> > <maven.compiler.target>1.7</maven.compiler.target> >>>>> > <encoding>UTF-8</encoding> >>>>> > <scala.version>2.10.4</scala.version> >>>>> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>>>> > </properties> >>>>> > >>>>> > <dependencies> >>>>> > <dependency> >>>>> > <groupId>org.scala-lang</groupId> >>>>> > <artifactId>scala-library</artifactId> >>>>> > <version>2.10.2</version> >>>>> > </dependency> >>>>> > <dependency> >>>>> > <groupId>org.apache.spark</groupId> >>>>> > <artifactId>spark-core</artifactId> >>>>> > <version>1.5.1</version> >>>>> > </dependency> >>>>> > <dependency> >>>>> > <groupId>org.apache.spark</groupId> >>>>> > <artifactId>spark-sql</artifactId> >>>>> > <version>1.5.1</version> >>>>> > </dependency> >>>>> > <dependency> >>>>> > <groupId>org.apache.spark</groupId> >>>>> > <artifactId>spark-hive_2.10</artifactId> >>>>> > <version>1.5.0</version> >>>>> > </dependency> >>>>> > <dependency> >>>>> > <groupId>com.databricks</groupId> >>>>> > <artifactId>spark-csv_2.11</artifactId> >>>>> > <version>1.3.0</version> >>>>> > </dependency> >>>>> > </dependencies> >>>>> > >>>>> > <build> >>>>> > <sourceDirectory>src/main/scala</sourceDirectory> >>>>> > <plugins> >>>>> > <plugin> >>>>> > <groupId>org.scala-tools</groupId> >>>>> > <artifactId>maven-scala-plugin</artifactId> >>>>> > <version>${maven-scala-plugin.version}</version> >>>>> > <executions> >>>>> > <execution> >>>>> > <goals> >>>>> > <goal>compile</goal> >>>>> > </goals> >>>>> > </execution> >>>>> > </executions> >>>>> > <configuration> >>>>> > <jvmArgs> >>>>> > <jvmArg>-Xms64m</jvmArg> >>>>> > <jvmArg>-Xmx1024m</jvmArg> >>>>> > </jvmArgs> >>>>> > </configuration> >>>>> > </plugin> >>>>> > <plugin> >>>>> > <groupId>org.apache.maven.plugins</groupId> >>>>> > <artifactId>maven-shade-plugin</artifactId> >>>>> > <version>1.6</version> >>>>> > <executions> >>>>> > <execution> >>>>> > <phase>package</phase> >>>>> > <goals> >>>>> > <goal>shade</goal> >>>>> > </goals> >>>>> > <configuration> >>>>> > <filters> >>>>> > <filter> >>>>> > <artifact>*:*</artifact> >>>>> > <excludes> >>>>> > <exclude>META-INF/*.SF</exclude> >>>>> > <exclude>META-INF/*.DSA</exclude> >>>>> > <exclude>META-INF/*.RSA</exclude> >>>>> > </excludes> >>>>> > </filter> >>>>> > </filters> >>>>> > <transformers> >>>>> > <transformer >>>>> > implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> >>>>> > <mainClass>com.group.id.Launcher1</mainClass> >>>>> > </transformer> >>>>> > </transformers> >>>>> > </configuration> >>>>> > </execution> >>>>> > </executions> >>>>> > </plugin> >>>>> > </plugins> >>>>> > </build> >>>>> > >>>>> > <artifactId>scala</artifactId> >>>>> > </project> >>>>> > >>>>> > >>>>> > I am sure I have omitted something? >>>>> > >>>>> > >>>>> > Thanks >>>>> > >>>>> > >>>>> > Dr Mich Talebzadeh >>>>> > >>>>> > >>>>> > >>>>> > LinkedIn >>>>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> > >>>>> > >>>>> > >>>>> > http://talebzadehmich.wordpress.com >>>>> > >>>>> > >
