in mvn the build mvn package will look for a file called pom.xml in sbt the build sbt package will look for a file called anything.smt
It works Keep it simple I will write a ksh script that will create both generic and sbt files on the fly in the correct directory (at the top of the tree) and remove them after the job finished. That will keep audit people happy as well 😁 HTh Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 15 March 2016 at 23:34, Jakob Odersky <ja...@odersky.com> wrote: > The artifactId in maven basically (in a simple case) corresponds to name > in sbt. > > Note however that you will manually need to append the > _scalaBinaryVersion to the artifactId in case you would like to build > against multiple scala versions (otherwise maven will overwrite the > generated jar with the latest one). > > > On Tue, Mar 15, 2016 at 4:27 PM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > ok Ted > > > > In sbt I have > > > > name := "ImportCSV" > > version := "1.0" > > scalaVersion := "2.10.4" > > > > which ends up in importcsv_2.10-1.0.jar as part of > > target/scala-2.10/importcsv_2.10-1.0.jar > > > > In mvn I have > > > > <version>1.0</version> > > <artifactId>scala</artifactId> > > > > > > Does it matter? > > > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > > > > > On 15 March 2016 at 23:17, Ted Yu <yuzhih...@gmail.com> wrote: > >> > >> <version>1.0</version> > >> ... > >> <artifactId>scala</artifactId> > >> > >> On Tue, Mar 15, 2016 at 4:14 PM, Mich Talebzadeh > >> <mich.talebza...@gmail.com> wrote: > >>> > >>> An observation > >>> > >>> Once compiled with MVN the job submit works as follows: > >>> > >>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages > >>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master > >>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12 > >>> --num-executors=2 target/scala-1.0.jar > >>> > >>> With sbt it takes this form > >>> > >>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages > >>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master > >>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12 > >>> --num-executors=2 target/scala-2.10/importcsv_2.10-1.0.jar > >>> > >>> They both return the same results. However, why mvnjar file name is > >>> different (may be a naive question!)? > >>> > >>> thanks > >>> > >>> > >>> Dr Mich Talebzadeh > >>> > >>> > >>> > >>> LinkedIn > >>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > >>> > >>> > >>> http://talebzadehmich.wordpress.com > >>> > >>> > >>> > >>> > >>> On 15 March 2016 at 22:43, Mich Talebzadeh <mich.talebza...@gmail.com> > >>> wrote: > >>>> > >>>> Many thanks Ted and thanks for heads up Jakob > >>>> > >>>> Just these two changes to dependencies > >>>> > >>>> <dependency> > >>>> <groupId>org.apache.spark</groupId> > >>>> <artifactId>spark-core_2.10</artifactId> > >>>> <version>1.5.1</version> > >>>> </dependency> > >>>> <dependency> > >>>> <groupId>org.apache.spark</groupId> > >>>> <artifactId>spark-sql_2.10</artifactId> > >>>> <version>1.5.1</version> > >>>> </dependency> > >>>> > >>>> > >>>> [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0 > >>>> [INFO] > >>>> > ------------------------------------------------------------------------ > >>>> [INFO] BUILD SUCCESS > >>>> [INFO] > >>>> > ------------------------------------------------------------------------ > >>>> [INFO] Total time: 01:04 min > >>>> [INFO] Finished at: 2016-03-15T22:55:08+00:00 > >>>> [INFO] Final Memory: 32M/1089M > >>>> [INFO] > >>>> > ------------------------------------------------------------------------ > >>>> > >>>> Dr Mich Talebzadeh > >>>> > >>>> > >>>> > >>>> LinkedIn > >>>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>>> > >>>> > >>>> > >>>> http://talebzadehmich.wordpress.com > >>>> > >>>> > >>>> > >>>> > >>>> On 15 March 2016 at 22:18, Jakob Odersky <ja...@odersky.com> wrote: > >>>>> > >>>>> Hi Mich, > >>>>> probably unrelated to the current error you're seeing, however the > >>>>> following dependencies will bite you later: > >>>>> <artifactId>spark-hive_2.10</artifactId> > >>>>> <artifactId>spark-csv_2.11</artifactId> > >>>>> the problem here is that you're using libraries built for different > >>>>> Scala binary versions (the numbers after the underscore). The simple > >>>>> fix here is to specify the Scala binary version you're project builds > >>>>> for (2.10 in your case, however note that version is EOL, you should > >>>>> upgrade to scala 2.11.8 if possible). > >>>>> > >>>>> On a side note, sbt takes care of handling correct scala versions for > >>>>> you (the double %% actually is a shorthand for appending > >>>>> "_scalaBinaryVersion" to your dependency). It also enables you to > >>>>> build and publish your project seamlessly against multiple versions. > I > >>>>> would strongly recommend to use it in Scala projects. > >>>>> > >>>>> cheers, > >>>>> --Jakob > >>>>> > >>>>> > >>>>> > >>>>> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh > >>>>> <mich.talebza...@gmail.com> wrote: > >>>>> > Hi, > >>>>> > > >>>>> > I normally use sbt and using this sbt file works fine for me > >>>>> > > >>>>> > cat ImportCSV.sbt > >>>>> > name := "ImportCSV" > >>>>> > version := "1.0" > >>>>> > scalaVersion := "2.10.4" > >>>>> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" > >>>>> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" > >>>>> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" > >>>>> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % > "1.3.0" > >>>>> > > >>>>> > This is my first trial using Mavan and pom > >>>>> > > >>>>> > > >>>>> > my pom.xml file looks like this but throws error at build > >>>>> > > >>>>> > > >>>>> > [DEBUG] com.univocity:univocity-parsers:jar:1.5.1:compile > >>>>> > [INFO] > >>>>> > > >>>>> > > ------------------------------------------------------------------------ > >>>>> > [INFO] BUILD FAILURE > >>>>> > [INFO] > >>>>> > > >>>>> > > ------------------------------------------------------------------------ > >>>>> > [INFO] Total time: 1.326 s > >>>>> > [INFO] Finished at: 2016-03-15T22:17:29+00:00 > >>>>> > [INFO] Final Memory: 14M/455M > >>>>> > [INFO] > >>>>> > > >>>>> > > ------------------------------------------------------------------------ > >>>>> > [ERROR] Failed to execute goal on project scala: Could not resolve > >>>>> > dependencies for project spark:scala:jar:1.0: The following > artifacts > >>>>> > could > >>>>> > not be resolved: org.apache.spark:spark-core:jar:1.5.1, > >>>>> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find > >>>>> > org.apache.spark:spark-core:jar:1.5.1 in > >>>>> > https://repo.maven.apache.org/maven2 was cached in the local > >>>>> > repository, > >>>>> > resolution will not be reattempted until the update interval of > >>>>> > central has > >>>>> > elapsed or updates are forced -> [Help 1] > >>>>> > > >>>>> > > >>>>> > My pom file is > >>>>> > > >>>>> > > >>>>> > cat pom.xml > >>>>> > <project xmlns="http://maven.apache.org/POM/4.0.0"; > >>>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > >>>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > >>>>> > http://maven.apache.org/maven-v4_0_0.xsd";> > >>>>> > <modelVersion>4.0.0</modelVersion> > >>>>> > <groupId>spark</groupId> > >>>>> > <version>1.0</version> > >>>>> > <name>${project.artifactId}</name> > >>>>> > > >>>>> > <properties> > >>>>> > <maven.compiler.source>1.7</maven.compiler.source> > >>>>> > <maven.compiler.target>1.7</maven.compiler.target> > >>>>> > <encoding>UTF-8</encoding> > >>>>> > <scala.version>2.10.4</scala.version> > >>>>> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> > >>>>> > </properties> > >>>>> > > >>>>> > <dependencies> > >>>>> > <dependency> > >>>>> > <groupId>org.scala-lang</groupId> > >>>>> > <artifactId>scala-library</artifactId> > >>>>> > <version>2.10.2</version> > >>>>> > </dependency> > >>>>> > <dependency> > >>>>> > <groupId>org.apache.spark</groupId> > >>>>> > <artifactId>spark-core</artifactId> > >>>>> > <version>1.5.1</version> > >>>>> > </dependency> > >>>>> > <dependency> > >>>>> > <groupId>org.apache.spark</groupId> > >>>>> > <artifactId>spark-sql</artifactId> > >>>>> > <version>1.5.1</version> > >>>>> > </dependency> > >>>>> > <dependency> > >>>>> > <groupId>org.apache.spark</groupId> > >>>>> > <artifactId>spark-hive_2.10</artifactId> > >>>>> > <version>1.5.0</version> > >>>>> > </dependency> > >>>>> > <dependency> > >>>>> > <groupId>com.databricks</groupId> > >>>>> > <artifactId>spark-csv_2.11</artifactId> > >>>>> > <version>1.3.0</version> > >>>>> > </dependency> > >>>>> > </dependencies> > >>>>> > > >>>>> > <build> > >>>>> > <sourceDirectory>src/main/scala</sourceDirectory> > >>>>> > <plugins> > >>>>> > <plugin> > >>>>> > <groupId>org.scala-tools</groupId> > >>>>> > <artifactId>maven-scala-plugin</artifactId> > >>>>> > <version>${maven-scala-plugin.version}</version> > >>>>> > <executions> > >>>>> > <execution> > >>>>> > <goals> > >>>>> > <goal>compile</goal> > >>>>> > </goals> > >>>>> > </execution> > >>>>> > </executions> > >>>>> > <configuration> > >>>>> > <jvmArgs> > >>>>> > <jvmArg>-Xms64m</jvmArg> > >>>>> > <jvmArg>-Xmx1024m</jvmArg> > >>>>> > </jvmArgs> > >>>>> > </configuration> > >>>>> > </plugin> > >>>>> > <plugin> > >>>>> > <groupId>org.apache.maven.plugins</groupId> > >>>>> > <artifactId>maven-shade-plugin</artifactId> > >>>>> > <version>1.6</version> > >>>>> > <executions> > >>>>> > <execution> > >>>>> > <phase>package</phase> > >>>>> > <goals> > >>>>> > <goal>shade</goal> > >>>>> > </goals> > >>>>> > <configuration> > >>>>> > <filters> > >>>>> > <filter> > >>>>> > <artifact>*:*</artifact> > >>>>> > <excludes> > >>>>> > <exclude>META-INF/*.SF</exclude> > >>>>> > <exclude>META-INF/*.DSA</exclude> > >>>>> > <exclude>META-INF/*.RSA</exclude> > >>>>> > </excludes> > >>>>> > </filter> > >>>>> > </filters> > >>>>> > <transformers> > >>>>> > <transformer > >>>>> > > >>>>> > > implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> > >>>>> > <mainClass>com.group.id.Launcher1</mainClass> > >>>>> > </transformer> > >>>>> > </transformers> > >>>>> > </configuration> > >>>>> > </execution> > >>>>> > </executions> > >>>>> > </plugin> > >>>>> > </plugins> > >>>>> > </build> > >>>>> > > >>>>> > <artifactId>scala</artifactId> > >>>>> > </project> > >>>>> > > >>>>> > > >>>>> > I am sure I have omitted something? > >>>>> > > >>>>> > > >>>>> > Thanks > >>>>> > > >>>>> > > >>>>> > Dr Mich Talebzadeh > >>>>> > > >>>>> > > >>>>> > > >>>>> > LinkedIn > >>>>> > > >>>>> > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>>>> > > >>>>> > > >>>>> > > >>>>> > http://talebzadehmich.wordpress.com > >>>>> > > >>>>> > > >>>> > >>>> > >>> > >> > > >
