that should read anything.sbt Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 16 March 2016 at 00:04, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > in mvn the build mvn package will look for a file called pom.xml > > in sbt the build sbt package will look for a file called anything.smt > > It works > > Keep it simple > > I will write a ksh script that will create both generic and sbt files on > the fly in the correct directory (at the top of the tree) and remove them > after the job finished. > > That will keep audit people happy as well 😁 > > HTh > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 15 March 2016 at 23:34, Jakob Odersky <ja...@odersky.com> wrote: > >> The artifactId in maven basically (in a simple case) corresponds to name >> in sbt. >> >> Note however that you will manually need to append the >> _scalaBinaryVersion to the artifactId in case you would like to build >> against multiple scala versions (otherwise maven will overwrite the >> generated jar with the latest one). >> >> >> On Tue, Mar 15, 2016 at 4:27 PM, Mich Talebzadeh >> <mich.talebza...@gmail.com> wrote: >> > ok Ted >> > >> > In sbt I have >> > >> > name := "ImportCSV" >> > version := "1.0" >> > scalaVersion := "2.10.4" >> > >> > which ends up in importcsv_2.10-1.0.jar as part of >> > target/scala-2.10/importcsv_2.10-1.0.jar >> > >> > In mvn I have >> > >> > <version>1.0</version> >> > <artifactId>scala</artifactId> >> > >> > >> > Does it matter? >> > >> > >> > Dr Mich Talebzadeh >> > >> > >> > >> > LinkedIn >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > >> > >> > >> > http://talebzadehmich.wordpress.com >> > >> > >> > >> > >> > On 15 March 2016 at 23:17, Ted Yu <yuzhih...@gmail.com> wrote: >> >> >> >> <version>1.0</version> >> >> ... >> >> <artifactId>scala</artifactId> >> >> >> >> On Tue, Mar 15, 2016 at 4:14 PM, Mich Talebzadeh >> >> <mich.talebza...@gmail.com> wrote: >> >>> >> >>> An observation >> >>> >> >>> Once compiled with MVN the job submit works as follows: >> >>> >> >>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages >> >>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master >> >>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12 >> >>> --num-executors=2 target/scala-1.0.jar >> >>> >> >>> With sbt it takes this form >> >>> >> >>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages >> >>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master >> >>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12 >> >>> --num-executors=2 target/scala-2.10/importcsv_2.10-1.0.jar >> >>> >> >>> They both return the same results. However, why mvnjar file name is >> >>> different (may be a naive question!)? >> >>> >> >>> thanks >> >>> >> >>> >> >>> Dr Mich Talebzadeh >> >>> >> >>> >> >>> >> >>> LinkedIn >> >>> >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >>> >> >>> >> >>> >> >>> http://talebzadehmich.wordpress.com >> >>> >> >>> >> >>> >> >>> >> >>> On 15 March 2016 at 22:43, Mich Talebzadeh <mich.talebza...@gmail.com >> > >> >>> wrote: >> >>>> >> >>>> Many thanks Ted and thanks for heads up Jakob >> >>>> >> >>>> Just these two changes to dependencies >> >>>> >> >>>> <dependency> >> >>>> <groupId>org.apache.spark</groupId> >> >>>> <artifactId>spark-core_2.10</artifactId> >> >>>> <version>1.5.1</version> >> >>>> </dependency> >> >>>> <dependency> >> >>>> <groupId>org.apache.spark</groupId> >> >>>> <artifactId>spark-sql_2.10</artifactId> >> >>>> <version>1.5.1</version> >> >>>> </dependency> >> >>>> >> >>>> >> >>>> [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0 >> >>>> [INFO] >> >>>> >> ------------------------------------------------------------------------ >> >>>> [INFO] BUILD SUCCESS >> >>>> [INFO] >> >>>> >> ------------------------------------------------------------------------ >> >>>> [INFO] Total time: 01:04 min >> >>>> [INFO] Finished at: 2016-03-15T22:55:08+00:00 >> >>>> [INFO] Final Memory: 32M/1089M >> >>>> [INFO] >> >>>> >> ------------------------------------------------------------------------ >> >>>> >> >>>> Dr Mich Talebzadeh >> >>>> >> >>>> >> >>>> >> >>>> LinkedIn >> >>>> >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >>>> >> >>>> >> >>>> >> >>>> http://talebzadehmich.wordpress.com >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On 15 March 2016 at 22:18, Jakob Odersky <ja...@odersky.com> wrote: >> >>>>> >> >>>>> Hi Mich, >> >>>>> probably unrelated to the current error you're seeing, however the >> >>>>> following dependencies will bite you later: >> >>>>> <artifactId>spark-hive_2.10</artifactId> >> >>>>> <artifactId>spark-csv_2.11</artifactId> >> >>>>> the problem here is that you're using libraries built for different >> >>>>> Scala binary versions (the numbers after the underscore). The simple >> >>>>> fix here is to specify the Scala binary version you're project >> builds >> >>>>> for (2.10 in your case, however note that version is EOL, you should >> >>>>> upgrade to scala 2.11.8 if possible). >> >>>>> >> >>>>> On a side note, sbt takes care of handling correct scala versions >> for >> >>>>> you (the double %% actually is a shorthand for appending >> >>>>> "_scalaBinaryVersion" to your dependency). It also enables you to >> >>>>> build and publish your project seamlessly against multiple >> versions. I >> >>>>> would strongly recommend to use it in Scala projects. >> >>>>> >> >>>>> cheers, >> >>>>> --Jakob >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh >> >>>>> <mich.talebza...@gmail.com> wrote: >> >>>>> > Hi, >> >>>>> > >> >>>>> > I normally use sbt and using this sbt file works fine for me >> >>>>> > >> >>>>> > cat ImportCSV.sbt >> >>>>> > name := "ImportCSV" >> >>>>> > version := "1.0" >> >>>>> > scalaVersion := "2.10.4" >> >>>>> > libraryDependencies += "org.apache.spark" %% "spark-core" % >> "1.5.1" >> >>>>> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" >> >>>>> > libraryDependencies += "org.apache.spark" %% "spark-hive" % >> "1.5.1" >> >>>>> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % >> "1.3.0" >> >>>>> > >> >>>>> > This is my first trial using Mavan and pom >> >>>>> > >> >>>>> > >> >>>>> > my pom.xml file looks like this but throws error at build >> >>>>> > >> >>>>> > >> >>>>> > [DEBUG] com.univocity:univocity-parsers:jar:1.5.1:compile >> >>>>> > [INFO] >> >>>>> > >> >>>>> > >> ------------------------------------------------------------------------ >> >>>>> > [INFO] BUILD FAILURE >> >>>>> > [INFO] >> >>>>> > >> >>>>> > >> ------------------------------------------------------------------------ >> >>>>> > [INFO] Total time: 1.326 s >> >>>>> > [INFO] Finished at: 2016-03-15T22:17:29+00:00 >> >>>>> > [INFO] Final Memory: 14M/455M >> >>>>> > [INFO] >> >>>>> > >> >>>>> > >> ------------------------------------------------------------------------ >> >>>>> > [ERROR] Failed to execute goal on project scala: Could not resolve >> >>>>> > dependencies for project spark:scala:jar:1.0: The following >> artifacts >> >>>>> > could >> >>>>> > not be resolved: org.apache.spark:spark-core:jar:1.5.1, >> >>>>> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find >> >>>>> > org.apache.spark:spark-core:jar:1.5.1 in >> >>>>> > https://repo.maven.apache.org/maven2 was cached in the local >> >>>>> > repository, >> >>>>> > resolution will not be reattempted until the update interval of >> >>>>> > central has >> >>>>> > elapsed or updates are forced -> [Help 1] >> >>>>> > >> >>>>> > >> >>>>> > My pom file is >> >>>>> > >> >>>>> > >> >>>>> > cat pom.xml >> >>>>> > <project xmlns="http://maven.apache.org/POM/4.0.0"; >> >>>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> >>>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >> >>>>> > http://maven.apache.org/maven-v4_0_0.xsd";> >> >>>>> > <modelVersion>4.0.0</modelVersion> >> >>>>> > <groupId>spark</groupId> >> >>>>> > <version>1.0</version> >> >>>>> > <name>${project.artifactId}</name> >> >>>>> > >> >>>>> > <properties> >> >>>>> > <maven.compiler.source>1.7</maven.compiler.source> >> >>>>> > <maven.compiler.target>1.7</maven.compiler.target> >> >>>>> > <encoding>UTF-8</encoding> >> >>>>> > <scala.version>2.10.4</scala.version> >> >>>>> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >> >>>>> > </properties> >> >>>>> > >> >>>>> > <dependencies> >> >>>>> > <dependency> >> >>>>> > <groupId>org.scala-lang</groupId> >> >>>>> > <artifactId>scala-library</artifactId> >> >>>>> > <version>2.10.2</version> >> >>>>> > </dependency> >> >>>>> > <dependency> >> >>>>> > <groupId>org.apache.spark</groupId> >> >>>>> > <artifactId>spark-core</artifactId> >> >>>>> > <version>1.5.1</version> >> >>>>> > </dependency> >> >>>>> > <dependency> >> >>>>> > <groupId>org.apache.spark</groupId> >> >>>>> > <artifactId>spark-sql</artifactId> >> >>>>> > <version>1.5.1</version> >> >>>>> > </dependency> >> >>>>> > <dependency> >> >>>>> > <groupId>org.apache.spark</groupId> >> >>>>> > <artifactId>spark-hive_2.10</artifactId> >> >>>>> > <version>1.5.0</version> >> >>>>> > </dependency> >> >>>>> > <dependency> >> >>>>> > <groupId>com.databricks</groupId> >> >>>>> > <artifactId>spark-csv_2.11</artifactId> >> >>>>> > <version>1.3.0</version> >> >>>>> > </dependency> >> >>>>> > </dependencies> >> >>>>> > >> >>>>> > <build> >> >>>>> > <sourceDirectory>src/main/scala</sourceDirectory> >> >>>>> > <plugins> >> >>>>> > <plugin> >> >>>>> > <groupId>org.scala-tools</groupId> >> >>>>> > <artifactId>maven-scala-plugin</artifactId> >> >>>>> > <version>${maven-scala-plugin.version}</version> >> >>>>> > <executions> >> >>>>> > <execution> >> >>>>> > <goals> >> >>>>> > <goal>compile</goal> >> >>>>> > </goals> >> >>>>> > </execution> >> >>>>> > </executions> >> >>>>> > <configuration> >> >>>>> > <jvmArgs> >> >>>>> > <jvmArg>-Xms64m</jvmArg> >> >>>>> > <jvmArg>-Xmx1024m</jvmArg> >> >>>>> > </jvmArgs> >> >>>>> > </configuration> >> >>>>> > </plugin> >> >>>>> > <plugin> >> >>>>> > <groupId>org.apache.maven.plugins</groupId> >> >>>>> > <artifactId>maven-shade-plugin</artifactId> >> >>>>> > <version>1.6</version> >> >>>>> > <executions> >> >>>>> > <execution> >> >>>>> > <phase>package</phase> >> >>>>> > <goals> >> >>>>> > <goal>shade</goal> >> >>>>> > </goals> >> >>>>> > <configuration> >> >>>>> > <filters> >> >>>>> > <filter> >> >>>>> > <artifact>*:*</artifact> >> >>>>> > <excludes> >> >>>>> > <exclude>META-INF/*.SF</exclude> >> >>>>> > <exclude>META-INF/*.DSA</exclude> >> >>>>> > <exclude>META-INF/*.RSA</exclude> >> >>>>> > </excludes> >> >>>>> > </filter> >> >>>>> > </filters> >> >>>>> > <transformers> >> >>>>> > <transformer >> >>>>> > >> >>>>> > >> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> >> >>>>> > <mainClass>com.group.id.Launcher1</mainClass> >> >>>>> > </transformer> >> >>>>> > </transformers> >> >>>>> > </configuration> >> >>>>> > </execution> >> >>>>> > </executions> >> >>>>> > </plugin> >> >>>>> > </plugins> >> >>>>> > </build> >> >>>>> > >> >>>>> > <artifactId>scala</artifactId> >> >>>>> > </project> >> >>>>> > >> >>>>> > >> >>>>> > I am sure I have omitted something? >> >>>>> > >> >>>>> > >> >>>>> > Thanks >> >>>>> > >> >>>>> > >> >>>>> > Dr Mich Talebzadeh >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > LinkedIn >> >>>>> > >> >>>>> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > http://talebzadehmich.wordpress.com >> >>>>> > >> >>>>> > >> >>>> >> >>>> >> >>> >> >> >> > >> > >
