Hi,
testing out the Spark Ec2 deployment plugin:
I try to compile using
$sbt sparkLaunchCluster
---
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: fr.janalyse#janalyse-ssh_2.10;0.9.13: not found
[warn] ::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] fr.janalyse:janalyse-ssh_2.10:0.9.13
[warn] +- com.gilt:lib-spark-manager_2.10:0.0.3.9
[warn] +- com.gilt:sbt-spark-ec2-plugin:0.1.5 (sbtVersion=0.13,
scalaVersion=2.10)
(/Users/saq/Work/spark-datapipeline/project/plugins.sbt#L6-7)
[warn] +- default:spark-datapipeline-build:0.1-SNAPSHOT
(sbtVersion=0.13, scalaVersion=2.10)
sbt.ResolveException: unresolved dependency:
fr.janalyse#janalyse-ssh_2.10;0.9.13: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
at sbt.IvySbt.withIvy(Ivy.scala:119)
at sbt.IvySbt.withIvy(Ivy.scala:116)
at sbt.IvySbt$Module.withModule(Ivy.scala:147)
--
I have to use scala version 2.10 as spark is using it and sbt version is
0.13
My sbt looks as follows:
name := scala-datapipeline
version := 1.0
scalaVersion := 2.10.4
scalacOptions ++= Seq( -deprecation, -unchecked, -feature)
sparkec2.Ec2SparkPluginSettings.sparkSettings
resolvers += Akka Repository at http://repo.akka.io/releases/;
resolvers += spray at http://repo.spray.io/;
resolvers += Resolver.url(
bintray Repository,
url(http://dl.bintray.com/felixgborrego/repo;))(
Resolver.ivyStylePatterns)
resolvers += JAnalyse Repository at http://www.janalyse.fr/repository/;
libraryDependencies ++= Seq(
org.apache.spark %% spark-core % 1.1.0,// 1.0.2
org.apache.spark %% spark-sql % 1.1.0,
org.apache.spark %% spark-hive % 1.1.0,
com.github.nscala-time %% nscala-time % 1.0.0,
org.json4s %% json4s-native % 3.2.10,
com.codahale %% jerkson_2.9.1 % 0.5.0,
fr.janalyse % janalyse-ssh % 0.9.10
)
On Fri, Sep 5, 2014 at 4:08 AM, andy petrella andy.petre...@gmail.com
wrote:
\o/ = will test it soon or sooner, gr8 idea btw
aℕdy ℙetrella
about.me/noootsab
[image: aℕdy ℙetrella on about.me]
http://about.me/noootsab
On Fri, Sep 5, 2014 at 12:37 PM, Felix Garcia Borrego fborr...@gilt.com
wrote:
As far as I know in other to deploy and execute jobs in EC2 you need to
assembly you project, copy your jar into the cluster, log into using ssh
and submit the job.
To avoid having to do this I've been prototyping an sbt plugin(1) that
allows to create and send Spark jobs to an Amazon EC2 cluster directly from
your local machine using sbt.
It's a simple plugin that actually rely on spark-ec2 and spark-submit,
but I'd like to have feedback and see if this plugin makes any sense
before going ahead with the final impl or if there is any other easy way to
do so.
(1) https://github.com/felixgborrego/sbt-spark-ec2-plugin
Thanks,
--
Kind Regards,
Shafaq
