Thanks for the info Burak, I will check the repo you mention, do you know concretely what is the 'magic' that spark-packages need or if is there any document with info about it ?
On Fri, Jul 15, 2016 at 10:12 PM, Luciano Resende <luckbr1...@gmail.com> wrote: > > On Fri, Jul 15, 2016 at 10:48 AM, Jacek Laskowski <ja...@japila.pl> wrote: > >> +1000 >> >> Thanks Ismael for bringing this up! I meant to have send it earlier too >> since I've been struggling with a sbt-based Scala project for a Spark >> package myself this week and haven't yet found out how to do local >> publishing. >> >> If such a guide existed for Maven I could use it for sbt easily too :-) >> >> Ping me Ismael if you don't hear back from the group so I feel invited >> for digging into the plugin's sources. >> >> Best, >> Jacek >> >> On 15 Jul 2016 2:29 p.m., "Ismaël Mejía" <ieme...@gmail.com> wrote: >> >> Hello, I would like to know if there is an easy way to package a new >> spark-package >> with maven, I just found this repo, but I am not an sbt user. >> >> https://github.com/databricks/sbt-spark-package >> >> One more question, is there a formal specification or documentation of >> what do >> you need to include in a spark-package (any special file, manifest, etc) >> ? I >> have not found any doc in the website. >> >> Thanks, >> Ismael >> >> >> > > I was under the impression that spark-packages was more like a place for > one to list/advertise their extensions, but when you do spark submit with > --packages, it will use maven to resolve your package > and as long as it succeeds, it will use it (e.g. you can do mvn clean > install for your local packages, and use --packages with a spark server > running on that same machine). > > From sbt, I think you can just use publishTo and define a local > repository, something like > > publishTo := Some("Local Maven Repository" at > "file://"+Path.userHome.absolutePath+"/.m2/repository") > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ >
