Thanks for the info!

Xiaoyong

From: Sean Owen [mailto:so...@cloudera.com]
Sent: Monday, December 14, 2015 12:20 AM
To: Xiaoyong Zhu <xiaoy...@microsoft.com>
Cc: user <user@spark.apache.org>
Subject: Re: Re: Spark assembly in Maven repo?


Yes, though I think the Maven Central repository is more canonical.

http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.5.2/<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2frepo1.maven.org%2fmaven2%2forg%2fapache%2fspark%2fspark-core_2.10%2f1.5.2%2f&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7cbc95bb73fe654d2c16df08d3045f61e3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=UmuArU%2bj9WRhKE0VpQkm63TC4ewQaq150Ne5YbwmiL4%3d>

On Mon, Dec 14, 2015, 06:35 Xiaoyong Zhu 
<xiaoy...@microsoft.com<mailto:xiaoy...@microsoft.com>> wrote:
Thanks! do you mean something here (for example for 1.5.1 using scala 2.10)?
https://repository.apache.org/content/repositories/releases/org/apache/spark/spark-core_2.10/1.5.1/<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2frepository.apache.org%2fcontent%2frepositories%2freleases%2forg%2fapache%2fspark%2fspark-core_2.10%2f1.5.1%2f&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7cbc95bb73fe654d2c16df08d3045f61e3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=XznNXDLoqiO7aE48Vg7BhTcnVico4ibnalzlmM67nec%3d>

Xiaoyong

From: Sean Owen [mailto:so...@cloudera.com<mailto:so...@cloudera.com>]
Sent: Saturday, December 12, 2015 12:45 AM
To: Xiaoyong Zhu <xiaoy...@microsoft.com<mailto:xiaoy...@microsoft.com>>
Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>>

Subject: Re: Re: Spark assembly in Maven repo?

That's exactly what the various artifacts in the Maven repo are for. The API 
classes for core are in the core artifact and so on. You don't need an assembly.

On Sat, Dec 12, 2015 at 12:32 AM, Xiaoyong Zhu 
<xiaoy...@microsoft.com<mailto:xiaoy...@microsoft.com>> wrote:
Yes, so our scenario is to treat the spark assembly as an “SDK” so users can 
develop Spark applications easily without downloading them. In this case which 
way do you guys think might be good?

Xiaoyong

From: fightf...@163.com<mailto:fightf...@163.com> 
[mailto:fightf...@163.com<mailto:fightf...@163.com>]
Sent: Friday, December 11, 2015 12:08 AM
To: Mark Hamstra <m...@clearstorydata.com<mailto:m...@clearstorydata.com>>
Cc: Xiaoyong Zhu <xiaoy...@microsoft.com<mailto:xiaoy...@microsoft.com>>; Jeff 
Zhang <zjf...@gmail.com<mailto:zjf...@gmail.com>>; user 
<user@spark.apache.org<mailto:user@spark.apache.org>>; Zhaomin Xu 
<z...@microsoft.com<mailto:z...@microsoft.com>>; Joe Zhang (SDE) 
<gui...@microsoft.com<mailto:gui...@microsoft.com>>
Subject: Re: Re: Spark assembly in Maven repo?

Agree with you that assembly jar is not good to publish. However, what he 
really need is to fetch
an updatable maven jar file.

________________________________
fightf...@163.com<mailto:fightf...@163.com>

From: Mark Hamstra<mailto:m...@clearstorydata.com>
Date: 2015-12-11 15:34
To: fightf...@163.com<mailto:fightf...@163.com>
CC: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>; Jeff 
Zhang<mailto:zjf...@gmail.com>; user<mailto:user@spark.apache.org>; Zhaomin 
Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com>
Subject: Re: RE: Spark assembly in Maven repo?
No, publishing a spark assembly jar is not fine.  See the doc attached to 
https://issues.apache.org/jira/browse/SPARK-11157<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fSPARK-11157&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7c999bc04245724187198808d3020238b3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=cUBvArmmYasxoOl9wrefYO4L7JiTF5qqqZDyY%2b%2bLP3Y%3d>
 and be aware that a likely goal of Spark 2.0 will be the elimination of 
assemblies.

On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com<mailto:fightf...@163.com> 
<fightf...@163.com<mailto:fightf...@163.com>> wrote:
Using maven to download the assembly jar is fine. I would recommend to deploy 
this
assembly jar to your local maven repo, i.e. nexus repo, Or more likey a 
snapshot repository

________________________________
fightf...@163.com<mailto:fightf...@163.com>

From: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>
Date: 2015-12-11 15:10
To: Jeff Zhang<mailto:zjf...@gmail.com>
CC: user@spark.apache.org<mailto:user@spark.apache.org>; Zhaomin 
Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com>
Subject: RE: Spark assembly in Maven repo?
Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s 
actually that we are building a certain plugin for IntelliJ where we want to 
distribute this jar. But since the jar is updated frequently we don't want to 
distribute it together with our plugin but we would like to download it via 
Maven.

In this case what’s the recommended way?

Xiaoyong

From: Jeff Zhang [mailto:zjf...@gmail.com<mailto:zjf...@gmail.com>]
Sent: Thursday, December 10, 2015 11:03 PM
To: Xiaoyong Zhu <xiaoy...@microsoft.com<mailto:xiaoy...@microsoft.com>>
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Spark assembly in Maven repo?

I don't think make the assembly jar as dependency a good practice. You may meet 
jar hell issue in that case.

On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu 
<xiaoy...@microsoft.com<mailto:xiaoy...@microsoft.com>> wrote:
Hi Experts,

We have a project which has a dependency for the following jar

spark-assembly-<version>-hadoop<version>.jar
for example:
spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar

since this assembly might be updated in the future, I am not sure if there is a 
Maven repo that has the above spark assembly jar? Or should we create & upload 
it to Maven central?

Thanks!

Xiaoyong




--
Best Regards

Jeff Zhang


Reply via email to