Github user witgo commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60214941
@ScrapCodes @pwendell
This patch will cause `maven-assembly-plugin` error:
` ./make-distribution.sh -Dhadoop.version=2.3.0-cdh5.0.1
-Dyarn.version=2.3.0-cdh5.0.1
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60325425
Thanks - I reverted this patch in master.
On Thu, Oct 23, 2014 at 2:39 AM, Guoqiang Li notificati...@github.com
wrote:
@ScrapCodes
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60041268
Not sure which is better ! Creating an empty project with just a pom file
in it. Or depending on random jar from maven central. ? I prefer first approach.
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60043845
I'll just publish an empty jar in maven central - we can use it forever.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60044464
I created a jar you can use. It might take a couple of hours to propagate
to maven central:
```
groupIdorg.spark-project.spark/groupId
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60128322
@ScrapCodes here's an approach that doesn't rely on maven central:
https://gist.github.com/vanzin/1ac8114faf39a6e8359d
Feel free to use / adapt it if you guys
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60163154
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60164022
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22045/consoleFull)
for PR 2673 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60171222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60171219
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22045/consoleFull)
for PR 2673 at commit
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60185661
Okay - both approach seem decent but the current one is a bit simpler for
developers (otherwise we have to explain this `emtpy` source directory). I'm
just gonna pull
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2673
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59889967
You were right @pwendell. I was just imagining things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19133971
--- Diff: pom.xml ---
@@ -248,7 +248,19 @@
/snapshots
/pluginRepository
/pluginRepositories
-
+ !--
+ This
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19134021
--- Diff: pom.xml ---
@@ -248,7 +248,19 @@
/snapshots
/pluginRepository
/pluginRepositories
-
+ !--
+ This
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19134051
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19134094
--- Diff: pom.xml ---
@@ -248,7 +248,19 @@
/snapshots
/pluginRepository
/pluginRepositories
-
+ !--
+ This
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19134071
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19134114
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59892653
Hey @ScrapCodes I had some small comments. If we can use the shade plug-in
to create these effective poms that would be great. If you add your scala 2.11
stuff on top
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59907791
Yeah I verified the resultant dependency tree as shown in gists above.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19140246
--- Diff: pom.xml ---
@@ -248,7 +248,19 @@
/snapshots
/pluginRepository
/pluginRepositories
-
+ !--
+ This
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19162353
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59961868
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59962873
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21996/consoleFull)
for PR 2673 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59981936
**[Tests timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21996/consoleFull)**
for PR 2673 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59981948
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60003900
Hey @ScrapCodes @pwendell,
Funny thing. I've been playing with a different part of the build and I've
found out I need exactly this to get what I wanted to work.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60009061
Yay, I just noticed your last patch takes exactly that approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19184047
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60012589
@vanzin yep that's exactly what we did!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60012622
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60013186
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22014/consoleFull)
for PR 2673 at commit
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19185880
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19187017
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19187067
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60023050
**[Tests timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22014/consoleFull)**
for PR 2673 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-60023054
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2673#discussion_r19194279
--- Diff: pom.xml ---
@@ -994,6 +1006,34 @@
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
+
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59736352
This is the gist of dependency tree for artifacts published by this patch.
https://gist.github.com/ScrapCodes/a5857e57d828b4b787ff
---
If your project is set up for
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59469813
We can definitely install other modules, I am afraid if the
resultant(effective) pom(s) will carry reference to parent pom. Let me try that
out.
---
If your
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59319525
What if we just add the maven-shade plug-in but we configure it in a way
where it effectively is doing no operations? I.e. we add a relocation of a
class that does not
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59319582
I think that a solution that doesn't require us using our own plug-in is
good, even if we do something silly like we have to shade a class that is in
fact un-used.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59331286
Even if we do this trick and shade something in all artifacts, what about
spark-parent ? There since we don't build jar shading plugin throws NPEs.
---
If your
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-59468461
Hm - is there any workaround to that? We don't actually need to publish the
parent pom anyways if we are using effective poms. Could we just install the
other modules
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58851300
@pwendell I don't see an easy way with maven shade plugin either ? Do you
?, One way is to include a fake dependency and then ask it to shade that across
all
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58619871
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21579/consoleFull)
for PR 2673 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58625132
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21579/consoleFull)
for PR 2673 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58625140
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58631408
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21589/consoleFull)
for PR 2673 at commit
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58631368
@pwendell I tried maven shade plugin to somehow work as effective pom
generator, but that does not happen unless we have dependencies apart from the
project's itself
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58637193
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58637190
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21589/consoleFull)
for PR 2673 at commit
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58319574
@vanzin yeah I agree - I think it was just sort of a red herring based on
the example in @ScrapCodes description. AFAIK this is totally unrelated to
Hadoop.
---
If
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58216712
Thanks for the explanation. The issue with the Scala versions makes sense.
What threw me off was the Hadoop example: I've always seen people say that the
Spark API is
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58225042
Hi @pwendell, I had a similar issue related to artifacts in Maven Central
and Hadoop versions.
Could you take a look at
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58004659
@pwendell Take a look, whenever you get time. It would be good if we can
publish https://github.com/ScrapCodes/effective-pom-plugin.
---
If your project is set up
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58005087
I will have to add a similar thing for
http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html. But
I am not sure about repository url field.
---
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58009844
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21323/consoleFull)
for PR 2673 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58009851
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58070415
@ScrapCodes what's the end goal here?
I understand what your change does, but I'm confused about how this
actually translates to the published bits, unless
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58138496
Well, sorry for the confusion. For the published bits used now one can
easily exclude and include another dependency as long as it is about a few
deps. But once we
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2673#issuecomment-58139100
Hey @vanzin this is something we need to support cross building in 2.10 and
2.11, but it's also a broadly useful thing that is lacking in the maven build
ecosystem (SBT
63 matches
Mail list logo
