[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370138#comment-14370138
]
Apache Spark commented on SPARK-4123:
-
User 'brennonyork' has created a pull request for this issue:
https://github.com/apache/spark/pull/5093
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Assignee: Brennon York
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351932#comment-14351932
]
Patrick Wendell commented on SPARK-4123:
Hey [~boyork] sorry for the delay. Are you still interested in doing this one?
You are right, the current approach requires a maven install, which won't work
well on jenkins becuase there are multiple pull request builds that share the
same repository. Unfortunately the maven -pl flag requires an install... it's
pretty annoying that it can't reason locally about the fact that it's part of a
multi project build. One thought I had was that it might be possible to just do
a mvn install into a local directory that is part of the specific build folder.
Some local testing revealed that even though maven supposedly supports setting
the localRepositoryPath option during installs, it doesn't seem to work.
Anyways, I came up with another way to do it. It's pretty brittle but it does
seem to work:
{code}
mvn dependency:build-classpath | grep -A 5 Building Spark Project Assembly |
tail -n 1 | tr : \n | rev | cut -d / -f 1 | rev | sort pr_path
{code}
I think using this we can make it work. I just tested it with the SPARK-6122
JIRA and it seemed to work well.
{code}
diff pr_path master_path
118,119c118,119
tachyon-0.6.0.jar
tachyon-client-0.6.0.jar
---
tachyon-0.5.0.jar
tachyon-client-0.5.0.jar
{code}
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335637#comment-14335637
]
Brennon York commented on SPARK-4123:
-
Gents, need a bit of input on this one. Looks like, to get the expected output
that [~pwendell] showed, I would need to run a {{mvn clean ... install}} before
I could do any sub-project traversal ([SO
reference|https://stackoverflow.com/questions/12223754/how-to-maven-sub-projects-with-inter-dependencies]
with similar issue). I've seen this issue in
[SPARK-3355|https://github.com/apache/spark/pull/4734] as well and, in that
case, just avoided it. I've retested on my machine by running the above lines
without installation, then with installation, then clearing org/apache/spark
from my ~/.m2 directory and without again. As expected, it failed, worked, and
then failed again. So... my question is how does Jenkins store state wrt the
local repository? I'm wondering if there might not be another way to grab this
information, but if we use Maven with the above command and we need to install
spark into a local directory I can imagine build failures everywhere, esp. if
we clear / update that directory.
Thoughts?
cc [~shaneknapp] for possible Jenkins advice
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334049#comment-14334049
]
Brennon York commented on SPARK-4123:
-
[~nchammas] have you started this? If not I can take this off your hands, just
let me know!
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334352#comment-14334352
]
Nicholas Chammas commented on SPARK-4123:
-
Go ahead! I haven't done anything for this yet.
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187590#comment-14187590
]
Patrick Wendell commented on SPARK-4123:
[~nchammas] - do you have any interest in doing this one?
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests
[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187598#comment-14187598
]
Nicholas Chammas commented on SPARK-4123:
-
Sure thing, [~pwendell], though I will wait to look into this until I post a
design doc for [SPARK-3821].
Is this an urgent issue?
Show new dependencies added in pull requests
Key: SPARK-4123
URL: https://issues.apache.org/jira/browse/SPARK-4123
Project: Spark
Issue Type: Improvement
Components: Project Infra
Reporter: Patrick Wendell
Priority: Critical
We should inspect the classpath of Spark's assembly jar for every pull
request. This only takes a few seconds in Maven and it will help weed out
dependency changes from the master branch. Ideally we'd post any dependency
changes in the pull request message.
{code}
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort my-classpath
$ git checkout apache/master
$ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
INFO | tr : \n | awk -F/ '{print $NF}' | sort master-classpath
$ diff my-classpath master-classpath
chill-java-0.3.6.jar
chill_2.10-0.3.6.jar
---
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
