[
https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351932#comment-14351932
]
Patrick Wendell commented on SPARK-4123:
----------------------------------------
Hey [~boyork] sorry for the delay. Are you still interested in doing this one?
You are right, the current approach requires a maven install, which won't work
well on jenkins becuase there are multiple pull request builds that share the
same repository. Unfortunately the maven "-pl" flag requires an install... it's
pretty annoying that it can't reason locally about the fact that it's part of a
multi project build. One thought I had was that it might be possible to just do
a mvn install into a local directory that is part of the specific build folder.
Some local testing revealed that even though maven supposedly supports setting
the localRepositoryPath option during installs, it doesn't seem to work.
Anyways, I came up with another way to do it. It's pretty brittle but it does
seem to work:
{code}
mvn dependency:build-classpath | grep -A 5 "Building Spark Project Assembly" |
tail -n 1 | tr ":" "\n" | rev | cut -d "/" -f 1 | rev | sort > pr_path
{code}
I think using this we can make it work. I just tested it with the SPARK-6122
JIRA and it seemed to work well.
{code}
> diff pr_path master_path
118,119c118,119
< tachyon-0.6.0.jar
< tachyon-client-0.6.0.jar
---
> tachyon-0.5.0.jar
> tachyon-client-0.5.0.jar
{code}
> Show new dependencies added in pull requests
> --------------------------------------------
>
> Key: SPARK-4123
> URL: https://issues.apache.org/jira/browse/SPARK-4123
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Reporter: Patrick Wendell
> Priority: Critical
>
> We should inspect the classpath of Spark's assembly jar for every pull
> request. This only takes a few seconds in Maven and it will help weed out
> dependency changes from the master branch. Ideally we'd post any dependency
> changes in the pull request message.
> {code}
> $ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
> INFO | tr : "\n" | awk -F/ '{print $NF}' | sort > my-classpath
> $ git checkout apache/master
> $ mvn -Phive -Phadoop-2.4 dependency:build-classpath -pl assembly | grep -v
> INFO | tr : "\n" | awk -F/ '{print $NF}' | sort > master-classpath
> $ diff my-classpath master-classpath
> < chill-java-0.3.6.jar
> < chill_2.10-0.3.6.jar
> ---
> > chill-java-0.5.0.jar
> > chill_2.10-0.5.0.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
