GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/10461
[SPARK-10359] Enumerate dependencies in a file and diff against it for new
pull requests
This patch adds a new build check which enumerates Spark's resolved runtime
classpath and saves it to a file, then diffs against that file to detect
whether pull requests have introduced dependency changes. The aim of this check
is to make it simpler to reason about whether pull request which modify the
build have introduced new dependencies or changed transitive dependencies in a
way that affects the final classpath.
This supplants the checks added in SPARK-4123 / #5093, which are currently
disabled due to bugs.
This patch is based on @pwendell's work in #8531.
Closes #8531.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-10359
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10461.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10461
----
commit 9cf442df2c04291ae1a7df567658b490eaa46708
Author: Patrick Wendell <[email protected]>
Date: 2015-08-28T22:43:25Z
Adding build test module
commit 20ae791d118af430320b8b9141f9cb8150313a9c
Author: Patrick Wendell <[email protected]>
Date: 2015-08-31T05:31:35Z
Adding new deps file without spark
commit 5670b6a8231f19eb1bba11fdae5c965f653ee18c
Author: Patrick Wendell <[email protected]>
Date: 2015-08-31T05:32:31Z
Moving build tests earlier
commit 5b2910b769bedca6a0151811dc5b87a3e6e69bdd
Author: Patrick Wendell <[email protected]>
Date: 2015-08-31T05:33:22Z
Adding license header
commit 0c8153ddb961195020ed050c39f0274d6ce174f6
Author: Patrick Wendell <[email protected]>
Date: 2015-08-31T05:45:06Z
Adding RAT exclude
commit f4243b97f67f6db4e7bb06b7bfe44f64c0e43e5c
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T03:51:09Z
Merge remote-tracking branch 'apache/master' into dependency-audits
Conflicts:
dev/run-tests-jenkins
dev/sparktestsupport/modules.py
commit 17970f781457751f511fb56dc164ee9ab4c645eb
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T03:54:11Z
Ignore spark itself in deps file
commit 16adab6567ffcdfbb7213013214b62a6a36067ba
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T04:04:19Z
Style fix
commit 8b973bf8261a151f12ce7ed7f2812e897608fd73
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T04:05:08Z
Adding test block
commit d181e3d15c1ca436084a05f76eb18cde4ac1a209
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T04:37:17Z
Only include compile dependencies when generating list
commit 84cab2c37bd7b0ad4904d49d7561205c994e194f
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T05:14:51Z
Adding Hadoop 1 profile
commit 5c8ccf68bc79bd25e388dfd0f9dbe93a0b732222
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T05:52:19Z
Adding new hadoop versions
commit fcea19f93a34b8c837de4af3744d5919ec333186
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T05:59:20Z
Bug fix
commit 2f4d3e5eb00b0c86cd5ec3bc7553ede129fd8aef
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T05:59:50Z
Adding new hadoop deps
commit 37230f0f7861c08b3302223f65388d2c371223e5
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T07:02:59Z
More RAT excludes
commit 1d91634a0a5803260090c528f4cfd49b71d82986
Author: Patrick Wendell <[email protected]>
Date: 2015-11-03T19:53:43Z
Testing new dependencies
commit 1e3ff47627b7c510bc9377193bf90986e359f0a1
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T20:11:39Z
Merge remote-tracking branch 'origin/master' into SPARK-10359
commit b0135146504ee779739caee260c3f6db37632647
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T22:09:35Z
Refactor to avoid compile in install.
commit 8f7007eec60ac9ee987fb5c28ff405756e8739a1
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T22:14:52Z
Don't generate version backup POMs.
commit 258e4f7f24fccec9c1d73c2e65fb514085beec52
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T22:17:57Z
Use bash exit traps to ensure proper cleanup
commit e8af27462fe47f404038232b05b78c4b572b2b55
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T22:19:03Z
Whoops, missed another -DgenerateBackupPoms=false
commit 8507ffd589211e1a7b4a5bc44f2d7ff6fa80dacc
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T22:36:37Z
Support multiple Hadoop profiles.
commit 27112525fc00c322aaf8dea68aa940a53f762858
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T22:46:34Z
Roll back test dependency change.
commit 4ad3ae49969ddfbe9d66009c82bd63caf0ee13dc
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T23:17:37Z
Fix RAT excludes (CLI RAT doesn't seem to support full path pattern
matching)
commit 7c86eb7a245b787dd6fe7f875f655cd0f036e160
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T23:32:09Z
dev/test-dependencies.sh is part of the build module.
commit 87918d8df6ce2608f18f70704ba6b8f900e8ca5d
Author: Josh Rosen <[email protected]>
Date: 2015-12-23T23:41:11Z
Use runtime scope (per Sean's comments)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
