GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/9575
[SPARK-7841] Stop using retrieveManaged to retrieve dependencies in SBT
This patch modifies Spark's SBT build so that it no longer uses
`retrieveManaged` / `lib_managed` to store its dependencies. The motivations
for this change are nicely described on the JIRA ticket
([SPARK-7841](https://issues.apache.org/jira/browse/SPARK-7841)); my personal
interest in doing this stems from the fact that `lib_managed` has caused me
some pain while debugging dependency issues in another PR of mine.
Removing our use of `lib_managed` would be trivial except for one snag: the
Datanucleus JARs, required by Spark SQL's Hive integration, cannot be included
in assembly JARs due to problems with merging OSGI `plugin.xml` files. As a
result, several places in the packaging and deployment pipeline assume that
these Datanucleus JARs are copied to `lib_managed/jars`. In the interest of
maintaining compatibility, I have chosen to retain the `lib_managed/jars`
directory _only_ for these Datanucleus JARs and have added custom code to
`SparkBuild.scala` to automatically copy those JARs to that folder as part of
the `assembly` task.
`dev/mima` also depended on `lib_managed` in a hacky way in order to set
classpaths when generating MiMa excludes; I've updated this to obtain the
classpaths directly from SBT instead.
/cc @dragos @marmbrus @pwendell @srowen
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-7841
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9575.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9575
----
commit 0993cb990b715135eab0ce07d1d0da220beec4ec
Author: Josh Rosen <[email protected]>
Date: 2015-11-09T21:59:37Z
Remove retrieveManaged and add logic to continue copying datanucleus jars.
commit 828c3b4207c2a1d01bf443f1e41a041ed6a982f1
Author: Josh Rosen <[email protected]>
Date: 2015-11-09T22:09:40Z
Fix MiMA script to cope with empty lib_managed
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
