GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/629
SPARK-1556. jets3t dep doesn't update properly with newer Hadoop versions
See related discussion at https://github.com/apache/spark/pull/468
This PR may still overstep what you have in mind, but let me put it on the
table to start. Besides fixing the issue, it has one substantive change, and
that is to manage Hadoop-specific things only in Hadoop-related profiles. This
does _not_ remove `yarn.version`.
- Moves the YARN and Hadoop profiles together in pom.xml. Sorry that this
makes the diff a little hard to grok but the changes are only as follows.
- Removes `hadoop.major.version`
- Introduce `hadoop-2.2` and `hadoop-2.3` profiles to control
Hadoop-specific changes:
- like the protobuf version issue - this was only 'solved' now by
enabling YARN for 2.2+, which is really an orthogonal issue
- like the jets3t version issue now
- Hadoop profiles set an appropriate default `hadoop.version`, that can be
overridden
- _(YARN profiles in the parent now only exist to add the sub-module)_
- Fixes the jets3t dependency issue
- and makes it a runtime dependency
- and centralizes config of this guy in the parent pom
- Updates build docs
- Updates SBT build too
- and fixes a regex problem along the way
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-1556
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/629.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #629
----
commit f21f35651dc8e9b2036f5e487f7465d844d35c72
Author: Sean Owen <[email protected]>
Date: 2014-05-04T09:08:15Z
Build changes to set up for jets3t fix
commit bbed8262fbaaad8037781e0e32b5141774250839
Author: Sean Owen <[email protected]>
Date: 2014-05-04T09:15:30Z
Use jets3t 0.9.0 for Hadoop 2.3+ (and correct similar regex issue in SBT
build)
commit 274f4f989a3fd27aba2ede75d3c7b713547aab68
Author: Sean Owen <[email protected]>
Date: 2014-05-04T09:21:58Z
Make jets3t a runtime dependency, and bring its exclusion up into parent
config
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
