[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445662#comment-16445662
]
Sean Busbey commented on HBASE-20332:
-------------------------------------
v3 correct unit test failures
- added a precommit check for old htrace uses
- stop excluding Hadoop's version of htrace
We've been excluding HTrace in most but not all places we reference Hadoop
artifacts, then selectively trying to add it back in since Hadoop needs it to
run. I think this was to avoid having it available at compile time to our code,
so we'd catch incorrect uses of the old version. Rather than all these
acrobatics, I've switched to a precommit check that looks for the incorrect
imports. Here's what it looks like:
{code}
-1 overall
_____ _ _ _
| ___|_ _(_) |_ _ _ __ ___| |
| |_ / _` | | | | | | '__/ _ \ |
| _| (_| | | | |_| | | | __/_|
|_| \__,_|_|_|\__,_|_| \___(_)
| Vote | Subsystem | Runtime | Comment
============================================================================
| | | | Prechecks
| -1 | hbaseanti | 0m 00s | The patch appears use HTrace 3 classes
| | | | instead of HTrace 4.
| -1 | hbaseanti | 0m 00s | The patch appears use pre-ASF HTrace 3
| | | | classes instead of HTrace 4.
| | | | HBASE-20332 Compile Tests
| 0 | mvndep | 0m 48s | Maven dependency ordering for branch
| | | | Patch Compile Tests
| 0 | mvndep | 0m 38s | Maven dependency ordering for patch
| | | | Other Tests
| | | 2m 08s |
|| Subsystem || Report/Notes ||
============================================================================
| Optional Tests | hbaseanti |
| uname | Darwin Busbey-MBA.local 15.6.0 Darwin Kernel Version 15.6.0: Tue Jan
30 11:45:51 PST 2018; root:xnu-3248.73.8~1/RELEASE_X86_64 x86_64 |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-20332 / f2d5fa43d3 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T02:58:13-05:00) |
| modules | C: hbase-common hbase-replication U: . |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
{code}
I think this will do, and it avoids us having a ton of references to a
dependency we don't even want. (FWIW, Hadoop 3 and Hadoop 2.8+ rely on htrace
4, so this also avoids us manually including an htrace version that's totally
unneeded by folks with those versions)
> shaded mapreduce module shouldn't include hadoop
> ------------------------------------------------
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
> Affects Versions: 2.0.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
