[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16513970#comment-16513970
]
Sean Busbey commented on HBASE-20332:
-------------------------------------
okay I got it.
the checkstyle output itself is not zero:
*
https://builds.apache.org/job/PreCommit-HBASE-Build/13265/artifact/patchprocess/maven-branch-checkstyle-root.txt
*
https://builds.apache.org/job/PreCommit-HBASE-Build/13265/artifact/patchprocess/maven-patch-checkstyle-root.txt
The patch version definitely shows a bunch of new warnings for htrace stuff.
The files I was looking at are the filtered versions for "checkstyle warnings
we care about"
Yetus' checkstyle plugin has an optimizations
([ref|https://github.com/apache/yetus/blob/rel/0.7.0/precommit/test-patch.d/checkstyle.sh#L213])
where it cuts the output down to just the files changed by the patch. We'll
need a way to flag that we care about all differences currently. I suspect
that'll mean either a change to yetus or keeping our own checkstyle plugin.
I'll link a jira once I figure out which. In either case I think the scope of
doing that is going to be too much to wait on.
Here's a manual diff of the two, there are 61 instances (for context, we have
16,189 existing checkstyle errors on master).
{code}
375a376,381
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:21:1:
> Illegal import - org.apache.htrace.core.HTraceConfiguration. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:22:1:
> Illegal import - org.apache.htrace.core.Sampler. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:23:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:24:1:
> Illegal import - org.apache.htrace.core.SpanReceiver. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:25:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:26:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
376a383
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/SpanReceiverHost.java:25:1:
> Illegal import - org.apache.htrace.core.SpanReceiver. [IllegalImport]
377a385
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/HBaseHTraceConfiguration.java:22:1:
> Illegal import - org.apache.htrace.core.HTraceConfiguration. [IllegalImport]
1326a1335
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java:66:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
1337a1347,1348
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java:33:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java:34:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
3534a3546
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java:51:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
4395a4408
> [ERROR]
> hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:32:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
5312a5326,5328
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java:26:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java:27:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java:28:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
5955a5972
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java:63:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
7516a7534
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:177:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8518a8537
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java:54:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8586a8606
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java:80:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8603a8624
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java:56:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8635a8657
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java:58:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
11309a11332,11337
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/trace/TestHTraceHooks.java:34:1:
> Illegal import - org.apache.htrace.core.POJOSpanReceiver. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/trace/TestHTraceHooks.java:35:1:
> Illegal import - org.apache.htrace.core.Sampler. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/trace/TestHTraceHooks.java:36:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/trace/TestHTraceHooks.java:37:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/trace/TraceTree.java:20:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/trace/TraceTree.java:21:1:
> Illegal import - org.apache.htrace.core.SpanId. [IllegalImport]
11652a11681,11684
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java:72:1:
> Illegal import - org.apache.htrace.core.ProbabilitySampler. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java:73:1:
> Illegal import - org.apache.htrace.core.Sampler. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java:74:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
> [ERROR]
> hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java:75:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
13907a13940,13942
> [ERROR]
> hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java:105:1:
> Illegal import - org.apache.htrace.core.ProbabilitySampler. [IllegalImport]
> [ERROR]
> hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java:106:1:
> Illegal import - org.apache.htrace.core.Sampler. [IllegalImport]
> [ERROR]
> hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java:107:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
14892a14928,14929
> [ERROR]
> hbase-it/src/test/java/org/apache/hadoop/hbase/trace/IntegrationTestSendTraceRequests.java:37:1:
> Illegal import - org.apache.htrace.core.Sampler. [IllegalImport]
> [ERROR]
> hbase-it/src/test/java/org/apache/hadoop/hbase/trace/IntegrationTestSendTraceRequests.java:38:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
15074a15112,15114
> [ERROR]
> hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java:67:1:
> Illegal import - org.apache.htrace.core.AlwaysSampler. [IllegalImport]
> [ERROR]
> hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java:68:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java:69:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
16214a16255
> [ERROR]
> hbase-external-blockcache/src/main/java/org/apache/hadoop/hbase/io/hfile/MemcachedBlockCache.java:44:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
{code}
> shaded mapreduce module shouldn't include hadoop
> ------------------------------------------------
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
> Affects Versions: 2.0.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
