[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448829#comment-16448829
]
Sean Busbey commented on HBASE-20332:
-------------------------------------
{quote}
patch you uploaded appears to actually be 4 patches, the first three of which
have been committed.
{quote}
Right, it depended on them and they weren't committed at the time. I'll rebase
and omit them on the next version.
{quote}
hbase-anti needs to print a footer link, I can't even find the results by
poking in the build output patchprocess directory. (fine to do in a follow on
if you can manually link me to the results)
{quote}
hbase-anti doesn't output anything to a log file AFAICT. it just greps the
patch file itself and throws up a vote. I can update it to output to a log and
put something in the footer, but it's going to give line numbers in the patch
file. will that be too confusing?
{quote}
htrace is retiring. there is a chance that it will go back to using org.htrace
package space if it lives on somewhere. we'll address that if it happens, i
suppose.
are we supposed to still exclude htrace from the ban transitive deps enforcer
rule? this is getting confusing.
{quote}
yeah a problem for another day I think; it's a mess for sure. I don't believe
we can ban it from the transitive dependencies so long as Hadoop needs it to
run (which is does for Hadoop < 2.8).
{quote}
bravo on commenting why we need some of the dependencies. valiant effort, i'm
sure it will be stale in two weeks, but at least it was up to date once.
{quote}
It is my honor an privilege to push the rock up the hill a few more times. ;)
{quote}
do we end up with jackson 1/2 conflict between ourselves and hadoop? looks like
you massaged it all away, maybe? do we need to make BlockCacheUtil and
ObjectModel go away?
{quote}
As far as I know this patch maintains any previous massaging of jackson 1/2
conflicts and does no new massaging of such conflicts. Maybe I have a side
effect I'm not seeing though?
I'd very much like to see the JSON needs in core modules removed, but it looked
like more work than was wise to fold into this change since I want it in 1.y
and 2.y.
{quote}
so now we build a shaded with hadoop and shaded without hadoop MR artifact?
{quote}
Yes. I have another jira that's a sibling to this one to make a shaded client
artifact without hadoop as well. That one is harder because artifact naming
will likely make the 3.0 version breaking compared to earlier lines.
{quote}
did we not have a netty.hadoop.version defined before? could've sworn I've seen
it
{quote}
We definitely had netty.hadoop.version prior to this patch. But we forgot to
include it in the "defaults when no profile is active" section that we added
for non-maven build systems.
> shaded mapreduce module shouldn't include hadoop
> ------------------------------------------------
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
> Affects Versions: 2.0.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
