The fix in HDFS-9427 can potentially bring in new customers because less
chance for new comer to encountering “port already in use” problem. If we
make change according to HDFS-12990, then this incompatible change does not
make incompatible change compatible. Other ports are not reverted
according to HDFS-12990. User will encounter the bad taste in the mouth
that HDFS-9427 attempt to solve. Please do consider both negative side
effects of reverting as well as incompatible minor release change. Thanks
Regards,
Eric
From: larry mccay <lmc...@apache.org>
Date: Wednesday, January 10, 2018 at 10:53 AM
To: Daryn Sharp <da...@oath.com>
Cc: "Aaron T. Myers" <a...@apache.org>, Eric Yang <ey...@hortonworks.com>,
Chris Douglas <cdoug...@apache.org>, Hadoop Common <
common-dev@hadoop.apache.org>
Subject: Re: When are incompatible changes acceptable (HDFS-12990)
On Wed, Jan 10, 2018 at 1:34 PM, Daryn Sharp <da...@oath.com<mailto:
da...@oath.com>> wrote:
I fully agree the port changes should be reverted. Although
"incompatible", the potential impact to existing 2.x deploys is huge. I'd
rather inconvenience 3.0 deploys that compromise <1% customers. An
incompatible change to revert an incompatible change is called
compatibility.
+1
Most importantly, consider that there is no good upgrade path existing
deploys, esp. large and/or multi-cluster environments. It’s only feasible
for first-time deploys or simple single-cluster upgrades willing to take
downtime. Let's consider a few reasons why:
1. RU is completely broken. Running jobs will fail. If MR on hdfs
bundles the configs, there's no way to transparently coordinate the switch
to the new bundle with the port changed. Job submissions will fail.
2. Users generally do not add the rpc port number to uris so unless their
configs are updated they will contact the wrong port. Seamlessly
coordinating the conf change without massive failures is impossible.
3. Even if client confs are updated, they will break in a multi-cluster
env with NNs using different ports. Users/services will be forced to add
the port. The cited hive "issue" is not a bug since it's the only way to
work in a multi-port env.
4. Coordinating the port add/change of uris is systems everywhere (you
know something will be missed), updating of confs, restarting all services,
requiring customers to redeploy their workflows in sync with the NN
upgrade, will cause mass disruption and downtime that will be unacceptable
for production environments.
This is a solution to a non-existent problem. Ports can be bound by
multiple processes but only 1 can listen. Maybe multiple listeners is an
issue for compute nodes but not responsibly managed service nodes. Ie. Who
runs arbitrary services on the NNs that bind to random ports? Besides, the
default port is and was ephemeral so it solved nothing.
This either standardizes ports to a particular customer's ports or is a
poorly thought out whim. In either case, the needs of the many outweigh
the needs of the few/none (3.0 users). The only logical conclusion is
revert. If a particular site wants to change default ports and deal with
the massive fallout, they can explicitly change the ports themselves.
Daryn
On Tue, Jan 9, 2018 at 11:22 PM, Aaron T. Myers <a...@apache.org<mailto:
a...@apache.org>> wrote:
On Tue, Jan 9, 2018 at 3:15 PM, Eric Yang <ey...@hortonworks.com<mailto:
ey...@hortonworks.com>> wrote:
While I agree the original port change was unnecessary, I don’t think
Hadoop NN port change is a bad thing.
I worked for a Hadoop distro that NN RPC port was default to port 9000.
When we migrate from BigInsights to IOP and now to HDP, we have to move
customer Hive metadata to new NN RPC port. It only took one developer
(myself) to write the tool for the migration. The incurring workload is
not as bad as most people anticipated because Hadoop depends on
configuration file for referencing namenode. Most of the code can work
transparently. It helped to harden the downstream testing tools to be
more
robust.
While there are of course ways to deal with this, the question really
should be whether or not it's a desirable thing to do to our users.
We will never know how many people are actively working on Hadoop 3.0.0.
Perhaps, couple hundred developers or thousands.
You're right that we can't know for sure, but I strongly suspect that this
is a substantial overestimate. Given how conservative Hadoop operators tend
to be, I view it as exceptionally unlikely that many deployments have been
created on or upgraded to Hadoop 3.0.0 since it was released less than a
month ago.
Further, I hope you'll agree that the number of
users/developers/deployments/applications which are currently on Hadoop 2.x
is *vastly* greater than anyone who might have jumped on Hadoop 3.0.0 so
quickly. When all of those users upgrade to any 3.x version, they will
encounter this needless incompatible change and be forced to work around
it.
I think the switch back may have saved few developers work, but there
could be more people getting impacted at unexpected minor release change
in
the future. I recommend keeping current values to avoid rule bending and
future frustrations.
That we allow this incompatible change now does not mean that we are
categorically allowing more incompatible changes in the future. My point is
that we should in all instances evaluate the merit of any incompatible
change on a case-by-case basis. This is not an exceptional circumstance -
we've made incompatible changes in the past when appropriate, e.g. breaking
some clients to address a security issue. I and others believe that in this
case the benefits greatly outweigh the downsides of changing this back to
what it has always been.
Best,
Aaron
Regards,
Eric
On 1/9/18, 11:21 AM, "Chris Douglas" <cdoug...@apache.org<mailto:
cdoug...@apache.org>> wrote:
Particularly since 9820 isn't in the contiguous range of ports in
HDFS-9427, is there any value in this change?
Let's change it back to prevent the disruption to users, but
downstream projects should treat this as a bug in their tests. Please
open JIRAs in affected projects. -C
On Tue, Jan 9, 2018 at 5:18 AM, larry mccay <lmc...@apache.org
<mailto:lmc...@apache.org>> wrote:
> On Mon, Jan 8, 2018 at 11:28 PM, Aaron T. Myers <a...@apache.org
<mailto:a...@apache.org>>
wrote:
>
>> Thanks a lot for the response, Larry. Comments inline.
>>
>> On Mon, Jan 8, 2018 at 6:44 PM, larry mccay <lmc...@apache.org
<mailto:lmc...@apache.org>>
wrote:
>>
>>> Question...
>>>
>>> Can this be addressed in some way during or before upgrade that
allows it
>>> to only affect new installs?
>>> Even a config based workaround prior to upgrade might make this a
change
>>> less disruptive.
>>>
>>> If part of the upgrade process includes a step (maybe even a
script) to
>>> set the NN RPC port explicitly beforehand then it would allow
existing
>>> deployments and related clients to remain whole - otherwise it
will uptake
>>> the new default port.
>>>
>>
>> Perhaps something like this could be done, but I think there are
downsides
>> to anything like this. For example, I'm sure there are plenty of
>> applications written on top of Hadoop that have tests which
hard-code the
>> port number. Nothing we do in a setup script will help here. If we
don't
>> change the default port back to what it was, these tests will
likely all
>> have to be updated.
>>
>>
>
> I may not have made my point clear enough.
> What I meant to say is to fix the default port but direct folks to
> explicitly set the port they are using in a deployment (the current
> default) so that it doesn't change out from under them - unless
they
are
> fine with it changing.
>
>
>>
>>> Meta note: we shouldn't be so pedantic about policy that we can't
back
>>> out something that is considered a bug or even mistake.
>>>
>>
>> This is my bigger point. Rigidly adhering to the compat guidelines
in this
>> instance helps almost no one, while hurting many folks.
>>
>> We basically made a mistake when we decided to change the default
NN port
>> with little upside, even between major versions. We discovered
this
very
>> quickly, and we have an opportunity to fix it now and in so doing
likely
>> disrupt very, very few users and downstream applications. If we
don't
>> change it, we'll be causing difficulty for our users, downstream
>> developers, and ourselves, potentially for years.
>>
>
> Agreed.
>
>
>>
>> Best,
>> Aaron
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
<mailto:common-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
<mailto:common-dev-h...@hadoop.apache.org>
