[
https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752889#comment-15752889
]
Sean Mackrory commented on HDFS-11096:
--------------------------------------
I've been through the JACC output [~andrew.wang] linked to and done some
triage. My comments are below - at this point I think it should be paired with
a discussion of which types of compatibility we want to allow (e.g. I'm
thinking backwards compatibility for clients so old clients off-cluster
continue to work during rolling upgrade, wire-compatibility to enabled the
rolling upgrade itself, and while we're at it doing due diligence on the
annotated intentions of compatibility).
- I think we can ignore "Added Methods" and "Other Changes in Data Types" (all
of those were just added fields). Same goes for cases where functions are no
longer abstract, throw more or fewer exception types, were overriden by a
subclass, were moved to a superclass, or are / aren't synchronized now. These
changes are unlikely to impact backward compatibility (by that I mean the
ability to use older clients and applications with newer versions of Hadoop).
They are likely to impact forward compatibility (i.e. a guarantee that current
Hadoop clusters will work for future Hadoop clients & applications), but I
think forward compatibility is less important and far less practical. It makes
it harder to introduce improvements (and isn't that the whole point of a major
release) and is a much hard guarantee to make.
- The following items have been deprecated at least since Hadoop 2.2, and I
think we can safely ignore these incompatibilities: FSDataOutputStream (at
least it's constructor), Hadoop Record I/O, Syncable.sync,
permission.AccessControlException, all removed functions in
DistributedFileSystem except getFileBlockStorageLocations, and
DEPRECATED_UMASK_LABEL.
- The org.apache.hadoop.ipc.ProtobufRpcEngine.Rpc* classes are LimitedPrivate
to RPC, and DomainSocket is LimitedPrivate to HDFS. I haven't looked too deep
into the RPC classes yet, but I believe neither of these affects
wire-compatibility, and due to their scope can be safely removed.
- The HttpServer removal is also okay. It is LimitedPrivate to HBase,
explicitly for compatibility on Hadoop 2.2 and 2.3. Doesn't affect
wire-compatibility, so IMO this is not concerning. I'll double check that it's
not still in use in recent HBase releases.
- The metrics framework has been removed, but I do not believe it was
deprecated. The metrics2 framework has also migrated to commons-configuration2,
which causes some slight incompatibilities. These may indeed be considered
incompatibilities, but perhaps we're okay with monitoring being less compatible
due to it's unique nature? It would be a shame to wait until Hadoop 4.x to drop
some baggage like that. HttpServer2 also had several similar changes because of
a Jetty upgrade.
- org.apache.hadoop.fs.s3.* has been completely removed in Hadoop 3. I do not
believe it was officially deprecated until quite recently. I believe it was
considered de-facto deprecated for a while, so possibly something we want to
consider dropping anyway, again, just because of the level of baggage.
- I haven't looked closely at org.apache.hadoop.mapreduce.* or
org.apache.hadoop.yarn.* yet, just as it's a bit less familiar.
- There are some classes that move between hadoop-hdfs and hadoop-hdfs-clients,
etc. but I don't think that affects us in any way we care about.
- SortedMapWritable was marked Public and Stable and was not deprecated in
Hadoop 2. NameNodeMXBean is the same. Not sure how significant these are from a
practical perspective, but they're the most serious contradictions of the
annotations.
- All other missing classes are Evolving or Unstable, and many are
LimitedPrivate (for instance, BlockStorageLocation and
DistributedFileSystem.getFileBlockStorageLocations).
- There are a lot of changes to LOG fields: migration from log4j to slf4j, make
it private or final, etc. We can easily reverse the latter changes, but I think
it's unlikely and unwise for people to be depending on class's LOGs (not sure
why they're not at least protected to begin with). I think the migration to
slf4j makes it worth it if it doesn't break practical compatibility in a big
way.
- Some items have reduced their visibility (Security.SecurityUtil, Shell.Shell,
CacheManager.setCachedLocations, ResourceCalculatorPlugin). That's easy to undo
if it's aproblem.
- Some items have become static or reduced their visibility. Easy to create
non-static proxy functions for static counterparts if we need to and reverse
the visibility changes... One item became final, but it was named in all caps
so I think it was sufficiently clear before that it was intended as a constant
(unless we want to be very strict to the contract).
- Some configuration key constants have been removed. They're in classes
annotated as public. We can add them back for Java API compatibility, but
they'd be meaningless otherwise.
- DU no longer extends Shell, FileStatus no longer implements Comparable
(probably easy to put back if we want) and the AccessControlException no longer
extends a different AccessControlException that was deprecated and is now
removed.
- A slight tweak in zkfc's USAGE text - which reminds me we should probably
look at any significant changes to the Hadoop CLI as well.
I agree testing should be a significant part of this to catch other issues.
Even if it something doesn't break the contract per se if it's a big enough
deal for compatibility and reasonable to fix, I'm sure we'll want to do that. I
can help with the testing and some of the changes suggested above. But please
let me know if any of the above analysis seems incorrect.
> Support rolling upgrade between 2.x and 3.x
> -------------------------------------------
>
> Key: HDFS-11096
> URL: https://issues.apache.org/jira/browse/HDFS-11096
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rolling upgrades
> Affects Versions: 3.0.0-alpha1
> Reporter: Andrew Wang
> Priority: Blocker
>
> trunk has a minimum software version of 3.0.0-alpha1. This means we can't
> rolling upgrade between branch-2 and trunk.
> This is a showstopper for large deployments. Unless there are very compelling
> reasons to break compatibility, let's restore the ability to rolling upgrade
> to 3.x releases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
