[
https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529416#comment-14529416
]
Chris Nauroth commented on HDFS-5223:
-------------------------------------
bq. The fact that presently all features are always enabled means that we
should consider ourselves obligated to make sure that all features work well
with all other features.
Yes, definitely agreed.
I don't think I communicated this point clearly. What I meant was that I see
additional complexity arising from the new combinations of "feature X on while
feature Y off". These represent new system states not covered by existing
tests, and this is where we get a combinatorial explosion in the test matrix.
It's particularly challenging if one feature is coupled to another, either in
code or as an implementation pre-requisite.
As an example, an early design proposal for ACLs would have involved
implementing xattrs first, followed by implementing ACLs in terms of private
xattrs. (This is a common implementation in other file systems.) We didn't do
it this way, but if we had, then we'd have a situation where the
{{EXTENDED_ACL}} feature is dependent upon {{XATTRS}}. What is the effect of
disabling {{XATTRS}} while {{EXTENDED_ACL}} is enabled? I suppose the correct
response is to block disabling {{XATTRS}} if {{EXTENDED_ACL}} is still on.
This becomes extra code to write and test. It also becomes extra knowledge for
operators to be aware that both must be enabled before the feature can be used.
The monotonically increasing (well, technically decreasing!) layout version has
the benefit of restricting possible system states, because it guarantees that
prior features in the lineage are enabled. The drawback is that it harms
flexibility. In this particular case, I prefer keeping that invariant and the
safety it brings over the increased flexibility.
bq. One might want to use the OOB ack feature just when doing a rolling restart
(no upgrade) to effect a configuration change, without the additional
complexity of metadata changes, etc.
FWIW, the existing rolling upgrade functionality doesn't really dictate what it
is that you're upgrading, and the design targeted a DN-only upgrade as one of
its use cases. It would be completely legitimate to skip the NN portion of the
rolling upgrade procedure and do just the DN portion to push a configuration
change with no code changes, like increasing the xceiver count.
> Allow edit log/fsimage format changes without changing layout version
> ---------------------------------------------------------------------
>
> Key: HDFS-5223
> URL: https://issues.apache.org/jira/browse/HDFS-5223
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.1.1-beta
> Reporter: Aaron T. Myers
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-5223-HDFS-Downgrade-Extended-Support.pdf,
> HDFS-5223.004.patch, HDFS-5223.005.patch
>
>
> Currently all HDFS on-disk formats are version by the single layout version.
> This means that even for changes which might be backward compatible, like the
> addition of a new edit log op code, we must go through the full `namenode
> -upgrade' process which requires coordination with DNs, etc. HDFS should
> support a lighter weight alternative.
> Copied description from HDFS-8075 which is a duplicate and now closed. (by
> sanjay on APril 7 2015)
> Background
> * HDFS image layout was changed to use Protobufs to allow easier forward and
> backward compatibility.
> * Hdfs has a layout version which is changed on each change (even if it an
> optional protobuf field was added).
> * Hadoop supports two ways of going back during an upgrade:
> ** downgrade: go back to old binary version but use existing image/edits so
> that newly created files are not lost
> ** rollback: go back to "checkpoint" created before upgrade was started -
> hence newly created files are lost.
> Layout needs to be revisited if we want to support downgrade is some
> circumstances which we dont today. Here are use cases:
> * Some changes can support downgrade even though they was a change in layout
> since there is not real data loss but only loss of new functionality. E.g.
> when we added ACLs one could have downgraded - there is no data loss but you
> will lose the newly created ACLs. That is acceptable for a user since one
> does not expect to retain the newly added ACLs in an old version.
> * Some changes may lead to data-loss if the functionality was used. For
> example, the recent truncate will cause data loss if the functionality was
> actually used. Now one can tell admins NOT use such new such new features
> till the upgrade is finalized in which case one could potentially support
> downgrade.
> * A fairly fundamental change to layout where a downgrade is not possible but
> a rollback is. Say we change the layout completely from protobuf to something
> else. Another example is when HDFS moves to support partial namespace in
> memory - they is likely to be a fairly fundamental change in layout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
