[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-7443:
---------------------------------------
Attachment: HDFS-7443.001.patch
This patch changes the upgrade path from non-blockid-based-layout to
blockid-based layout so that it uses the jdk7 {{Files#createLink}} function,
instead of our hand-rolled hardlink code.
This avoids the dilemma of detecting EEXIST on all the various platforms that
we hacked in support for in {{HardLink.java}}, such as Linux shell-based (no
libhadoop.so case), cygwin, Windows native, and Linux JNI-based. It might be
possible to distinguish regular errors from EEXIST on all those platforms, but
writing all that code would be a very big job.
I did not remove or alter any other code in {{HardLink.java}} in this patch. I
think clearly we should think about refactoring that code to use jdk7 later,
but that is a bigger change that is not as critical as this fix. We also can't
get rid of {{HardLink.java}} completely because we are unfortunately depending
on reading the hard link count of files in a few places-- something jdk7 does
not support.
Another weird thing about the {{HardLink}} class is that all it actually
contains is statistics information-- every important method is {{static}}. So
that's why we continue to use a {{HardLink}} instance in the upgrade code. I
think in the future, we should simply use the {{HardLink#Statistics}} class
directly, since the outer class provides no value (it has only static methods).
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> ------------------------------------------------------------------------------------------------------
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Kihwal Lee
> Assignee: Colin Patrick McCabe
> Priority: Blocker
> Attachments: HDFS-7443.001.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
