[
https://issues.apache.org/jira/browse/HDFS-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639217#comment-13639217
]
Chris Nauroth commented on HDFS-4732:
-------------------------------------
There are 2 components to this patch:
# I'm providing a new copy of hadoop-22-dfs-dir.tgz, attached to this issue.
It's equivalent to the old one, but does not contain links. As a side effect,
the file is bigger, because bytes are duplicated where previously they were
shared via hard links. Before: ~170K compressed, ~40 MB uncompressed. After:
~310K compressed, ~80 MB uncompressed. The compressed version is what's stored
in the repo, so this doesn't really make the source tree much bigger.
# While investigating, I noticed a code path in {{TestDFSUpgradeFromImage}}
where we could fail to shutdown a {{MiniDFSCluster}}. In practice, this isn't
an issue, but I'd like to clean it up.
To the committer: please download and commit hadoop-22-dfs-dir.tgz independent
of the patch file. It's a binary file, so I couldn't include it in the patch
file. It needs to go in hadoop-hdfs-project/hadoop-hdfs/src/test/resources and
overwrite the existing file.
I tested this successfully on Mac (bsdtar), Ubuntu (GNU tar), and Windows
(commons-compress).
> TestDFSUpgradeFromImage fails on Windows due to failure to unpack old image
> tarball that contains hard links
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-4732
> URL: https://issues.apache.org/jira/browse/HDFS-4732
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 3.0.0
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: hadoop-22-dfs-dir.tgz
>
>
> On non-Windows, {{FileUtil#unTar}} is implemented using external Unix shell
> commands. On Windows, {{FileUtil#unTar}} is implemented in Java using
> commons-compress. {{TestDFSUpgradeFromImage}} uses a testing tarball image
> of an old HDFS layout version, hadoop-22-dfs-dir.tgz. This file contains
> hard links. It appears that commons-compress cannot handle the hard links
> correctly. When it unpacks the file, each hard link ends up as a 0-length
> file. This causes the test to fail during cluster startup, because the
> 0-length block files are considered corrupt.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
