Xiaoyu Yao created HDFS-9230:
--------------------------------
Summary: Report space overhead of unfinalized
upgrade/rollingUpgrade
Key: HDFS-9230
URL: https://issues.apache.org/jira/browse/HDFS-9230
Project: Hadoop HDFS
Issue Type: Improvement
Components: HDFS
Reporter: Xiaoyu Yao
DataNodes do not delete block files during upgrades to allow rollback. This is
often confusing to administrators since they sometimes delete files before
finalize upgrade but don't see the DFS used space reduce.
Ideally, HDFS should report the un-finalized upgrade overhead along with its
message on NN UI "Upgrade in progress. Not yet finalized." Or, this can be
improve with better NN UI message and document that space won't be reclaimed
for deletion until upgrade is finalized.
For non-rolling upgrade, it is not easy to track this due to hard link. Say NN
initialized upgrade at T1, the block files on DNs that exist before T1 are
still under 'current' directory but is just a hard link to 'previous'
directory. When those files are deleted after T1 due to deletion, the block
file usage on DN won't get deleted until upgrade is finalized.
So we need to book keeping files created before T1 but deleted after T1 as the
un-finalized upgrade overhead here.
For rolling upgrade, it is relative easy to track space overhead as we are not
using hard link.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
