Hi Vinay,
If I rephrase the question,
Does a RU rollback snapshot provide a consistent snapshot of the distributed
file system?
I don't think we aimed it to be a completely consistent snapshot. It is meant
to be a safe place to go back with the old version of software. This is
normally used as a last resort. By design, the datanodes will have extra blocks
on rollback, which will be invalidated quickly. But the short presence of
blocks with "future" block ids still can interfere with block allocations after
rolling back, if the cluster is used right away. As you pointed out, the
under-construction block length is not recorded either.
>But now, extra bytes are seen after rollback. Is this correct?I think it is a
>reasonable compromise. If you can make a general argument against it, we can
>revisit the design and try to fix it.
Kihwal
From: Vinayakumar B <vinayakumar...@huawei.com>
To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Sent: Thursday, May 25, 2017 8:10 AM
Subject: RollingUpgrade Rollback openfiles issue
Hi all,
Have a doubt with expected behavior in case of RollingUpgrade and rollback.
Scenario:
1. file was being written before rolling upgrade started and written some
bytes, say X, with hsync().
2. Rolling upgrade done and writer continued to write and added some more bytes
and file closed with X+Y bytes.
3. Now rollback done.
i. Current state of the File is UNDERCONSTRUCTION.
Ii. getFileStatus() returns with size X. BUT in replicas there is a FINALIZED
replica with size X+Y.
iii. recoverLease() on the file closes the file with X+Y bytes.
Question:
What should be the size here after rollback + recoverLease()?
Since user always writes with hsync(), application might have some other
track of how much bytes written. But now, extra bytes are seen after rollback.
Is this correct?
-vinay
