Hi Sanjeev,
Thank you. It does help.
At what points is the checksum calculated?
Thx,
TK
On 10/20/2020 3:03 AM, संजीव (Sanjeev Tripurari) wrote:
For Missing blocks and corrupted blocks, do check if all the datanode
services are up, non of the disks where hdfs data is stored is
accessible and have no issues, hosts are reachable from namenode,
If you are able to re-generate the data and write its great, otherwise
hadoop cannot correct itself.
Could you please elaborate on this? Does it mean I have to continuously
access a file for HDFS to be able to detect corrupt blocks and correct
itself?
*"Does HDFS check that the data node is up, data disk is mounted, path to
the file exists and file can be read?"*
-- yes, only after it fails it will say missing blocks.
*Or does it also do a filesystem check on that data disk as well as
perhaps a checksum to ensure block integrity?*
-- yes, every file cheksum is maintained and cross checked, if it
fails it will say corrupted blocks.
hope this helps.
-Sanjeev
*
*
On Tue, 20 Oct 2020 at 09:52, TomK <tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Hello,
HDFS Missing Blocks / Corrupt Blocks Logic: What are the specific
checks done to determine a block is bad and needs to be replicated?
Does HDFS check that the data node is up, data disk is mounted,
path to
the file exists and file can be read?
Or does it also do a filesystem check on that data disk as well as
perhaps a checksum to ensure block integrity?
I've googled on this quite a bit. I don't see the exact answer I'm
looking for. I would like to know exactly what happens during file
integrity verification that then constitutes missing blocks or
corrupt
blocks in the reports.
--
Thank You,
TK.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
<mailto:user-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: user-h...@hadoop.apache.org
<mailto:user-h...@hadoop.apache.org>
