Datanode's BlockScanner periodically verifies checksum of all the blocks on each datanode.
Koji On Wed, Oct 21, 2020 at 10:26 AM संजीव (Sanjeev Tripurari) < sanjeevtripur...@gmail.com> wrote: > Hi Tom > > Therefore, if I write a file to HDFS but access it two years later, then > the checksum will be computed only twice, at the beginning of the two years > and again at the end when a client connects? Correct? As long as no > process ever accesses the file between now and two years from now, the > checksum is never redone and compared to the two year old checksum in the > fsimage? > > yes, Exactly unless data is read checksum is not verified. (when data is > written and when the data is read), > if checksum is mismatched, there is no way to correct it, you will have to > re-write that file. > > When datanode is added back in, there is no real read operation on the > files themselves. The datanode just reports the blocks but doesn't really > read the blocks that are there to re-verify the files and ensure > consistency? > > yes, Exactly, datanode maintains list of files and their blocks, which it > reports, along with total disk size and used size. > Namenode only has list of blocks, unless datanodes is connected it wont > know where the blocks are stored. > > Regards > -Sanjeev > > > On Wed, 21 Oct 2020 at 18:31, TomK <tomk...@mdevsys.com> wrote: > >> Hey Sanjeev, >> >> Thank you very much again. This confirms my suspision. >> >> Therefore, if I write a file to HDFS but access it two years later, then >> the checksum will be computed only twice, at the beginning of the two years >> and again at the end when a client connects? Correct? As long as no >> process ever accesses the file between now and two years from now, the >> checksum is never redone and compared to the two year old checksum in the >> fsimage? >> >> When datanode is added back in, there is no real read operation on the >> files themselves. The datanode just reports the blocks but doesn't really >> read the blocks that are there to re-verify the files and ensure >> consistency? >> >> Thx, >> TK >> >> >> >> On 10/21/2020 12:38 AM, संजीव (Sanjeev Tripurari) wrote: >> >> Hi Tom, >> >> Every datanode sends heartbeat to namenode, on its list of blocks it has. >> >> When a datanode which is disconnected for a while, after connecting will >> send heartbeat to namenode, with list of blocks it has (till then namenode >> will have under-replicated blocks). >> As soon as the datanode is connected to namenode, it will clear >> under-replicatred blocks. >> >> *When a client connects to read or write a file, it will run checksum to >> validate the file.* >> >> There is no independent process running to do checksum, as it will be >> heavy process on each node. >> >> Regards >> -Sanjeev >> >> On Wed, 21 Oct 2020 at 00:18, Tom <t...@mdevsys.com> wrote: >> >>> Thank you. That part I understand and am Ok with it. >>> >>> What I would like to know next is when again the CRC32C checksum is ran >>> and checked against the fsimage that the block file has not changed or >>> become corrupted? >>> >>> For example, if I take a datanode out, and within 15 minutes, plug it >>> back in, does HDF rerun the CRC 32C on all data disks on that node to make >>> sure blocks are ok? >>> >>> Cheers, >>> TK >>> >>> Sent from my iPhone >>> >>> On Oct 20, 2020, at 1:39 PM, संजीव (Sanjeev Tripurari) < >>> sanjeevtripur...@gmail.com> wrote: >>> >>> its done as sson as a file is stored on disk.. >>> >>> Sanjeev >>> >>> On Tuesday, 20 October 2020, TomK <tomk...@mdevsys.com> wrote: >>> >>>> Thanks again. >>>> >>>> At what points is the checksum validated (checked) after that? For >>>> example, is it done on a daily basis or is it done only when the file is >>>> accessed? >>>> >>>> Thx, >>>> TK >>>> >>>> On 10/20/2020 10:18 AM, संजीव (Sanjeev Tripurari) wrote: >>>> >>>> As soon as the file is written first time checksum is calculated and >>>> updated in fsimage (first in edit logs), and same is replicated other >>>> replicas. >>>> >>>> >>>> >>>> On Tue, 20 Oct 2020 at 19:15, TomK <tomk...@mdevsys.com> wrote: >>>> >>>>> Hi Sanjeev, >>>>> >>>>> Thank you. It does help. >>>>> >>>>> At what points is the checksum calculated? >>>>> >>>>> Thx, >>>>> TK >>>>> >>>>> On 10/20/2020 3:03 AM, संजीव (Sanjeev Tripurari) wrote: >>>>> >>>>> For Missing blocks and corrupted blocks, do check if all the datanode >>>>> services are up, non of the disks where hdfs data is stored is accessible >>>>> and have no issues, hosts are reachable from namenode, >>>>> >>>>> If you are able to re-generate the data and write its great, otherwise >>>>> hadoop cannot correct itself. >>>>> >>>>> Could you please elaborate on this? Does it mean I have to >>>>> continuously access a file for HDFS to be able to detect corrupt blocks >>>>> and >>>>> correct itself? >>>>> >>>>> >>>>> >>>>> >>>>> *"Does HDFS check that the data node is up, data disk is mounted, path >>>>> to the file exists and file can be read?"* >>>>> -- yes, only after it fails it will say missing blocks. >>>>> >>>>> >>>>> *Or does it also do a filesystem check on that data disk as well as >>>>> perhaps a checksum to ensure block integrity?* >>>>> -- yes, every file cheksum is maintained and cross checked, if it >>>>> fails it will say corrupted blocks. >>>>> >>>>> hope this helps. >>>>> >>>>> -Sanjeev >>>>> >>>>> >>>>> On Tue, 20 Oct 2020 at 09:52, TomK <tomk...@mdevsys.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> HDFS Missing Blocks / Corrupt Blocks Logic: What are the specific >>>>>> checks done to determine a block is bad and needs to be replicated? >>>>>> >>>>>> Does HDFS check that the data node is up, data disk is mounted, path >>>>>> to >>>>>> the file exists and file can be read? >>>>>> >>>>>> Or does it also do a filesystem check on that data disk as well as >>>>>> perhaps a checksum to ensure block integrity? >>>>>> >>>>>> I've googled on this quite a bit. I don't see the exact answer I'm >>>>>> looking for. I would like to know exactly what happens during file >>>>>> integrity verification that then constitutes missing blocks or >>>>>> corrupt >>>>>> blocks in the reports. >>>>>> >>>>>> -- >>>>>> Thank You, >>>>>> TK. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >>>>>> For additional commands, e-mail: user-h...@hadoop.apache.org >>>>>> >>>>>> >>>>> >>>> -- >>>> Thx, >>>> TK. >>>> >>> >> -- >> Thx, >> TK. >> >
