[
https://issues.apache.org/jira/browse/HDFS-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Ramesh updated HDFS-12092:
---------------------------------
Description:
Restarted a datanode, corrupted the metafile for blk_1073741825 with something
like echo 'zzzzzzzz' > blk_1073741825_1001.meta, and datanode logs reveal that
the VolumeScanner exits due to an illegal argument exception. Here is the
relevant trace:
------------------------------------------------------------------------------------------------------
{code}
2017-07-05 22:03:41,878 [VolumeScannerThread()] DEBUG datanode.VolumeScanner:
start scanning block BP-955735389-###########-1494002319684:blk_1073741825_1001
2017-07-05 22:03:41,879 [VolumeScannerThread()] ERROR datanode.VolumeScanner:
VolumeScanner() exiting because of exception
java.lang.IllegalArgumentException: id=122 out of range [0, 5)
at
org.apache.hadoop.util.DataChecksum$Type.valueOf(DataChecksum.java:67)
at
org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:123)
at
org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:178)
at
org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:142)
at
org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:156)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.loadLastPartialChunkChecksum(FsVolumeImpl.java:1022)
at
org.apache.hadoop.hdfs.server.datanode.FinalizedReplica.getLastChecksumAndDataLen(FinalizedReplica.java:104)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:259)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:484)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:614)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:699)
2017-07-05 22:03:41,879 [VolumeScannerThread()] INFO datanode.VolumeScanner:
VolumeScanner() exiting.
{code}
was:
Restarted a datanode, corrupted the metafile for blk_1073741825 with something
like echo 'zzzzzzzz' > blk_1073741825_1001.meta, and datanode logs reveal that
the VolumeScanner exits due to an illegal argument exception. Here is the
relevant trace
2017-07-05 22:03:41,878
[VolumeScannerThread(/grid/0/tmp/hadoop-hdfsqa/dfs/data)] DEBUG
datanode.VolumeScanner: start scanning block
BP-955735389-10.215.76.172-1494002319684:blk_1073741825_1001
2017-07-05 22:03:41,879
[VolumeScannerThread(/grid/0/tmp/hadoop-hdfsqa/dfs/data)] ERROR
datanode.VolumeScanner: VolumeScanner(/grid/0/tmp/hadoop-hdfsqa/dfs/data,
DS-7817e9a3-c179-4901-8757-af965b27b689) exiting because of exception
java.lang.IllegalArgumentException: id=122 out of range [0, 5)
at
org.apache.hadoop.util.DataChecksum$Type.valueOf(DataChecksum.java:67)
at
org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:123)
at
org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:178)
at
org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:142)
at
org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:156)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.loadLastPartialChunkChecksum(FsVolumeImpl.java:1022)
at
org.apache.hadoop.hdfs.server.datanode.FinalizedReplica.getLastChecksumAndDataLen(FinalizedReplica.java:104)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:259)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:484)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:614)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:699)
2017-07-05 22:03:41,879
[VolumeScannerThread(/grid/0/tmp/hadoop-hdfsqa/dfs/data)] INFO
datanode.VolumeScanner: VolumeScanner(/grid/0/tmp/hadoop-hdfsqa/dfs/data,
DS-7817e9a3-c179-4901-8757-af965b27b689) exiting.
> VolumeScanner exits when block metadata file is corrupted on datanode.
> ----------------------------------------------------------------------
>
> Key: HDFS-12092
> URL: https://issues.apache.org/jira/browse/HDFS-12092
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs
> Affects Versions: 2.8.0
> Reporter: Ashwin Ramesh
>
> Restarted a datanode, corrupted the metafile for blk_1073741825 with
> something like echo 'zzzzzzzz' > blk_1073741825_1001.meta, and datanode logs
> reveal that the VolumeScanner exits due to an illegal argument exception.
> Here is the relevant trace:
> ------------------------------------------------------------------------------------------------------
> {code}
> 2017-07-05 22:03:41,878 [VolumeScannerThread()] DEBUG datanode.VolumeScanner:
> start scanning block
> BP-955735389-###########-1494002319684:blk_1073741825_1001
> 2017-07-05 22:03:41,879 [VolumeScannerThread()] ERROR datanode.VolumeScanner:
> VolumeScanner() exiting because of exception
> java.lang.IllegalArgumentException: id=122 out of range [0, 5)
> at
> org.apache.hadoop.util.DataChecksum$Type.valueOf(DataChecksum.java:67)
> at
> org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:123)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:178)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:142)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:156)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.loadLastPartialChunkChecksum(FsVolumeImpl.java:1022)
> at
> org.apache.hadoop.hdfs.server.datanode.FinalizedReplica.getLastChecksumAndDataLen(FinalizedReplica.java:104)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:259)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:484)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:614)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:699)
> 2017-07-05 22:03:41,879 [VolumeScannerThread()] INFO datanode.VolumeScanner:
> VolumeScanner() exiting.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
