[
https://issues.apache.org/jira/browse/HDFS-8809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HDFS-8809:
-----------------------------
Description:
Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files
in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit
idle for a couple hours, it is still in the corrupt state. If HBase is shut
down, the problem goes away. If HBase is then restarted, the problem recurs.
This was observed with Hadoop 2.7.1 and HBase 1.1.1, and did NOT happen with
Hadoop 2.4 and HBase 0.98.
{code}
hades1:/var/opt/teradata/packages # su hdfs
hdfs@hades1:/var/opt/teradata/packages> hdfs fsck /
Connecting to namenode via
http://hades1.labs.teradata.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24
20:40:17 GMT 2015
...
/apps/hbase/data/WALs/hades4.labs.teradata.com,16020,1435168292684/hades4.labs.teradata.com%2C16020%2C1435168292684.default.1435175500556:
MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466..meta.1435175562144.meta:
MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466.default.1435175498500:
MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades6.labs.teradata.com,16020,1435168292373/hades6.labs.teradata.com%2C16020%2C1435168292373.default.1435175500301:
MISSING 1 blocks of total size 83
B..................................................................................................
....................................................................................................
....................................................................................................
........................................................................................Status:
CORRUPT
Total size: 723977553 B (Total open files size: 332 B)
Total dirs: 79
Total files: 388
Total symlinks: 0 (Files currently being written: 5)
Total blocks (validated): 387 (avg. block size 1870743 B) (Total open
file blocks (not validated): 4)
********************************
UNDER MIN REPL'D BLOCKS: 4 (1.0335917 %)
dfs.namenode.replication.min: 1
CORRUPT FILES: 4
MISSING BLOCKS: 4
MISSING SIZE: 332 B
********************************
Minimally replicated blocks: 387 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds
The filesystem under path '/' is CORRUPT
hdfs@hades1:/var/opt/teradata/packages>
{code}
was:
Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files
in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit
idle for a couple hours, it is still in the corrupt state. If HBase is shut
down, the problem goes away. If HBase is then restarted, the problem recurs.
{code}
hades1:/var/opt/teradata/packages # su hdfs
hdfs@hades1:/var/opt/teradata/packages> hdfs fsck /
Connecting to namenode via
http://hades1.labs.teradata.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24
20:40:17 GMT 2015
...
/apps/hbase/data/WALs/hades4.labs.teradata.com,16020,1435168292684/hades4.labs.teradata.com%2C16020%2C1435168292684.default.1435175500556:
MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466..meta.1435175562144.meta:
MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466.default.1435175498500:
MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades6.labs.teradata.com,16020,1435168292373/hades6.labs.teradata.com%2C16020%2C1435168292373.default.1435175500301:
MISSING 1 blocks of total size 83
B..................................................................................................
....................................................................................................
....................................................................................................
........................................................................................Status:
CORRUPT
Total size: 723977553 B (Total open files size: 332 B)
Total dirs: 79
Total files: 388
Total symlinks: 0 (Files currently being written: 5)
Total blocks (validated): 387 (avg. block size 1870743 B) (Total open
file blocks (not validated): 4)
********************************
UNDER MIN REPL'D BLOCKS: 4 (1.0335917 %)
dfs.namenode.replication.min: 1
CORRUPT FILES: 4
MISSING BLOCKS: 4
MISSING SIZE: 332 B
********************************
Minimally replicated blocks: 387 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds
The filesystem under path '/' is CORRUPT
hdfs@hades1:/var/opt/teradata/packages>
{code}
> HDFS fsck reports HBase WALs files (under construction) as "CORRUPT" (missing
> blocks) when HBase is running
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-8809
> URL: https://issues.apache.org/jira/browse/HDFS-8809
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Affects Versions: 2.7.0
> Environment: Hadoop 2.7.1 and HBase 1.1.1, on SUSE11sp3 (other
> Linuxes not tested, probably not platform-dependent). This did NOT happen
> with Hadoop 2.4 and HBase 0.98.
> Reporter: Sudhir Prakash
>
> Whenever HBase is running, the "hdfs fsck /" reports four hbase-related
> files in the path "hbase/data/WALs/" as CORRUPT. Even after letting the
> cluster sit idle for a couple hours, it is still in the corrupt state. If
> HBase is shut down, the problem goes away. If HBase is then restarted, the
> problem recurs. This was observed with Hadoop 2.7.1 and HBase 1.1.1, and did
> NOT happen with Hadoop 2.4 and HBase 0.98.
> {code}
> hades1:/var/opt/teradata/packages # su hdfs
> hdfs@hades1:/var/opt/teradata/packages> hdfs fsck /
> Connecting to namenode via
> http://hades1.labs.teradata.com:50070/fsck?ugi=hdfs&path=%2F
> FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24
> 20:40:17 GMT 2015
> ...
> /apps/hbase/data/WALs/hades4.labs.teradata.com,16020,1435168292684/hades4.labs.teradata.com%2C16020%2C1435168292684.default.1435175500556:
> MISSING 1 blocks of total size 83 B.
> /apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466..meta.1435175562144.meta:
> MISSING 1 blocks of total size 83 B.
> /apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466.default.1435175498500:
> MISSING 1 blocks of total size 83 B.
> /apps/hbase/data/WALs/hades6.labs.teradata.com,16020,1435168292373/hades6.labs.teradata.com%2C16020%2C1435168292373.default.1435175500301:
> MISSING 1 blocks of total size 83
> B..................................................................................................
> ....................................................................................................
> ....................................................................................................
> ........................................................................................Status:
> CORRUPT
> Total size: 723977553 B (Total open files size: 332 B)
> Total dirs: 79
> Total files: 388
> Total symlinks: 0 (Files currently being written: 5)
> Total blocks (validated): 387 (avg. block size 1870743 B) (Total open
> file blocks (not validated): 4)
> ********************************
> UNDER MIN REPL'D BLOCKS: 4 (1.0335917 %)
> dfs.namenode.replication.min: 1
> CORRUPT FILES: 4
> MISSING BLOCKS: 4
> MISSING SIZE: 332 B
> ********************************
> Minimally replicated blocks: 387 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 3.0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> Number of data-nodes: 3
> Number of racks: 1
> FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds
> The filesystem under path '/' is CORRUPT
> hdfs@hades1:/var/opt/teradata/packages>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
