[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304885#comment-16304885
]
stack commented on HBASE-18309:
-------------------------------
[~reidchan] this change spews logging into Master log. The 'undeletable'
messages are disorientating. I'm trying to figure why it would print these
messages in first place. Am guessing that the delete fails because the dir is
not empty? Will report back... meantime, here is extract from a currently
running master log:
{code}
2017-12-26 09:34:37,088 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,088 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/big
2017-12-26 09:34:37,088 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/big
2017-12-26 09:34:37,088 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
CleanerTask 990 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/meta
and itself.
2017-12-26 09:34:37,090 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/meta
2017-12-26 09:34:37,090 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/meta
2017-12-26 09:34:37,100 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,100 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/meta
2017-12-26 09:34:37,100 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/meta
2017-12-26 09:34:37,100 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
CleanerTask 1034 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/big
and itself.
2017-12-26 09:34:37,102 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/big
2017-12-26 09:34:37,102 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/big
2017-12-26 09:34:37,107 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,108 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/meta
2017-12-26 09:34:37,108 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/meta
2017-12-26 09:34:37,108 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
CleanerTask 1046 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/big
and itself.
2017-12-26 09:34:37,109 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/big
2017-12-26 09:34:37,109 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/big
2017-12-26 09:34:37,117 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,117 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/meta
2017-12-26 09:34:37,117 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/meta
2017-12-26 09:34:37,117 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
CleanerTask 1003 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/big
and itself.
2017-12-26 09:34:37,119 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/big
2017-12-26 09:34:37,119 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/big
2017-12-26 09:34:37,126 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,126 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/big
2017-12-26 09:34:37,126 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/big
2017-12-26 09:34:37,126 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
CleanerTask 1044 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/meta
and itself.
2017-12-26 09:34:37,128 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/meta
2017-12-26 09:34:37,128 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/meta
2017-12-26 09:34:37,134 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,134 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/meta
2017-12-26 09:34:37,134 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/meta
2017-12-26 09:34:37,134 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
CleanerTask 1047 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/big
and itself.
2017-12-26 09:34:37,136 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/big
2017-12-26 09:34:37,136 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/big
2017-12-26 09:34:37,141 DEBUG [ForkJoinPool-1-worker-22] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/0ec3a8f9348565cfc1a1f4bc49d214d7/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,141 DEBUG [ForkJoinPool-1-worker-22] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/0ec3a8f9348565cfc1a1f4bc49d214d7/big
2017-12-26 09:34:37,141 DEBUG [ForkJoinPool-1-worker-22] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/0ec3a8f9348565cfc1a1f4bc49d214d7/big
2017-12-26 09:34:37,141 DEBUG [ForkJoinPool-1-worker-22] cleaner.CleanerChore:
CleanerTask 1035 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/0ec3a8f9348565cfc1a1f4bc49d214d7/meta
and itself.
2017-12-26 09:34:37,143 DEBUG [ForkJoinPool-1-worker-22] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/0ec3a8f9348565cfc1a1f4bc49d214d7/meta
2017-12-26 09:34:37,143 DEBUG [ForkJoinPool-1-worker-22] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/0ec3a8f9348565cfc1a1f4bc49d214d7/meta
2017-12-26 09:34:37,148 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,148 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/big
2017-12-26 09:34:37,148 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/big
2017-12-26 09:34:37,148 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
CleanerTask 1047 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/tiny
and itself.
2017-12-26 09:34:37,149 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/tiny
2017-12-26 09:34:37,149 DEBUG [ForkJoinPool-1-worker-29] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/faa195bbce663e4cc517909d66dac075/tiny
2017-12-26 09:34:37,155 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,155 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/meta
2017-12-26 09:34:37,155 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe/meta
2017-12-26 09:34:37,155 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Couldn't delete subdirs completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/2b9ae2c41380afa5abfcf4d6a2b26bbe
with reasons: undeletable, please check.
2017-12-26 09:34:37,155 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
CleanerTask 1044 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/338b1a409120ee6f7ba4238bea0dff6d/tiny
and itself.
2017-12-26 09:34:37,156 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/338b1a409120ee6f7ba4238bea0dff6d/tiny
2017-12-26 09:34:37,156 DEBUG [ForkJoinPool-1-worker-25] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/338b1a409120ee6f7ba4238bea0dff6d/tiny
2017-12-26 09:34:37,161 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,161 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/big
2017-12-26 09:34:37,161 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8/big
2017-12-26 09:34:37,161 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Couldn't delete subdirs completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/d70f1cd5286beb95db53159412c87bd8
with reasons: undeletable, please check.
2017-12-26 09:34:37,161 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
CleanerTask 1003 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/c2178b96d7bea143b7ff5338892ad04d/tiny
and itself.
2017-12-26 09:34:37,162 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/c2178b96d7bea143b7ff5338892ad04d/tiny
2017-12-26 09:34:37,162 DEBUG [ForkJoinPool-1-worker-7] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/c2178b96d7bea143b7ff5338892ad04d/tiny
2017-12-26 09:34:37,168 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,168 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/big
2017-12-26 09:34:37,168 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/big
2017-12-26 09:34:37,168 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
CleanerTask 1046 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/tiny
and itself.
2017-12-26 09:34:37,169 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/tiny
2017-12-26 09:34:37,169 DEBUG [ForkJoinPool-1-worker-15] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/43d06017c951a75355b55236fa8ff6e0/tiny
2017-12-26 09:34:37,179 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,179 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/big
2017-12-26 09:34:37,179 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/big
2017-12-26 09:34:37,179 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
CleanerTask 1034 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/tiny
and itself.
2017-12-26 09:34:37,180 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/tiny
2017-12-26 09:34:37,180 DEBUG [ForkJoinPool-1-worker-10] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/4af779a7b1991d201f62a2795d1bd3bf/tiny
2017-12-26 09:34:37,187 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,187 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/meta
2017-12-26 09:34:37,187 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/meta
2017-12-26 09:34:37,187 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
CleanerTask 990 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/tiny
and itself.
2017-12-26 09:34:37,188 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/tiny
2017-12-26 09:34:37,188 DEBUG [ForkJoinPool-1-worker-30] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/b595581be1aeea5a46032079dfcd9b25/tiny
2017-12-26 09:34:37,192 DEBUG [ForkJoinPool-1-worker-24] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/09b21fd4db87ff79bfc036b91c2eabde/meta
with reasons: undeletable, please check.
2017-12-26 09:34:37,192 DEBUG [ForkJoinPool-1-worker-24] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/09b21fd4db87ff79bfc036b91c2eabde/meta
2017-12-26 09:34:37,192 DEBUG [ForkJoinPool-1-worker-24] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/09b21fd4db87ff79bfc036b91c2eabde/meta
2017-12-26 09:34:37,192 DEBUG [ForkJoinPool-1-worker-24] cleaner.CleanerChore:
CleanerTask 1032 starts cleaning dirs and files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/09b21fd4db87ff79bfc036b91c2eabde/tiny
and itself.
2017-12-26 09:34:37,193 DEBUG [ForkJoinPool-1-worker-24] cleaner.CleanerChore:
There is no subdir under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/09b21fd4db87ff79bfc036b91c2eabde/tiny
2017-12-26 09:34:37,193 DEBUG [ForkJoinPool-1-worker-24] cleaner.CleanerChore:
Start deleting files under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/09b21fd4db87ff79bfc036b91c2eabde/tiny
2017-12-26 09:34:37,200 DEBUG [ForkJoinPool-1-worker-2] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/c2178b96d7bea143b7ff5338892ad04d/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,201 DEBUG [ForkJoinPool-1-worker-2] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/c2178b96d7bea143b7ff5338892ad04d/big
2017-12-26 09:34:37,201 DEBUG [ForkJoinPool-1-worker-2] cleaner.CleanerChore:
Finish deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/c2178b96d7bea143b7ff5338892ad04d/big
2017-12-26 09:34:37,212 DEBUG [ForkJoinPool-1-worker-13] cleaner.CleanerChore:
Couldn't delete files completely under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/683497e638c139cb98d00219d7690f1a/big
with reasons: undeletable, please check.
2017-12-26 09:34:37,212 DEBUG [ForkJoinPool-1-worker-13] cleaner.CleanerChore:
Start deleting subdirs under
hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/683497e638c139cb98d00219d7690f1a/big
....
{code}
... and this is only a snippet.
Thanks.
> Support multi threads in CleanerChore
> -------------------------------------
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Reporter: binlijin
> Assignee: Reid Chan
> Fix For: 3.0.0, 2.0.0-beta-1
>
> Attachments: HBASE-18309.addendum.patch,
> HBASE-18309.master.001.patch, HBASE-18309.master.002.patch,
> HBASE-18309.master.004.patch, HBASE-18309.master.005.patch,
> HBASE-18309.master.006.patch, HBASE-18309.master.007.patch,
> HBASE-18309.master.008.patch, HBASE-18309.master.009.patch,
> HBASE-18309.master.010.patch, HBASE-18309.master.011.patch,
> HBASE-18309.master.012.patch, space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
> What's more, currently there's only one thread iterating the archive
> directory, and we could use multiple threads cleaning sub directories in
> parallel to speed it up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
