[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2019-03-04 Thread Guanghao Zhang (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20770:
---
Fix Version/s: 2.2.0
   3.0.0

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 3.0.0, 1.3.3, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
>

[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2019-02-01 Thread Andrew Purtell (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-20770:
---
Fix Version/s: (was: 1.5.0)

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/1a46700fbc434574a005c0b55879d5ed
> 2018-06-21

[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2018-12-12 Thread Andrew Purtell (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-20770:
---
Fix Version/s: 1.3.3

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 1.5.0, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/1a46700fbc434574a005c0b55879d5ed
> 2018-06-21 01:19:13,011

[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2018-06-26 Thread Andrew Purtell (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-20770:
---
Fix Version/s: 1.4.6
   1.5.0

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/1a46700fbc434574a005c0b55879d5ed
>

[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2018-06-25 Thread stack (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20770:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to branch-2.0+. Thanks for review [~reidchan]

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
>

[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2018-06-24 Thread stack (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20770:
--
Assignee: stack
  Status: Patch Available  (was: Open)

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/1a46700fbc434574a005c0b55879d5ed
> 2018-06-21 01:19:13,011

[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do

2018-06-24 Thread stack (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20770:
--
Attachment: HBASE-20770.branch-2.0.001.patch

> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
>  Issue Type: Bug
>  Components: logging
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster, 
> Master log is in a continuous spew of logging output fililng disks. It is 
> stuck making no progress but hard to tell because it is logging minutiae 
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
>  exit...
> 2018-06-21 01:19:12,823 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms 
> for deleting 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
>  exit...
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG 
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing 
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/1a46700fbc434574a005c0b55879d5ed
> 2018-06-21 01:19:13,011 DEBUG 
>