[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541785#comment-16541785 ] Sean Busbey commented on HBASE-20877: - please don't rm WALs without understanding why they're present. The most likely reason for them still being around is that replication has not yet managed to successfully send them to your indexer. Please follow the steps from my email to user@hbase: https://lists.apache.org/thread.html/221e18a4a861ff6736cb17036ce17f410027046fd7f00fb80bfd11f1@%3Cuser.hbase.apache.org%3E > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[
https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541236#comment-16541236
]
Reid Chan commented on HBASE-20877:
---
actual data are stored in {{/hbase/data}} directory.
WALs are used for replay when rs is crashed, and oldWALs are no longer used for
replay that's why they are being cleaned. The problem why {{oldWALs}} grows so
huge it's cleaning up speed much slower than producing speed, and this's the
background for HBASE-18309.
> Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
>
>
> Key: HBASE-20877
> URL: https://issues.apache.org/jira/browse/HBASE-20877
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.2.0
>Reporter: Manjeet Singh
>Priority: Major
>
> Hbase version 1.2.0 OldWals are getting filled and showing as below
> 7.2 K 21.5 K /hbase/.hbase-snapshot
> 0 0 /hbase/.tmp
> 0 0 /hbase/MasterProcWALs
> 18.3 G 60.2 G /hbase/WALs
> 28.7 G 86.1 G /hbase/archive
> 0 0 /hbase/corrupt
> 1.7 T 5.2 T /hbase/data
> 42 126 /hbase/hbase.id
> 7 21 /hbase/hbase.version
> 7.2 T 21.6 T /hbase/oldWALs
>
> It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in
> master background chore, HBASE-20352(for 1.x version) is created to speed up
> cleaning oldWals, in our case it's not happening.
> hbase.master.logcleaner.ttl is 1 minutes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[
https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541233#comment-16541233
]
Reid Chan commented on HBASE-20877:
---
I'm not familiar with solr, but it makes much more sense to use current wal
under {{/hbase/WALs}} other than being cleaned wals under {{/hbase/oldWALs}}.
At least, you can start {{hdfs rm}} from the oldest subdir under
{{/hbase/oldWALs}} to if you have some worries.
Pardon me if i'm wrong.
Or try this way
{quote}apply that patch manually
{quote}
> Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
>
>
> Key: HBASE-20877
> URL: https://issues.apache.org/jira/browse/HBASE-20877
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.2.0
>Reporter: Manjeet Singh
>Priority: Major
>
> Hbase version 1.2.0 OldWals are getting filled and showing as below
> 7.2 K 21.5 K /hbase/.hbase-snapshot
> 0 0 /hbase/.tmp
> 0 0 /hbase/MasterProcWALs
> 18.3 G 60.2 G /hbase/WALs
> 28.7 G 86.1 G /hbase/archive
> 0 0 /hbase/corrupt
> 1.7 T 5.2 T /hbase/data
> 42 126 /hbase/hbase.id
> 7 21 /hbase/hbase.version
> 7.2 T 21.6 T /hbase/oldWALs
>
> It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in
> master background chore, HBASE-20352(for 1.x version) is created to speed up
> cleaning oldWals, in our case it's not happening.
> hbase.master.logcleaner.ttl is 1 minutes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541228#comment-16541228 ] Manjeet Singh commented on HBASE-20877: --- If I remove OldWal, it's not impacting on my actual data? and not even on Hbase solr indexer? Please note it's a production system. > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541224#comment-16541224 ] Reid Chan commented on HBASE-20877: --- Or you can clean oldWALs directly through hdfs -rm command. > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541221#comment-16541221 ] Reid Chan commented on HBASE-20877: --- hbase-1.2.0 doesn't have HBASE-20352 this patch. You may need to upgrade to higher version or apply that patch manually. > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
