[
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109274#comment-15109274
]
Enis Soztutar commented on HBASE-15100:
---------------------------------------
Some more info from the logs. I think the first left entry was 168.log and last
is 280.log.
Seems 279 is not complete:
{code}
2016-01-20 11:31:52,741 INFO [cn017:16000.activeMasterManager]
util.FSHDFSUtils: recoverLease=true, attempt=0 on
file=hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/MasterProcWALs/state-00000000000000000279.log
after 1ms
2016-01-20 11:31:52,744 WARN [cn017:16000.activeMasterManager]
wal.WALProcedureStore: Unable to read tracker for
hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/MasterProcWALs/state-00000000000000000279.log
- Missing trailer: size=10 startPos=10
2016-01-20 11:31:52,800 INFO [cn017:16000.activeMasterManager]
wal.WALProcedureStore: Lease acquired for flushLogId: 280
2016-01-20 11:31:52,805 WARN [cn017:16000.activeMasterManager]
wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2016-01-20 11:31:52,805 INFO [cn017:16000.activeMasterManager]
wal.ProcedureWALFormatReader: No active entry found in state log
hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/MasterProcWALs/state-00000000000000000279.log.
removing it
{code}
After these all logs between 168 and 280 got deleted, but not 168 and 280.
Although the master says that it is deleting:
{code}
2016-01-20 11:31:53,757 INFO [cn017:16000.activeMasterManager]
wal.WALProcedureStore: Remove log:
hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/MasterProcWALs/state-00000000000000000169.log
2016-01-20 11:31:53,757 INFO [cn017:16000.activeMasterManager]
wal.WALProcedureStore: Removed logs:
[hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/MasterProcWALs/state-00000000000000000168.log,
hdfs://cn017.l42scl.hortonworks.com:8020/apps/hbase/data/MasterProcWALs/state-00000000000000000280.log]
{code}
The logs are still there:
{code}
[root@cn017 cluster-perf]# hadoop fs -ls /apps/hbase/data/MasterProcWALs
Found 2 items
-rw-r--r-- 3 hbase hdfs 9572 2016-01-15 20:27
/apps/hbase/data/MasterProcWALs/state-00000000000000000168.log
-rw-r--r-- 3 hbase hdfs 39 2016-01-20 11:31
/apps/hbase/data/MasterProcWALs/state-00000000000000000280.log
{code}
Is it normal that logs are deleted out of order?
For what it is worth, 168 contains ServerCrashProcedure, so maybe that is
something to look at.
> Master WALProcs still never clean up
> ------------------------------------
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
> Affects Versions: 1.2.0
> Reporter: Elliott Clark
> Assignee: Matteo Bertozzi
> Priority: Blocker
> Attachments: HBASE-15100-v0.patch, TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
