[
https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Kellerman updated HADOOP-1820:
----------------------------------
Status: In Progress (was: Patch Available)
Almost from the very beginning TestHBaseCluster had problems with the mini dfs
cluster:
[junit] 07/09/23 21:09:13 WARN fs.FSNamesystem: Not able to place enough
replicas, still in need of 1
This continued until the test timed out. Basically no forward progress could be
made. The changes in this patch do not effect this test, so there must be some
kind of race going on between the dfs mini cluster's servers and HBase's.
The next patch will back off some of the more aggressive test settings for this
test to see if the real problem is timing related. Since the tests that failed
previously succeeded this time, apparently, for some tests, the aggressive
parameters used for testing are just a bit too aggressive for a machine which
is "resource challenged" as hudson appears to be.
> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
> Key: HADOOP-1820
> URL: https://issues.apache.org/jira/browse/HADOOP-1820
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.15.0
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.15.0
>
> Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt,
> patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying. Its
> used recoverying state if a regionserver crashes: edits that have not been
> persisted to an HStoreFile are rerun to populate memcache which in turn is
> converted to an on-filesytem HStoreFile. On a period, the log is rotated and
> a new one is opened. While the region server is up, the logs grow in number
> without bound. Only the most recent contain unpersisted edits. If the
> region server goes down clean, then its logs are cleaned up. If a region
> server crashes, as part of recovery, the logs of edits are sorted and split
> per region. Recovery would run faster if it did not have to plough through
> reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer. As well as
> rolling logs, it could check for edit logs to clean. When rolled, logs could
> be marked with the sequence id of their last contained edit. The thread
> could on a period ask each hosted region for the "lowest highest" sequence id
> of all regions deployed. Once this number had crossed out that on a
> particular log, the log could be cleaned up safely.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
