[jira] [Updated] (HBASE-5872) Improve hadoopqa script to include checks for hadoop 0.23 build
[ https://issues.apache.org/jira/browse/HBASE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5872: -- Attachment: hbase-5872-part2.patch An addendum that makes the success/failure message explicit and more informative. > Improve hadoopqa script to include checks for hadoop 0.23 build > --- > > Key: HBASE-5872 > URL: https://issues.apache.org/jira/browse/HBASE-5872 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 0.96.0 > > Attachments: hbase-5872-part2.patch, hbase-5872.patch > > > There have been a few patches that have made it into hbase trunk that have > broken the compile of hbase against hadoop 0.23.x, without being known for a > few days. > We could have the bot do a few things: > 1) verify that patch compiles against hadoop 23 > 2) verify that unit tests pass against hadoop 23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5887: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524790/hbase-5887-92.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1662//console This message is automatically generated.) > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0, 0.92.1, 0.94.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-5887-92.patch, hbase-5887.patch > > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5672) TestLruBlockCache#testBackgroundEvictionThread fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263374#comment-13263374
]
Hudson commented on HBASE-5672:
---
Integrated in HBase-TRUNK-security #186 (See
[https://builds.apache.org/job/HBase-TRUNK-security/186/])
HBASE-5672 TestLruBlockCache#testBackgroundEvictionThread fails
occasionally (Revision 1330971)
Result = SUCCESS
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
> TestLruBlockCache#testBackgroundEvictionThread fails occasionally
> -
>
> Key: HBASE-5672
> URL: https://issues.apache.org/jira/browse/HBASE-5672
> Project: HBase
> Issue Type: Bug
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: HBASE-5672.patch, HBASE-5672v2.patch, HBASE-5672v3.patch
>
>
> We find TestLruBlockCache#testBackgroundEvictionThread fails occasionally.
> I think it's a problem of the test case.
> Because runEviction() only do evictionThread.evict():
> {code}
> public void evict() {
> synchronized(this) {
> this.notify(); // FindBugs NN_NAKED_NOTIFY
> }
> }
> {code}
> However when we call evictionThread.evict(), the evictionThread may haven't
> been in run() in the TestLruBlockCache#testBackgroundEvictionThread.
> If we run the test many times, we could find failture easily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5829) Inconsistency between the "regions" map and the "servers" map in AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263372#comment-13263372
]
Hudson commented on HBASE-5829:
---
Integrated in HBase-TRUNK-security #186 (See
[https://builds.apache.org/job/HBase-TRUNK-security/186/])
HBASE-5829 Inconsistency between the "regions" map and the "servers" map in
AssignmentManager (Revision 1330993)
Result = SUCCESS
stack :
Files :
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Inconsistency between the "regions" map and the "servers" map in
> AssignmentManager
> --
>
> Key: HBASE-5829
> URL: https://issues.apache.org/jira/browse/HBASE-5829
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Fix For: 0.96.0
>
> Attachments: HBASE-5829-0.90.patch, HBASE-5829-trunk.patch
>
>
> There are occurrences in AM where this.servers is not kept consistent with
> this.regions. This might cause balancer to offline a region from the RS that
> already returned NotServingRegionException at a previous offline attempt.
> In AssignmentManager.unassign(HRegionInfo, boolean)
> try {
> // TODO: We should consider making this look more like it does for the
> // region open where we catch all throwables and never abort
> if (serverManager.sendRegionClose(server, state.getRegion(),
> versionOfClosingNode)) {
> LOG.debug("Sent CLOSE to " + server + " for region " +
> region.getRegionNameAsString());
> return;
> }
> // This never happens. Currently regionserver close always return true.
> LOG.warn("Server " + server + " region CLOSE RPC returned false for " +
> region.getRegionNameAsString());
> } catch (NotServingRegionException nsre) {
> LOG.info("Server " + server + " returned " + nsre + " for " +
> region.getRegionNameAsString());
> // Presume that master has stale data. Presume remote side just split.
> // Presume that the split message when it comes in will fix up the
> master's
> // in memory cluster state.
> } catch (Throwable t) {
> if (t instanceof RemoteException) {
> t = ((RemoteException)t).unwrapRemoteException();
> if (t instanceof NotServingRegionException) {
> if (checkIfRegionBelongsToDisabling(region)) {
> // Remove from the regionsinTransition map
> LOG.info("While trying to recover the table "
> + region.getTableNameAsString()
> + " to DISABLED state the region " + region
> + " was offlined but the table was in DISABLING state");
> synchronized (this.regionsInTransition) {
> this.regionsInTransition.remove(region.getEncodedName());
> }
> // Remove from the regionsMap
> synchronized (this.regions) {
> this.regions.remove(region);
> }
> deleteClosingOrClosedNode(region);
> }
> }
> // RS is already processing this region, only need to update the
> timestamp
> if (t instanceof RegionAlreadyInTransitionException) {
> LOG.debug("update " + state + " the timestamp.");
> state.update(state.getState());
> }
> }
> In AssignmentManager.assign(HRegionInfo, RegionState, boolean, boolean,
> boolean)
> synchronized (this.regions) {
> this.regions.put(plan.getRegionInfo(), plan.getDestination());
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263373#comment-13263373
]
Hudson commented on HBASE-5864:
---
Integrated in HBase-TRUNK-security #186 (See
[https://builds.apache.org/job/HBase-TRUNK-security/186/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331058)
Result = SUCCESS
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
> Error while reading from hfile in 0.94
> --
>
> Key: HBASE-5864
> URL: https://issues.apache.org/jira/browse/HBASE-5864
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
> HBASE-5864_3.patch, HBASE-5864_test.patch
>
>
> Got the following stacktrace during region split.
> {noformat}
> 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
> Failed getting store size for value
> java.io.IOException: Requested block is out of range: 2906737606134037404,
> lastDataBlockOffset: 84764558
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263370#comment-13263370 ] Hudson commented on HBASE-5862: --- Integrated in HBase-TRUNK-security #186 (See [https://builds.apache.org/job/HBase-TRUNK-security/186/]) HBASE-5862 After Region Close remove the Operation Metrics (Revision 1330997) Result = SUCCESS stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/OperationMetrics.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionMetricsStorage.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash
[ https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263369#comment-13263369 ] Hudson commented on HBASE-5844: --- Integrated in HBase-TRUNK-security #186 (See [https://builds.apache.org/job/HBase-TRUNK-security/186/]) HBASE-5844 Delete the region servers znode after a regions server crash; REVERT (Revision 1330983) Result = SUCCESS stack : Files : * /hbase/trunk/bin/hbase-daemon.sh * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Delete the region servers znode after a regions server crash > > > Key: HBASE-5844 > URL: https://issues.apache.org/jira/browse/HBASE-5844 > Project: HBase > Issue Type: Improvement > Components: regionserver, scripts >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Fix For: 0.96.0 > > Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch > > > today, if the regions server crashes, its znode is not deleted in ZooKeeper. > So the recovery process will stop only after a timeout, usually 30s. > By deleting the znode in start script, we remove this delay and the recovery > starts immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5872) Improve hadoopqa script to include checks for hadoop 0.23 build
[ https://issues.apache.org/jira/browse/HBASE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263371#comment-13263371 ] Hudson commented on HBASE-5872: --- Integrated in HBase-TRUNK-security #186 (See [https://builds.apache.org/job/HBase-TRUNK-security/186/]) HBASE-5872 Improve hadoopqa script to include checks for hadoop 0.23 build (Revision 1331143) Result = SUCCESS jmhsieh : Files : * /hbase/trunk/dev-support/test-patch.sh > Improve hadoopqa script to include checks for hadoop 0.23 build > --- > > Key: HBASE-5872 > URL: https://issues.apache.org/jira/browse/HBASE-5872 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 0.96.0 > > Attachments: hbase-5872.patch > > > There have been a few patches that have made it into hbase trunk that have > broken the compile of hbase against hadoop 0.23.x, without being known for a > few days. > We could have the bot do a few things: > 1) verify that patch compiles against hadoop 23 > 2) verify that unit tests pass against hadoop 23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263361#comment-13263361
]
Jieshan Bean commented on HBASE-5878:
-
bq.We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
+1 on this idea.
> Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
> ---
>
> Key: HBASE-5878
> URL: https://issues.apache.org/jira/browse/HBASE-5878
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> SequencFileLogReader:
> Currently Hbase using getFileLength api from DFSInputStream class by
> reflection. DFSInputStream is not exposed as public. So, this may change in
> future. Now HDFS exposed HdfsDataInputStream as public API.
> We can make use of it, when we are not able to find the getFileLength api
> from DFSInputStream as a else condition. So, that we will not have any sudden
> surprise like we are facing today.
> Also, it is just logging one warn message and proceeding if it throws any
> exception while getting the length. I think we can re-throw the exception
> because there is no point in continuing with dataloss.
> {code}
> long adjust = 0;
> try {
> Field fIn = FilterInputStream.class.getDeclaredField("in");
> fIn.setAccessible(true);
> Object realIn = fIn.get(this.in);
> // In hadoop 0.22, DFSInputStream is a standalone class. Before
> this,
> // it was an inner class of DFSClient.
> if (realIn.getClass().getName().endsWith("DFSInputStream")) {
> Method getFileLength = realIn.getClass().
> getDeclaredMethod("getFileLength", new Class []{});
> getFileLength.setAccessible(true);
> long realLength = ((Long)getFileLength.
> invoke(realIn, new Object []{})).longValue();
> assert(realLength >= this.length);
> adjust = realLength - this.length;
> } else {
> LOG.info("Input stream class: " + realIn.getClass().getName() +
> ", not adjusting length");
> }
> } catch(Exception e) {
> SequenceFileLogReader.LOG.warn(
> "Error while trying to get accurate file length. " +
> "Truncation / data loss may occur if RegionServers die.", e);
> }
> return adjust + super.getPos();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5863) Improve the graceful_stop.sh CLI help (especially about reloads)
[
https://issues.apache.org/jira/browse/HBASE-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5863:
-
Fix Version/s: (was: 0.94.1)
0.94.0
> Improve the graceful_stop.sh CLI help (especially about reloads)
>
>
> Key: HBASE-5863
> URL: https://issues.apache.org/jira/browse/HBASE-5863
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 0.94.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
> Fix For: 0.92.2, 0.94.0
>
> Attachments: HBASE-5863.patch
>
>
> Right now, graceful_stop.sh prints:
> {code}
> Usage: graceful_stop.sh [--config ] [--restart] [--reload]
> [--thrift] [--rest]
> thrift If we should stop/start thrift before/after the hbase stop/start
> restIf we should stop/start rest before/after the hbase stop/start
> restart If we should restart after graceful stop
> reload Move offloaded regions back on to the stopped server
> debug Move offloaded regions back on to the stopped server
> hostnameHostname of server we are to stop
> {code}
> This does not help us specify that "reload" is actually a sub/additive-option
> to "restart".
> Also, the "debug" line seems to still have an old copy/paste mistake.
> I've updated these two in the patch here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263347#comment-13263347
]
Hadoop QA commented on HBASE-5611:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524810/HBASE-5611-trunk-v2-minorchange.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1666//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1666//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1666//console
This message is automatically generated.
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5865) test-util.sh broken with unittest updates
[ https://issues.apache.org/jira/browse/HBASE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5865: - Fix Version/s: (was: 0.94.1) 0.94.0 > test-util.sh broken with unittest updates > - > > Key: HBASE-5865 > URL: https://issues.apache.org/jira/browse/HBASE-5865 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.96.0, 0.94.1 >Reporter: Jesse Yates >Assignee: Jesse Yates >Priority: Minor > Fix For: 0.94.0, 0.96.0 > > Attachments: sh_HBASE-5865-v0.patch > > > Since the default maven test is meant to be run on the server, this test > script always fails. Needs to take into account the location of where the > script is being run as well as some debugging options for future fixes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5836) Backport per region metrics from HBASE-3614 to 0.94.1
[ https://issues.apache.org/jira/browse/HBASE-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5836: - Fix Version/s: (was: 0.94.1) 0.94.0 > Backport per region metrics from HBASE-3614 to 0.94.1 > - > > Key: HBASE-5836 > URL: https://issues.apache.org/jira/browse/HBASE-5836 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.1 >Reporter: stack >Assignee: Elliott Clark > Fix For: 0.94.0 > > Attachments: HBASE-5836-0.patch > > > This would be good to have in 0.94. Can go into 0.94.1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263336#comment-13263336
]
Hadoop QA commented on HBASE-5611:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524812/HBASE-5611-94-minorchange.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1667//console
This message is automatically generated.
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263337#comment-13263337
]
Jieshan Bean commented on HBASE-5611:
-
This problem seems not present in 90 branch. RegionServerAccounting is not in
90. And the global memstore size is calculated by below methods:
{noformat}
public long getGlobalMemStoreSize() {
long total = 0;
for (HRegion region : onlineRegions.values()) {
total += region.memstoreSize.get();
}
return total;
}
{noformat}
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.
[ https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263335#comment-13263335 ] Zhihong Yu commented on HBASE-5874: --- +1 on patch. > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > > > Key: HBASE-5874 > URL: https://issues.apache.org/jira/browse/HBASE-5874 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch > > > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to > the code. > hbck exception: > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:128) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596) > at > org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332) > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907) > > Merge exception: > [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge > 381] exiting due to error > java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634) > at > org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276) > at > org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261) > at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: (was: HBASE-5611-trunk.patch)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: (was: HBASE-5611-trunk-v2.patch)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.
[ https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263334#comment-13263334 ] Hadoop QA commented on HBASE-5874: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524806/HBASE-5874-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1665//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1665//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1665//console This message is automatically generated. > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > > > Key: HBASE-5874 > URL: https://issues.apache.org/jira/browse/HBASE-5874 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch > > > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to > the code. > hbck exception: > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:128) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596) > at > org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332) > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907) > > Merge exception: > [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge > 381] exiting due to error > java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634) > at > org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276) > at > org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261) > at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115) > at org.
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: (was: HBASE-5611-94.patch)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: HBASE-5611-94-minorchange.patch
HBASE-5611-92.patch
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch,
> HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1326#comment-1326
]
Hadoop QA commented on HBASE-5611:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524809/HBASE-5611-trunk-v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.util.TestHBaseFsck
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1664//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1664//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1664//console
This message is automatically generated.
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch,
> HBASE-5611-trunk-v2-minorchange.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5881) BuiltIn Gzip compressor & decompressor not getting pooled, leading to native memory leak
[
https://issues.apache.org/jira/browse/HBASE-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263332#comment-13263332
]
Gopinathan A commented on HBASE-5881:
-
OK, I feel.
> BuiltIn Gzip compressor & decompressor not getting pooled, leading to native
> memory leak
>
>
> Key: HBASE-5881
> URL: https://issues.apache.org/jira/browse/HBASE-5881
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 0.92.1
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
>
> This issue will occur only in hadoop 23.x & above/
> In hadoop 0.20.x
> {code}
> public static void returnDecompressor(Decompressor decompressor) {
> if (decompressor == null) {
> return;
> }
> decompressor.reset();
> payback(decompressorPool, decompressor);
> }
> {code}
> In hadoop 0.23.x
> {code}
> public static void returnDecompressor(Decompressor decompressor) {
> if (decompressor == null) {
> return;
> }
> // if the decompressor can't be reused, don't pool it.
> if (decompressor.getClass().isAnnotationPresent(DoNotPool.class)) {
> return;
> }
> decompressor.reset();
> payback(decompressorPool, decompressor);
> }
> {code}
> Here annotation has been added. By default this library will be loaded if
> there are no native library.
> {code}
> @DoNotPool
> public class BuiltInGzipDecompressor
> {code}
> Due to this each time new compressor/decompressor will be loaded, this leads
> to native memory leak.
> {noformat}
> 2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor [.gz]
> 2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor [.gz]
> 2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor [.gz]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: HBASE-5611-trunk-v2-minorchange.patch
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch,
> HBASE-5611-trunk-v2-minorchange.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263327#comment-13263327
]
Zhihong Yu commented on HBASE-5611:
---
{code}
+ * Roll back the global MemStore size when a region can't open.
{code}
The above is not accurate: we're only rolling back the replay edits size for
specified region from global MemStore size.
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263326#comment-13263326
]
Jieshan Bean commented on HBASE-5611:
-
Will upload the patches for 92 and 90 today:)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: HBASE-5611-trunk-v2.patch
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: (was: HBASE-5611-trunk-v2.patch)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5611:
--
Comment: was deleted
(was: -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524808/HBASE-5611-94.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1663//console
This message is automatically generated.)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263324#comment-13263324
]
Hadoop QA commented on HBASE-5611:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524808/HBASE-5611-94.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1663//console
This message is automatically generated.
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.
[ https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5874: -- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > > > Key: HBASE-5874 > URL: https://issues.apache.org/jira/browse/HBASE-5874 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch > > > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to > the code. > hbck exception: > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:128) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596) > at > org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332) > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907) > > Merge exception: > [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge > 381] exiting due to error > java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634) > at > org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276) > at > org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261) > at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: HBASE-5611-94.patch
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-94.patch, HBASE-5611-trunk-v2.patch,
> HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-5611:
Attachment: (was: HBASE-5611-94.patch)
> Replayed edits from regions that failed to open during recovery aren't
> removed from the global MemStore size
>
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5611-trunk-v2.patch, HBASE-5611-trunk.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
> it's still possible to hit it if a region fails to open for more obscure
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now
> being opened by a new RS. The first thing it does is to read the recovery
> files and put the edits in the {{MemStores}}. If this process takes a long
> time, the master will move that region away. At that point the edits are
> still accounted for in the global {{MemStore}} size but they are dropped when
> the {{HRegion}} gets cleaned up. It's completely invisible until the
> {{MemStoreFlusher}} needs to force flush a region and that none of them have
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
> for entry null
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the
> {{MemStore}} during recovery that I'm over the low barrier although in fact
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when
> the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5888) Clover profile in build
[
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-5888:
-
Attachment: hbase-clover_v1.patch
Patch against trunk. I'll provide backwards patches once we are settled.
Replicating the patch comment:
Profile for running clover. You need to have a clover license under
~/.clover.license for ${clover.version}
or you can provide the license with
-Dmaven.clover.licenseLocation=/path/to/license. Committers can find
the license under
https://svn.apache.org/repos/private/committers/donated-licenses/clover/
Note that clover 2.6.3 does not run with maven 3, so you have to use
maven2. The report will be generated
under target/site/clover/index.html when you run
MAVEN_OPTS=-Xmx2048m mvn clean test -Pclover site
> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
> Issue Type: Improvement
> Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-clover_v1.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables
> clover reports. We can also backport this to 0.92, and 0.94, since we are
> also interested in test coverage for those branches.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5888) Clover profile in build
Enis Soztutar created HBASE-5888: Summary: Clover profile in build Key: HBASE-5888 URL: https://issues.apache.org/jira/browse/HBASE-5888 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Enis Soztutar Assignee: Enis Soztutar Clover is disabled right now. I would like to add a profile that enables clover reports. We can also backport this to 0.92, and 0.94, since we are also interested in test coverage for those branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.
[ https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fulin wang updated HBASE-5874: -- Attachment: HBASE-5874-trunk.patch The TestHBaseFsck and TestMergeTool test case of trunk, I have tested. > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > > > Key: HBASE-5874 > URL: https://issues.apache.org/jira/browse/HBASE-5874 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch > > > The HBase do not configure the 'fs.default.name' attribute, the hbck tool and > Merge tool throw IllegalArgumentException. > the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to > the code. > hbck exception: > Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:128) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596) > at > org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332) > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907) > > Merge exception: > [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge > 381] exiting due to error > java.lang.IllegalArgumentException: Wrong FS: > hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: > file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412) > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634) > at > org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276) > at > org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261) > at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263288#comment-13263288 ] Elliott Clark commented on HBASE-5885: -- Stack and I were looking into this; I'll pick it back up first thing tomorrow morning. Brain dump below so that I remember it all in the morning. Running a standalone instance with hbase.regionserver.checksum.verify to true will see an error when trying to read. Running a standalone instance on the same data with hbase.regionserver.checksum.verify set to false will not see this error and all data looks intact. The first read through an HFileReader seems to be fine no matter what. It's the second read that seems to be erroring out. Looking at data files they all start with 'DATABLK*' and there are several of those headers in the files. So data too much data is being written per block, or not enough is being read. > Invalid HFile block magic on Local file System > -- > > Key: HBASE-5885 > URL: https://issues.apache.org/jira/browse/HBASE-5885 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.96.0 >Reporter: Elliott Clark >Priority: Blocker > Fix For: 0.94.0 > > > ERROR: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=7, exceptions: > Thu Apr 26 11:19:18 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for > reader > reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, > compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], > firstKey=01/info:data/1335463981520/Put, > lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, > avgValueLen=1000, entries=1215085, length=1264354417, > cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.io.IOException: Invalid HFile block magic: > \xEC\xD5\x9D\xB4\xC2bfo > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > ... 12 more > Thu Apr 26 11:19:19 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: java.lang.IllegalArgumentException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) > at s
[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5885: - Priority: Blocker (was: Major) Fix Version/s: 0.94.0 > Invalid HFile block magic on Local file System > -- > > Key: HBASE-5885 > URL: https://issues.apache.org/jira/browse/HBASE-5885 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.96.0 >Reporter: Elliott Clark >Priority: Blocker > Fix For: 0.94.0 > > > ERROR: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=7, exceptions: > Thu Apr 26 11:19:18 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for > reader > reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, > compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], > firstKey=01/info:data/1335463981520/Put, > lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, > avgValueLen=1000, entries=1215085, length=1264354417, > cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.io.IOException: Invalid HFile block magic: > \xEC\xD5\x9D\xB4\xC2bfo > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > ... 12 more > Thu Apr 26 11:19:19 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: java.lang.IllegalArgumentException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:216) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > at > org.apache.ha
[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263286#comment-13263286 ] Lars Hofhansl commented on HBASE-5885: -- Are you working on this Elliot? > Invalid HFile block magic on Local file System > -- > > Key: HBASE-5885 > URL: https://issues.apache.org/jira/browse/HBASE-5885 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.96.0 >Reporter: Elliott Clark >Priority: Blocker > Fix For: 0.94.0 > > > ERROR: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=7, exceptions: > Thu Apr 26 11:19:18 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for > reader > reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, > compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], > firstKey=01/info:data/1335463981520/Put, > lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, > avgValueLen=1000, entries=1215085, length=1264354417, > cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.io.IOException: Invalid HFile block magic: > \xEC\xD5\x9D\xB4\xC2bfo > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > ... 12 more > Thu Apr 26 11:19:19 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: java.lang.IllegalArgumentException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:216) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.jav
[jira] [Commented] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263276#comment-13263276 ] Hadoop QA commented on HBASE-5887: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524789/hbase-5887.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1661//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1661//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1661//console This message is automatically generated. > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0, 0.92.1, 0.94.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-5887-92.patch, hbase-5887.patch > > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263278#comment-13263278 ] Hadoop QA commented on HBASE-5887: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524790/hbase-5887-92.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1662//console This message is automatically generated. > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0, 0.92.1, 0.94.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-5887-92.patch, hbase-5887.patch > > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5872) Improve hadoopqa script to include checks for hadoop 0.23 build
[ https://issues.apache.org/jira/browse/HBASE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263266#comment-13263266 ] Hudson commented on HBASE-5872: --- Integrated in HBase-TRUNK #2820 (See [https://builds.apache.org/job/HBase-TRUNK/2820/]) HBASE-5872 Improve hadoopqa script to include checks for hadoop 0.23 build (Revision 1331143) Result = SUCCESS jmhsieh : Files : * /hbase/trunk/dev-support/test-patch.sh > Improve hadoopqa script to include checks for hadoop 0.23 build > --- > > Key: HBASE-5872 > URL: https://issues.apache.org/jira/browse/HBASE-5872 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 0.96.0 > > Attachments: hbase-5872.patch > > > There have been a few patches that have made it into hbase trunk that have > broken the compile of hbase against hadoop 0.23.x, without being known for a > few days. > We could have the bot do a few things: > 1) verify that patch compiles against hadoop 23 > 2) verify that unit tests pass against hadoop 23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[
https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263264#comment-13263264
]
Jonathan Hsieh edited comment on HBASE-5887 at 4/27/12 12:09 AM:
-
Yes. When you use the tool wrapper stuff you get parsers for that for free.
You could do something like this to override any of these values:
{code}
$ hbase org.apache.hadoop.hbase.TestAcidGuarantees -Dmillis=6
-DnumWriters=25 -DnumGetters=10 -DnumScanners=5 -DnumUniqueRows=5
{code}
I should probably add a '\-h' or '\-?' option to prompt usage instructions.
was (Author: jmhsieh):
Yes. When you use the tool wrapper stuff you get parsers for that for free.
You could do something like this to override any of these values:
{code}
$ hbase org.apache.hadoop.hbase.TestAcidGuarantees -Dmillis=6
-DnumWriters=25 -DnumGetters=10 -DnumScanners=5 -DnumUniqueRows=5
{code}
I should probably add a '-h' or '-?' option to prompt usage instructions.
> Make TestAcidGuarantees usable for system testing.
> --
>
> Key: HBASE-5887
> URL: https://issues.apache.org/jira/browse/HBASE-5887
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-5887-92.patch, hbase-5887.patch
>
>
> Currently, the TestAcidGuarantees run via main() will always abort with an
> NPE because it digs into a non-existant HBaseTestingUtility for a flusher
> thread. We should tool this up so that it works properly from the command
> line. This would be a very useful long running test when used in conjunction
> with fault injections to verify row acid properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[
https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263264#comment-13263264
]
Jonathan Hsieh commented on HBASE-5887:
---
Yes. When you use the tool wrapper stuff you get parsers for that for free.
You could do something like this to override any of these values:
{code}
$ hbase org.apache.hadoop.hbase.TestAcidGuarantees -Dmillis=6
-DnumWriters=25 -DnumGetters=10 -DnumScanners=5 -DnumUniqueRows=5
{code}
I should probably add a '-h' or '-?' option to prompt usage instructions.
> Make TestAcidGuarantees usable for system testing.
> --
>
> Key: HBASE-5887
> URL: https://issues.apache.org/jira/browse/HBASE-5887
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-5887-92.patch, hbase-5887.patch
>
>
> Currently, the TestAcidGuarantees run via main() will always abort with an
> NPE because it digs into a non-existant HBaseTestingUtility for a flusher
> thread. We should tool this up so that it works properly from the command
> line. This would be a very useful long running test when used in conjunction
> with fault injections to verify row acid properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5887: -- Attachment: hbase-5887-92.patch > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0, 0.92.1, 0.94.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-5887-92.patch, hbase-5887.patch > > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[
https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263261#comment-13263261
]
Zhihong Yu commented on HBASE-5887:
---
{code}
+int millis = c.getInt("millis", 5000);
+int numWriters = c.getInt("numWriters", 50);
+int numGetters = c.getInt("numGetters", 2);
+int numScanners = c.getInt("numScanners", 2);
+int numUniqueRows = c.getInt("numUniqueRows", 3);
{code}
Can user specify these config parameters from the command line ?
> Make TestAcidGuarantees usable for system testing.
> --
>
> Key: HBASE-5887
> URL: https://issues.apache.org/jira/browse/HBASE-5887
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-5887.patch
>
>
> Currently, the TestAcidGuarantees run via main() will always abort with an
> NPE because it digs into a non-existant HBaseTestingUtility for a flusher
> thread. We should tool this up so that it works properly from the command
> line. This would be a very useful long running test when used in conjunction
> with fault injections to verify row acid properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5887: -- Affects Version/s: 0.94.0 0.92.0 0.92.1 Status: Patch Available (was: Open) > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.1, 0.92.0, 0.94.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-5887.patch > > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5887: -- Attachment: hbase-5887.patch > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0, 0.92.1, 0.94.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-5887.patch > > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Attachment: v5.txt Compiles > Move SplitLogManager splitlog taskstate and AssignmentManager > RegionTransitionData znode datas to pb > - > > Key: HBASE-5869 > URL: https://issues.apache.org/jira/browse/HBASE-5869 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: firstcut.txt, secondcut.txt, v4.txt, v5.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263248#comment-13263248 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/ --- (Updated 2012-04-26 23:27:07.073810) Review request for hbase. Changes --- Updated for Stack's comments. Passes all unit tests. Summary --- Adds PB-based calls replacing HMasterRegionInterface. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. This addresses bug HBASE-5444. https://issues.apache.org/jira/browse/HBASE-5444 Diffs (updated) - src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 3c7c091 src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java efcf74d src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 src/main/protobuf/RegionServerStatus.proto PRE-CREATION src/main/protobuf/hbase.proto 12e6053 src/main/resources/hbase-webapps/master/table.jsp 3ef1190 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java d039be3 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 36046f8 src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 Diff: https://reviews.apache.org/r/4463/diff Testing --- Ran jenkins job, all unit tests passed. Thanks, Gregory > Add PB-based calls to HMasterRegionInterface > > > Key: HBASE-5444 > URL: https://issues.apache.org/jira/browse/HBASE-5444 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Todd Lipcon >Assignee: Gregory Chanan > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5887: -- Description: Currently, the TestAcidGuarantees run via main() will always abort with an NPE because it digs into a non-existant HBaseTestingUtility for a flusher thread. We should tool this up so that it works properly from the command line. This would be a very useful long running test when used in conjunction with fault injections to verify row acid properties. (was: Currently, the TestAcidGuarantees run via main() will always abort with an NPE because it digs into a non-existant HBaseTestingUtility for a flusher thread. We should tool this up so that it works properly from the command line. This is a would be a very useful long running test when used in conjunction with fault injections to verify row acid properties.) > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This would be a very useful long running test when used in conjunction > with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh reassigned HBASE-5887: - Assignee: Jonathan Hsieh > Make TestAcidGuarantees usable for system testing. > -- > > Key: HBASE-5887 > URL: https://issues.apache.org/jira/browse/HBASE-5887 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > > Currently, the TestAcidGuarantees run via main() will always abort with an > NPE because it digs into a non-existant HBaseTestingUtility for a flusher > thread. We should tool this up so that it works properly from the command > line. This is a would be a very useful long running test when used in > conjunction with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
Jonathan Hsieh created HBASE-5887: - Summary: Make TestAcidGuarantees usable for system testing. Key: HBASE-5887 URL: https://issues.apache.org/jira/browse/HBASE-5887 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Currently, the TestAcidGuarantees run via main() will always abort with an NPE because it digs into a non-existant HBaseTestingUtility for a flusher thread. We should tool this up so that it works properly from the command line. This is a would be a very useful long running test when used in conjunction with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5872) Improve hadoopqa script to include checks for hadoop 0.23 build
[ https://issues.apache.org/jira/browse/HBASE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5872: -- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for reviews and Ted and Stack. Committed to trunk/0.96. > Improve hadoopqa script to include checks for hadoop 0.23 build > --- > > Key: HBASE-5872 > URL: https://issues.apache.org/jira/browse/HBASE-5872 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 0.96.0 > > Attachments: hbase-5872.patch > > > There have been a few patches that have made it into hbase trunk that have > broken the compile of hbase against hadoop 0.23.x, without being known for a > few days. > We could have the bot do a few things: > 1) verify that patch compiles against hadoop 23 > 2) verify that unit tests pass against hadoop 23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263206#comment-13263206 ] Lars Hofhansl commented on HBASE-5885: -- So specifically this only happens in local mode (where HBase writes directly to the local file system)? If so, can we disable this feature in local mode (where it is pointless anyway) > Invalid HFile block magic on Local file System > -- > > Key: HBASE-5885 > URL: https://issues.apache.org/jira/browse/HBASE-5885 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.96.0 >Reporter: Elliott Clark > > ERROR: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=7, exceptions: > Thu Apr 26 11:19:18 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for > reader > reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, > compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], > firstKey=01/info:data/1335463981520/Put, > lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, > avgValueLen=1000, entries=1215085, length=1264354417, > cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.io.IOException: Invalid HFile block magic: > \xEC\xD5\x9D\xB4\xC2bfo > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > ... 12 more > Thu Apr 26 11:19:19 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: java.lang.IllegalArgumentException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:216) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) >
[jira] [Updated] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5801: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Jimmy. Committed to 0.90. > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch, hbase_5801_v3.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[
https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263194#comment-13263194
]
Jonathan Hsieh commented on HBASE-5801:
---
Hey Jimmy, code looks good, I found a few typos, will fix before commit.
Typos: are -> is ; multipe -> multiple
{code}
+ LOG.warn("Region " + hbi.toString() + " are deployed on multiple
region servers."
++ " Please fix the multiple assignments before fixing multipe
table desc.");
{code}
Remove @param / @praam / @throws (or fill them in).
{code}
+ * Replace the .regioninfo with a new one with the expected table desc,
+ * then re-assign the region.
+ *
+ * @param admin
+ * @praam hsa
+ * @param hbi
+ * @param htd
+ * @param sidelineTableDir
+ * @throws IOException
+ * @throws KeeperException
+ * @throws InterruptedException
+
{code}
typo: regioninfoSidelinPath -> regioninfoSidelinePath
{code}
+Path regioninfoSidelinPath = new Path(sidelineRegionDir,
HRegion.REGIONINFO_FILE);
{code}
> [hbck] Hbck should handle case where some regions have different HTD settings
> in .regioninfo files (0.90 specific)
> ---
>
> Key: HBASE-5801
> URL: https://issues.apache.org/jira/browse/HBASE-5801
> Project: HBase
> Issue Type: Improvement
> Components: hbck
>Affects Versions: 0.90.7
>Reporter: Jonathan Hsieh
>Assignee: Jimmy Xiang
> Fix For: 0.90.7
>
> Attachments: hbase_5801_v2.patch, hbase_5801_v3.patch
>
>
> Recently, we encountered a case where some regions in a table have different
> HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo
> file. hbck expects all HTDs within a table to be the same and currently
> bails out in this situation.
> We need to either point out a proper set of actions for the user to execute
> or automatically convert the region to a common HTD (likely the most common
> on, or possibly the "first one".)
> Not sure if this requires reformatting data but may require closing and
> restarting a region.
> This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single
> .tableinfo file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263185#comment-13263185
]
Hudson commented on HBASE-5864:
---
Integrated in HBase-TRUNK #2819 (See
[https://builds.apache.org/job/HBase-TRUNK/2819/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331058)
Result = SUCCESS
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
> Error while reading from hfile in 0.94
> --
>
> Key: HBASE-5864
> URL: https://issues.apache.org/jira/browse/HBASE-5864
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
> HBASE-5864_3.patch, HBASE-5864_test.patch
>
>
> Got the following stacktrace during region split.
> {noformat}
> 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
> Failed getting store size for value
> java.io.IOException: Requested block is out of range: 2906737606134037404,
> lastDataBlockOffset: 84764558
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263183#comment-13263183 ] Alex Newman commented on HBASE-2600: I needed to rebase again. I'll upload a patch in a second after I run it through my jenkins. > Change how we do meta tables; from tablename+STARTROW+randomid to instead, > tablename+ENDROW+randomid > > > Key: HBASE-2600 > URL: https://issues.apache.org/jira/browse/HBASE-2600 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Alex Newman > Attachments: > 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, > 0001-HBASE-2600-v11.patch, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, > 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, > 0001-HBASE-2600.v10.patch, 2600-trunk-01-17.txt, > HBASE-2600+5217-Sun-Mar-25-2012-v3.patch, > HBASE-2600+5217-Sun-Mar-25-2012-v4.patch, hbase-2600-root.dir.tgz, jenkins.pdf > > > This is an idea that Ryan and I have been kicking around on and off for a > while now. > If regionnames were made of tablename+endrow instead of tablename+startrow, > then in the metatables, doing a search for the region that contains the > wanted row, we'd just have to open a scanner using passed row and the first > row found by the scan would be that of the region we need (If offlined > parent, we'd have to scan to the next row). > If we redid the meta tables in this format, we'd be using an access that is > natural to hbase, a scan as opposed to the perverse, expensive > getClosestRowBefore we currently have that has to walk backward in meta > finding a containing region. > This issue is about changing the way we name regions. > If we were using scans, prewarming client cache would be near costless (as > opposed to what we'll currently have to do which is first a > getClosestRowBefore and then a scan from the closestrowbefore forward). > Converting to the new method, we'd have to run a migration on startup > changing the content in meta. > Up to this, the randomid component of a region name has been the timestamp of > region creation. HBASE-2531 "32-bit encoding of regionnames waaay > too susceptible to hash clashes" proposes changing the randomid so that it > contains actual name of the directory in the filesystem that hosts the > region. If we had this in place, I think it would help with the migration to > this new way of doing the meta because as is, the region name in fs is a hash > of regionname... changing the format of the regionname would mean we generate > a different hash... so we'd need hbase-2531 to be in place before we could do > this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263169#comment-13263169 ] Hadoop QA commented on HBASE-5801: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524474/hbase_5801_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1660//console This message is automatically generated. > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch, hbase_5801_v3.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5877) When a query fails because the region has moved, let the regionserver return the new address to the client
[
https://issues.apache.org/jira/browse/HBASE-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263140#comment-13263140
]
nkeywal commented on HBASE-5877:
bq. This patch will benefit any move, not just rolling restart, right?
Yes, but as there is a wait time between two tries, I think the benefit will be
minimal vs. the wait time for a single client. I could add an heuristic like if
region was closed more than 2 seconds ago, consider that it's now available on
the new server and don't sleep before the next retry. That could lead of having
more network messages if the rule is wrong (and the rule will be wrong when the
system is overloaded), and it will add some complexity to the client code.
Having the real status of the region would solve this.
Anyway, with the dev already done to cut the link between master & clients, it
can help to save a reconnect to master. And during a rolling restart with
regions moving everywhere, I think it will make a real difference.
bq. I don't see changes to make use of this new functionality? I'd expect the
balancer in master to make use of it?
Yes, it's the changes in AssignmentManager: the changes are in the patch, but
are quite small at the end: basically:
{noformat}
-unassign(plan.getRegionInfo());
+unassign(plan.getRegionInfo(), false, plan.getDestination());
{noformat}
I still need to manage the case when the destination is not specified at the
beginning.
> When a query fails because the region has moved, let the regionserver return
> the new address to the client
> --
>
> Key: HBASE-5877
> URL: https://issues.apache.org/jira/browse/HBASE-5877
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5877.v1.patch
>
>
> This is mainly useful when we do a rolling restart. This will decrease the
> load on the master and the network load.
> Note that a region is not immediately opened after a close. So:
> - it seems preferable to wait before retrying on the other server. An
> optimisation would be to have an heuristic depending on when the region was
> closed.
> - during a rolling restart, the server moves the regions then stops. So we
> may have failures when the server is stopped, and this patch won't help.
> The implementation in the first patch does:
> - on the region move, there is an added parameter on the regionserver#close
> to say where we are sending the region
> - the regionserver keeps a list of what was moved. Each entry is kept 100
> seconds.
> - the regionserver sends a specific exception when it receives a query on a
> moved region. This exception contains the new address.
> - the client analyses the exeptions and update its cache accordingly...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263136#comment-13263136
]
Hudson commented on HBASE-5864:
---
Integrated in HBase-0.94-security #22 (See
[https://builds.apache.org/job/HBase-0.94-security/22/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331057)
Result = FAILURE
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
> Error while reading from hfile in 0.94
> --
>
> Key: HBASE-5864
> URL: https://issues.apache.org/jira/browse/HBASE-5864
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
> HBASE-5864_3.patch, HBASE-5864_test.patch
>
>
> Got the following stacktrace during region split.
> {noformat}
> 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
> Failed getting store size for value
> java.io.IOException: Requested block is out of range: 2906737606134037404,
> lastDataBlockOffset: 84764558
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5873) TimeOut Monitor thread should be started after atleast one region server registers.
[
https://issues.apache.org/jira/browse/HBASE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263135#comment-13263135
]
Hudson commented on HBASE-5873:
---
Integrated in HBase-0.94-security #22 (See
[https://builds.apache.org/job/HBase-0.94-security/22/])
HBASE-5873 TimeOut Monitor thread should be started after atleast one
region server registers. (Revision 1330549)
Result = FAILURE
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> TimeOut Monitor thread should be started after atleast one region server
> registers.
> ---
>
> Key: HBASE-5873
> URL: https://issues.apache.org/jira/browse/HBASE-5873
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: ramkrishna.s.vasudevan
>Assignee: rajeshbabu
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5873-trunk.txt, HBASE-5873.patch
>
>
> Currently timeout monitor thread is started even before the region server has
> registered with the master.
> In timeout monitor we depend on the region server to be online
> {code}
> boolean allRSsOffline = this.serverManager.getOnlineServersList().
> isEmpty();
> {code}
> Now when the master starts up it sees there are no online servers and hence
> sets
> allRSsOffline to true.
> {code}
> setAllRegionServersOffline(allRSsOffline);
> {code}
> So this.allRegionServersOffline is also true.
> By this time an RS has come up,
> Now timeout comes up again (after 10secs) in the next cycle he sees
> allRSsOffline as false.
> Hence
> {code}
> else if (this.allRegionServersOffline && !allRSsOffline) {
> // if some RSs just came back online, we can start the
> // the assignment right away
> actOnTimeOut(regionState);
> {code}
> This condition makes him to take action based on timeout.
> Because of this even if one Region assignment of ROOT is going on, this piece
> of code triggers another assignment and thus we get RegionAlreadyinTransition
> Exception. Later we need to wait for 30 mins for assigning ROOT itself.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263134#comment-13263134 ] Hudson commented on HBASE-5862: --- Integrated in HBase-0.94-security #22 (See [https://builds.apache.org/job/HBase-0.94-security/22/]) HBASE-5862 After Region Close remove the Operation Metrics; ADDENDUM -- missing import (Revision 1331040) HBASE-5862 After Region Close remove the Operation Metrics (Revision 1330998) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/OperationMetrics.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionMetricsStorage.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5745) SequenceFileLogReader#getPos may get wrong length on DFS restart
[ https://issues.apache.org/jira/browse/HBASE-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263130#comment-13263130 ] Uma Maheswara Rao G commented on HBASE-5745: I have just committed the HDFS issue in trunk and branch-2. For supporting older released versions, we may have to implement the similar logic in Hbase SFLR, to ensure the correct length.Otherwise this is a dataloss case from Hbase perspective. > SequenceFileLogReader#getPos may get wrong length on DFS restart > > > Key: HBASE-5745 > URL: https://issues.apache.org/jira/browse/HBASE-5745 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.96.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Critical > > This is actually a kind of integration bug from Hbase perspective. > Currently HDFS will count the partial block length as 0, if there are no > locations found for the partial block. This can happend opn DFS restart, > before DNs completely reports to NN. > Explained the scenario in HDFS-3222. Actually this is a bug in HDFS. we may > solve this in latest versions. > So, whatever the versions Hbase using may have this bug. HMaster may not be > able to replay the complete edits if there is an Hmaster switch also at the > same time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263129#comment-13263129 ] Elliott Clark commented on HBASE-5886: -- * This patch fails unit tests. org.apache.hadoop.hbase.io.TestHeapSize * recordPossibleDataLossNoWal seems like the wrong name. There's no possible data loss until a region server crashes without flushing. Maybe something like recordPutWithoutWal. * I don't think the warn in recordPossibleDataLossNoWal is the right level. Nothing has gone wrong; someone has just decided to opt for a speedy put vs a safe one. I'm not even sure that a log is needed here. * How far off of the actual size of the data is the heapSize of the kv ? * Putting data about puts without WAL into org.apache.hadoop.hbase.regionserver.metrics.OperationMetrics seems like it would be good. > Add new metric for possible data loss due to puts without WAL > -- > > Key: HBASE-5886 > URL: https://issues.apache.org/jira/browse/HBASE-5886 > Project: HBase > Issue Type: New Feature > Components: metrics, regionserver >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Labels: metrics > Attachments: HBASE-5886-v0.patch > > > Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263111#comment-13263111 ] Hadoop QA commented on HBASE-5886: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524562/HBASE-5886-v0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.TestHeapSize Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1659//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1659//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1659//console This message is automatically generated. > Add new metric for possible data loss due to puts without WAL > -- > > Key: HBASE-5886 > URL: https://issues.apache.org/jira/browse/HBASE-5886 > Project: HBase > Issue Type: New Feature > Components: metrics, regionserver >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Labels: metrics > Attachments: HBASE-5886-v0.patch > > > Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263099#comment-13263099 ] Hudson commented on HBASE-5862: --- Integrated in HBase-0.94 #151 (See [https://builds.apache.org/job/HBase-0.94/151/]) HBASE-5862 After Region Close remove the Operation Metrics; ADDENDUM -- missing import (Revision 1331040) Result = ABORTED stack : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263100#comment-13263100
]
Hudson commented on HBASE-5864:
---
Integrated in HBase-0.94 #151 (See
[https://builds.apache.org/job/HBase-0.94/151/])
HBASE-5864 Error while reading from hfile in 0.94 (Ram) (Revision 1331057)
Result = ABORTED
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
> Error while reading from hfile in 0.94
> --
>
> Key: HBASE-5864
> URL: https://issues.apache.org/jira/browse/HBASE-5864
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
> HBASE-5864_3.patch, HBASE-5864_test.patch
>
>
> Got the following stacktrace during region split.
> {noformat}
> 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
> Failed getting store size for value
> java.io.IOException: Requested block is out of range: 2906737606134037404,
> lastDataBlockOffset: 84764558
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263094#comment-13263094 ] Lars Hofhansl commented on HBASE-5862: -- Thanks Elliot. > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5866) Canary in tool package but says its in tools.
[ https://issues.apache.org/jira/browse/HBASE-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5866: - Issue Type: Bug (was: New Feature) > Canary in tool package but says its in tools. > - > > Key: HBASE-5866 > URL: https://issues.apache.org/jira/browse/HBASE-5866 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.94.0, 0.96.0 > > Attachments: 5866.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5866) Canary in tool package but says its in tools.
[ https://issues.apache.org/jira/browse/HBASE-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5866: - Issue Type: New Feature (was: Bug) > Canary in tool package but says its in tools. > - > > Key: HBASE-5866 > URL: https://issues.apache.org/jira/browse/HBASE-5866 > Project: HBase > Issue Type: New Feature >Reporter: stack >Assignee: stack > Fix For: 0.94.0, 0.96.0 > > Attachments: 5866.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5737) Minor Improvements related to balancer.
[ https://issues.apache.org/jira/browse/HBASE-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5737: - Issue Type: Improvement (was: Bug) > Minor Improvements related to balancer. > --- > > Key: HBASE-5737 > URL: https://issues.apache.org/jira/browse/HBASE-5737 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5737.patch, HBASE-5737_1.patch, > HBASE-5737_2.patch, HBASE-5737_3.patch > > > Currently in Am.getAssignmentByTable() we use a result map which is currenly > a hashmap. It could be better if we have a treeMap. Even in > MetaReader.fullScan we have the treeMap only so that we have the naming order > maintained. I felt this change could be very useful in cases where we are > extending the DefaultLoadBalancer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5864:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.94 and 0.96.
Thanks for the Ram.
Thanks for reviews Dhruba and Ted.
> Error while reading from hfile in 0.94
> --
>
> Key: HBASE-5864
> URL: https://issues.apache.org/jira/browse/HBASE-5864
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
> HBASE-5864_3.patch, HBASE-5864_test.patch
>
>
> Got the following stacktrace during region split.
> {noformat}
> 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
> Failed getting store size for value
> java.io.IOException: Requested block is out of range: 2906737606134037404,
> lastDataBlockOffset: 84764558
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262994#comment-13262994
]
Todd Lipcon commented on HBASE-5886:
I think the metrics names could be improved here -- "putWithoutWAL" could be
"numPutsWithoutWAL" and "possibleDataLossSize" perhaps "mbInMemoryWithoutWAL"?
- in recordPossibleDataLossNoWal, use {{getAndIncrement}} instead of
{{incrementAndGet}}, and compare the result to 0 -- otherwise you're doing an
extra atomic op and have a potential race
- the warning message should include the client IP address as well as the
region ID. Perhaps something like "Client 123.123.123.123 writing data to
region abcdef12345 with WAL disabled. Data may be lost in the event of a crash.
Will not log further warnings for this region."
- the debug message when setting possibleDataLossSize back to 0 seems
unnecessary.
- DEFAULT_WARN_NO_WAL_INTERVAL is unused
> Add new metric for possible data loss due to puts without WAL
> --
>
> Key: HBASE-5886
> URL: https://issues.apache.org/jira/browse/HBASE-5886
> Project: HBase
> Issue Type: New Feature
> Components: metrics, regionserver
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Labels: metrics
> Attachments: HBASE-5886-v0.patch
>
>
> Add a metrics to keep track of puts without WAL and possible data loss size.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5886: --- Status: Patch Available (was: Open) > Add new metric for possible data loss due to puts without WAL > -- > > Key: HBASE-5886 > URL: https://issues.apache.org/jira/browse/HBASE-5886 > Project: HBase > Issue Type: New Feature > Components: metrics, regionserver >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Labels: metrics > Attachments: HBASE-5886-v0.patch > > > Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5886: --- Attachment: HBASE-5886-v0.patch > Add new metric for possible data loss due to puts without WAL > -- > > Key: HBASE-5886 > URL: https://issues.apache.org/jira/browse/HBASE-5886 > Project: HBase > Issue Type: New Feature > Components: metrics, regionserver >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Labels: metrics > Attachments: HBASE-5886-v0.patch > > > Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
Matteo Bertozzi created HBASE-5886: -- Summary: Add new metric for possible data loss due to puts without WAL Key: HBASE-5886 URL: https://issues.apache.org/jira/browse/HBASE-5886 Project: HBase Issue Type: New Feature Components: metrics, regionserver Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5851) TestProcessBasedCluster sometimes fails; currently disabled -- needs to be fixed and reenabled
[
https://issues.apache.org/jira/browse/HBASE-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-5851:
---
Status: Open (was: Patch Available)
Agree with Stack, still flaky, will look into it more later.
> TestProcessBasedCluster sometimes fails; currently disabled -- needs to be
> fixed and reenabled
> --
>
> Key: HBASE-5851
> URL: https://issues.apache.org/jira/browse/HBASE-5851
> Project: HBase
> Issue Type: Test
>Reporter: Zhihong Yu
>Assignee: Jimmy Xiang
> Attachments: disable.txt, hbase-5851.patch, hbase-5851_v2.patch,
> metahang.txt, zkfail.txt
>
>
> TestProcessBasedCluster failed in
> https://builds.apache.org/job/HBase-TRUNK-security/178
> Looks like cluster failed to start:
> {code}
> 2012-04-21 14:22:32,666 INFO [Thread-1]
> util.ProcessBasedLocalHBaseCluster(176): Waiting for HBase to startup.
> Retries left: 2
> java.io.IOException: Giving up trying to location region in meta: thread is
> interrupted.
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1173)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:956)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:917)
> at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:252)
> at org.apache.hadoop.hbase.client.HTable.(HTable.java:192)
> at
> org.apache.hadoop.hbase.util.ProcessBasedLocalHBaseCluster.startHBase(ProcessBasedLocalHBaseCluster.java:174)
> at
> org.apache.hadoop.hbase.util.TestProcessBasedCluster.testProcessBasedCluster(TestProcessBasedCluster.java:56)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
> java.lang.InterruptedException: sleep interrupted at
> java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:134)
> at
> org.apache.hadoop.hbase.util.ProcessBasedLocalHBaseCluster.startHBase(ProcessBasedLocalHBaseCluster.java:178)
> at
> org.apache.hadoop.hbase.util.TestProcessBasedCluster.testProcessBasedCluster(TestProcessBasedCluster.java:56)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262880#comment-13262880 ] Elliott Clark commented on HBASE-5885: -- I don't get an exception when that is set to false. > Invalid HFile block magic on Local file System > -- > > Key: HBASE-5885 > URL: https://issues.apache.org/jira/browse/HBASE-5885 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.96.0 >Reporter: Elliott Clark > > ERROR: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=7, exceptions: > Thu Apr 26 11:19:18 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for > reader > reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, > compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], > firstKey=01/info:data/1335463981520/Put, > lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, > avgValueLen=1000, entries=1215085, length=1264354417, > cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.io.IOException: Invalid HFile block magic: > \xEC\xD5\x9D\xB4\xC2bfo > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > ... 12 more > Thu Apr 26 11:19:19 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: java.lang.IllegalArgumentException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:216) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > at > org.apache.hadoop.hbas
[jira] [Commented] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy
[ https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262841#comment-13262841 ] Phabricator commented on HBASE-5879: Kannan has accepted the revision "[jira] [HBASE-5879] [89-fb] Enable JMX metrics collection for the Thrift proxy". REVISION DETAIL https://reviews.facebook.net/D2955 BRANCH enable_jmx_metrics_collection_for_the_thrift_HBASE-5879 > Enable JMX metrics collection for the Thrift proxy > -- > > Key: HBASE-5879 > URL: https://issues.apache.org/jira/browse/HBASE-5879 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D2955.1.patch > > > We need to enable JMX on the Thrift proxy on a separate port different from > the JMX port used by regionserver. This is necessary for metrics collection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy
[ https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5879: --- Attachment: D2955.1.patch mbautin requested code review of "[jira] [HBASE-5879] [89-fb] Enable JMX metrics collection for the Thrift proxy". Reviewers: Kannan, Liyin, sc, tedyu, JIRA We need to enable JMX on the Thrift proxy on a separate port different from the JMX port used by regionserver. This is necessary for metrics collection. TEST PLAN - Deploy to dev cluster. - Verify that it is possible to collect metrics through JMX from the Thrift proxy. REVISION DETAIL https://reviews.facebook.net/D2955 AFFECTED FILES bin/hbase-config.sh MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6735/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Enable JMX metrics collection for the Thrift proxy > -- > > Key: HBASE-5879 > URL: https://issues.apache.org/jira/browse/HBASE-5879 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D2955.1.patch > > > We need to enable JMX on the Thrift proxy on a separate port different from > the JMX port used by regionserver. This is necessary for metrics collection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
[
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262837#comment-13262837
]
stack commented on HBASE-5885:
--
What if you disable this setting? Does it still fail?
{code}
+ /**
+ * If this parameter is set to true, then hbase will read
+ * data and then verify checksums. Checksum verification
+ * inside hdfs will be switched off. However, if the hbase-checksum
+ * verification fails, then it will switch back to using
+ * hdfs checksums for verifiying data that is being read from storage.
+ *
+ * If this parameter is set to false, then hbase will not
+ * verify any checksums, instead it will depend on checksum verification
+ * being done in the hdfs client.
+ */
+ public static final String HBASE_CHECKSUM_VERIFICATION =
+ "hbase.regionserver.checksum.verify";
{code}
> Invalid HFile block magic on Local file System
> --
>
> Key: HBASE-5885
> URL: https://issues.apache.org/jira/browse/HBASE-5885
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Elliott Clark
>
> ERROR: java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=7, exceptions:
> Thu Apr 26 11:19:18 PDT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException:
> java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for
> reader
> reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
> compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false],
> firstKey=01/info:data/1335463981520/Put,
> lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30,
> avgValueLen=1000, entries=1215085, length=1264354417,
> cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
> Caused by: java.io.IOException: Invalid HFile block magic:
> \xEC\xD5\x9D\xB4\xC2bfo
> at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
> ... 12 more
> Thu Apr 26 11:19:19 PDT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException:
> java.io.IOException: java.lang.IllegalArgumentException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImp
[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262828#comment-13262828 ] Elliott Clark commented on HBASE-5885: -- Forgot to add that I also tried this with and without HBASE-5864 > Invalid HFile block magic on Local file System > -- > > Key: HBASE-5885 > URL: https://issues.apache.org/jira/browse/HBASE-5885 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.96.0 >Reporter: Elliott Clark > > ERROR: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=7, exceptions: > Thu Apr 26 11:19:18 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for > reader > reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, > compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], > firstKey=01/info:data/1335463981520/Put, > lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, > avgValueLen=1000, entries=1215085, length=1264354417, > cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.io.IOException: Invalid HFile block magic: > \xEC\xD5\x9D\xB4\xC2bfo > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > ... 12 more > Thu Apr 26 11:19:19 PDT 2012, > org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: > java.io.IOException: java.lang.IllegalArgumentException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:216) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) > at > org.apach
[jira] [Created] (HBASE-5885) Invalid HFile block magic on Local file System
Elliott Clark created HBASE-5885: Summary: Invalid HFile block magic on Local file System Key: HBASE-5885 URL: https://issues.apache.org/jira/browse/HBASE-5885 Project: HBase Issue Type: Bug Affects Versions: 0.94.0, 0.96.0 Reporter: Elliott Clark ERROR: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=7, exceptions: Thu Apr 26 11:19:18 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for reader reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=01/info:data/1335463981520/Put, lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, avgValueLen=1000, entries=1215085, length=1264354417, cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) Caused by: java.io.IOException: Invalid HFile block magic: \xEC\xD5\x9D\xB4\xC2bfo at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) at org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:254) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) ... 12 more Thu Apr 26 11:19:19 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:216) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:406) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) at org.apache.hadoop.hbase.region
[jira] [Commented] (HBASE-5877) When a query fails because the region has moved, let the regionserver return the new address to the client
[
https://issues.apache.org/jira/browse/HBASE-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262820#comment-13262820
]
stack commented on HBASE-5877:
--
@N This patch will benefit any move, not just rolling restart, right?
Here are a quick couple of comments on the patch.
I like addition of RegionMovedException.
Convention is to capitalize static defines: '+ private static final String
hostField = "hostname=";' so it should be HOSTFIELD.
Its super ugly that you have to parse exception message to get exception data
member fields... but thats not your fault.
Please keep the style of the surrounding code. This kinda thing is
unconventional:
{code}
+private void updateCachedLocations(
+ Set updateHistory,
+ HRegionLocation hrl,
+ Object t) {
{code}
Ugh on how ugly it is updating cache. Again, not your fault.
Ted suggests updating interface version. Maybe don't. If you do, you can't
get this into a 0.94.1, etc.
I don't see changes to make use of this new functionality? I'd expect the
balancer in master to make use of it?
> When a query fails because the region has moved, let the regionserver return
> the new address to the client
> --
>
> Key: HBASE-5877
> URL: https://issues.apache.org/jira/browse/HBASE-5877
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5877.v1.patch
>
>
> This is mainly useful when we do a rolling restart. This will decrease the
> load on the master and the network load.
> Note that a region is not immediately opened after a close. So:
> - it seems preferable to wait before retrying on the other server. An
> optimisation would be to have an heuristic depending on when the region was
> closed.
> - during a rolling restart, the server moves the regions then stops. So we
> may have failures when the server is stopped, and this patch won't help.
> The implementation in the first patch does:
> - on the region move, there is an added parameter on the regionserver#close
> to say where we are sending the region
> - the regionserver keeps a list of what was moved. Each entry is kept 100
> seconds.
> - the regionserver sends a specific exception when it receives a query on a
> moved region. This exception contains the new address.
> - the client analyses the exeptions and update its cache accordingly...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-5801: --- Attachment: hbase_5801_v3.patch > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch, hbase_5801_v3.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-5801: --- Status: Patch Available (was: Open) I ran TestHBaseFsck 10x with v3 patch and all passed. > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch, hbase_5801_v3.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-5801: --- Status: Open (was: Patch Available) > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch, hbase_5801_v3.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262816#comment-13262816 ] Jonathan Hsieh commented on HBASE-5801: --- Thanks for following through Jimmy. Adding more flakey tests is just going to cause more trouble down the line and it is better if we figure out and catch them before they get in! > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262813#comment-13262813 ] Elliott Clark commented on HBASE-5862: -- @lars Yes there's a missing api. Hadoop metrics keeps a copy of all metrics created. That copy is used to expose the data to jmx and other consumers. There is no remove function. HADOOP-8313 was filed to correct this. However until that changes reflection was the only way. > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[
https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262808#comment-13262808
]
Zhihong Yu commented on HBASE-5862:
---
{code}
+//per hfile. Figuring out which cfs, hfiles, ...
{code}
Should cfs be in expanded form (column families) ?
{code}
+//and on the next tick of the metrics everything that is still relevant
will be
+//re-added.
{code}
're-added' -> 'added' or 'added again'
The initialization work in clear() should be moved to
RegionServerDynamicMetrics ctor because it is one time operation.
> After Region Close remove the Operation Metrics.
>
>
> Key: HBASE-5862
> URL: https://issues.apache.org/jira/browse/HBASE-5862
> Project: HBase
> Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
> Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch,
> HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch,
> HBASE-5862-94-3.patch, TSD.png
>
>
> If a region is closed then Hadoop metrics shouldn't still be reporting about
> that region.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262812#comment-13262812
]
Lars Hofhansl commented on HBASE-5864:
--
Going to commit in a few unless there're objections.
> Error while reading from hfile in 0.94
> --
>
> Key: HBASE-5864
> URL: https://issues.apache.org/jira/browse/HBASE-5864
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
> HBASE-5864_3.patch, HBASE-5864_test.patch
>
>
> Got the following stacktrace during region split.
> {noformat}
> 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
> Failed getting store size for value
> java.io.IOException: Requested block is out of range: 2906737606134037404,
> lastDataBlockOffset: 84764558
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
> at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262810#comment-13262810 ] Lars Hofhansl commented on HBASE-5862: -- I don't get the Hadoop private field accessor stuff. Why do we need to clear out private fields? Is there an API missing for this in Hadoop? > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5801) [hbck] Hbck should handle case where some regions have different HTD settings in .regioninfo files (0.90 specific)
[ https://issues.apache.org/jira/browse/HBASE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262807#comment-13262807 ] Jimmy Xiang commented on HBASE-5801: I looked into it and found out why deleteTable fails: one of the region is not closed. I will put up another patch soon. > [hbck] Hbck should handle case where some regions have different HTD settings > in .regioninfo files (0.90 specific) > --- > > Key: HBASE-5801 > URL: https://issues.apache.org/jira/browse/HBASE-5801 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.90.7 >Reporter: Jonathan Hsieh >Assignee: Jimmy Xiang > Fix For: 0.90.7 > > Attachments: hbase_5801_v2.patch > > > Recently, we encountered a case where some regions in a table have different > HTableDescriptor settings serialized into HDFS their HRegionInfo .regioninfo > file. hbck expects all HTDs within a table to be the same and currently > bails out in this situation. > We need to either point out a proper set of actions for the user to execute > or automatically convert the region to a common HTD (likely the most common > on, or possibly the "first one".) > Not sure if this requires reformatting data but may require closing and > restarting a region. > This issue is hbase 0.90.x specific -- 0.92+ keep all table info in a single > .tableinfo file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5862: - Resolution: Fixed Release Note: Fixes the case where when a region closed, its metrics did not; they stayed up associated w/ old hosting server even though region may have moved elsewhere. Status: Resolved (was: Patch Available) Committed trunk and 0.94. Thanks for the patch Elliott > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262802#comment-13262802 ] Hadoop QA commented on HBASE-5862: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524470/HBASE-5862-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1658//console This message is automatically generated. > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5829) Inconsistency between the "regions" map and the "servers" map in AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5829:
-
Resolution: Fixed
Fix Version/s: 0.96.0
Assignee: Maryann Xue
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Applied to trunk. Letting patch hang out in case someone wants to apply it to
other branches.
I added you as a contributor Maryann and assigned you this issue (You can
assign yourself issues going forward). Thanks for the patch.
> Inconsistency between the "regions" map and the "servers" map in
> AssignmentManager
> --
>
> Key: HBASE-5829
> URL: https://issues.apache.org/jira/browse/HBASE-5829
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Fix For: 0.96.0
>
> Attachments: HBASE-5829-0.90.patch, HBASE-5829-trunk.patch
>
>
> There are occurrences in AM where this.servers is not kept consistent with
> this.regions. This might cause balancer to offline a region from the RS that
> already returned NotServingRegionException at a previous offline attempt.
> In AssignmentManager.unassign(HRegionInfo, boolean)
> try {
> // TODO: We should consider making this look more like it does for the
> // region open where we catch all throwables and never abort
> if (serverManager.sendRegionClose(server, state.getRegion(),
> versionOfClosingNode)) {
> LOG.debug("Sent CLOSE to " + server + " for region " +
> region.getRegionNameAsString());
> return;
> }
> // This never happens. Currently regionserver close always return true.
> LOG.warn("Server " + server + " region CLOSE RPC returned false for " +
> region.getRegionNameAsString());
> } catch (NotServingRegionException nsre) {
> LOG.info("Server " + server + " returned " + nsre + " for " +
> region.getRegionNameAsString());
> // Presume that master has stale data. Presume remote side just split.
> // Presume that the split message when it comes in will fix up the
> master's
> // in memory cluster state.
> } catch (Throwable t) {
> if (t instanceof RemoteException) {
> t = ((RemoteException)t).unwrapRemoteException();
> if (t instanceof NotServingRegionException) {
> if (checkIfRegionBelongsToDisabling(region)) {
> // Remove from the regionsinTransition map
> LOG.info("While trying to recover the table "
> + region.getTableNameAsString()
> + " to DISABLED state the region " + region
> + " was offlined but the table was in DISABLING state");
> synchronized (this.regionsInTransition) {
> this.regionsInTransition.remove(region.getEncodedName());
> }
> // Remove from the regionsMap
> synchronized (this.regions) {
> this.regions.remove(region);
> }
> deleteClosingOrClosedNode(region);
> }
> }
> // RS is already processing this region, only need to update the
> timestamp
> if (t instanceof RegionAlreadyInTransitionException) {
> LOG.debug("update " + state + " the timestamp.");
> state.update(state.getState());
> }
> }
> In AssignmentManager.assign(HRegionInfo, RegionState, boolean, boolean,
> boolean)
> synchronized (this.regions) {
> this.regions.put(plan.getRegionInfo(), plan.getDestination());
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-5862: - Attachment: HBASE-5862-4.patch Patch with more comments. Also added a return if reflection was un-successful as there is no need to try more. > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5829) Inconsistency between the "regions" map and the "servers" map in AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262798#comment-13262798
]
Zhihong Yu commented on HBASE-5829:
---
The latest patch is good to go.
Useless statement can be addressed elsewhere.
> Inconsistency between the "regions" map and the "servers" map in
> AssignmentManager
> --
>
> Key: HBASE-5829
> URL: https://issues.apache.org/jira/browse/HBASE-5829
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Maryann Xue
> Attachments: HBASE-5829-0.90.patch, HBASE-5829-trunk.patch
>
>
> There are occurrences in AM where this.servers is not kept consistent with
> this.regions. This might cause balancer to offline a region from the RS that
> already returned NotServingRegionException at a previous offline attempt.
> In AssignmentManager.unassign(HRegionInfo, boolean)
> try {
> // TODO: We should consider making this look more like it does for the
> // region open where we catch all throwables and never abort
> if (serverManager.sendRegionClose(server, state.getRegion(),
> versionOfClosingNode)) {
> LOG.debug("Sent CLOSE to " + server + " for region " +
> region.getRegionNameAsString());
> return;
> }
> // This never happens. Currently regionserver close always return true.
> LOG.warn("Server " + server + " region CLOSE RPC returned false for " +
> region.getRegionNameAsString());
> } catch (NotServingRegionException nsre) {
> LOG.info("Server " + server + " returned " + nsre + " for " +
> region.getRegionNameAsString());
> // Presume that master has stale data. Presume remote side just split.
> // Presume that the split message when it comes in will fix up the
> master's
> // in memory cluster state.
> } catch (Throwable t) {
> if (t instanceof RemoteException) {
> t = ((RemoteException)t).unwrapRemoteException();
> if (t instanceof NotServingRegionException) {
> if (checkIfRegionBelongsToDisabling(region)) {
> // Remove from the regionsinTransition map
> LOG.info("While trying to recover the table "
> + region.getTableNameAsString()
> + " to DISABLED state the region " + region
> + " was offlined but the table was in DISABLING state");
> synchronized (this.regionsInTransition) {
> this.regionsInTransition.remove(region.getEncodedName());
> }
> // Remove from the regionsMap
> synchronized (this.regions) {
> this.regions.remove(region);
> }
> deleteClosingOrClosedNode(region);
> }
> }
> // RS is already processing this region, only need to update the
> timestamp
> if (t instanceof RegionAlreadyInTransitionException) {
> LOG.debug("update " + state + " the timestamp.");
> state.update(state.getState());
> }
> }
> In AssignmentManager.assign(HRegionInfo, RegionState, boolean, boolean,
> boolean)
> synchronized (this.regions) {
> this.regions.put(plan.getRegionInfo(), plan.getDestination());
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
