[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806005#comment-13806005 ] Himanshu Vashishtha commented on HBASE-8741: The above 2 findbugs are coming from 7525; the test org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster passes for me on local and failure looks unrelated (timed out after 180sec). Thanks. > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.2.2.patch, > HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9715) Backport -in_memory option support for LoadTestTool from trunk
[ https://issues.apache.org/jira/browse/HBASE-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805991#comment-13805991 ] Hudson commented on HBASE-9715: --- SUCCESS: Integrated in HBase-0.94 #1185 (See [https://builds.apache.org/job/HBase-0.94/1185/]) HBASE-9715. Backport -in_memory option support for LoadTestTool (apurtell: rev 1535903) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java > Backport -in_memory option support for LoadTestTool from trunk > -- > > Key: HBASE-9715 > URL: https://issues.apache.org/jira/browse/HBASE-9715 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.13 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 0.94.14 > > Attachments: 9715.patch > > > Backport the -in_memory option for LoadTestTool from trunk so it can create > tables with IN_MEMORY set to true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805986#comment-13805986 ] Lars Hofhansl commented on HBASE-8369: -- I want that in 0.94. :) > MapReduce over snapshot files > - > > Key: HBASE-8369 > URL: https://issues.apache.org/jira/browse/HBASE-8369 > Project: HBase > Issue Type: New Feature > Components: mapreduce, snapshots >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.98.0 > > Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, > HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, > HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, > HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch > > > The idea is to add an InputFormat, which can run the mapreduce job over > snapshot files directly bypassing hbase server layer. The IF is similar in > usage to TableInputFormat, taking a Scan object from the user, but instead of > running from an online table, it runs from a table snapshot. We do one split > per region in the snapshot, and open an HRegion inside the RecordReader. A > RegionScanner is used internally for doing the scan without any HRegionServer > bits. > Users have been asking and searching for ways to run MR jobs by reading > directly from hfiles, so this allows new use cases if reading from stale data > is ok: > - Take snapshots periodically, and run MR jobs only on snapshots. > - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster > without HBase cluster. > - (Future use case) Combine snapshot data with online hbase data: Scan from > yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805974#comment-13805974
]
Gustavo Anatoly commented on HBASE-9808:
Ted,
I'm still doing the changes, to resubmit a new patch.
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805970#comment-13805970
]
Hadoop QA commented on HBASE-8741:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610421/HBASE-8741-trunk-v6.2.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7639//console
This message is automatically generated.
> Scope sequenceid to the region rather than regionserver (WAS: Mutations on
> Regions in recovery mode might have same sequenceIDs)
>
>
> Key: HBASE-8741
> URL: https://issues.apache.org/jira/browse/HBASE-8741
> Project: HBase
> Issue Type: Bug
> Components: MTTR
>Affects Versions: 0.95.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.98.0
>
> Attachments: HBASE-8741-trunk-v6.1-rebased.patch,
> HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.2.2.patch,
> HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch,
> HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch,
> HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch
>
>
> Currently, when opening a region, we find the maximum sequence ID from all
> its HFiles and then set the LogSequenceId of the log (in case the later is at
> a small value). This works good in recovered.edits case as we are not writing
> to the region until we have replayed all of its previous edits.
> With distributed log replay, if we want to enable writes while a region is
> under recovery, we need to make sure that the logSequenceId > maximum
> logSequenceId of the old regionserver. Otherwise, we might have a situation
> where new edits have same (or smaller) sequenceIds.
> We can store region level information in the WALTrailer, than this scenario
> could be avoided by:
> a) reading the trailer of the "last completed" file, i.e., last wal file
> which has a trailer and,
> b) completely reading the last wal file (this file would not have
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805949#comment-13805949 ] Hudson commented on HBASE-9837: --- SUCCESS: Integrated in HBase-TRUNK #4648 (See [https://builds.apache.org/job/HBase-TRUNK/4648/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535843) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805946#comment-13805946
]
Hudson commented on HBASE-7525:
---
SUCCESS: Integrated in HBase-TRUNK #4648 (See
[https://builds.apache.org/job/HBase-TRUNK/4648/])
HBASE-7525 A canary monitoring program specifically for regionserver
(takeshi.miao) (eclark: rev 1535846)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9080) Retain assignment should be used when re-enabling table(s)
[ https://issues.apache.org/jira/browse/HBASE-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805947#comment-13805947 ] Hudson commented on HBASE-9080: --- SUCCESS: Integrated in HBase-TRUNK #4648 (See [https://builds.apache.org/job/HBase-TRUNK/4648/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535843) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Retain assignment should be used when re-enabling table(s) > -- > > Key: HBASE-9080 > URL: https://issues.apache.org/jira/browse/HBASE-9080 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.10 >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.94.11 > > Attachments: 9080-0.94.txt, 9080-0.94-v2.txt > > > Region servers are picked in round robin fashion when a table is re-enabled. > This results in poor data locality. > We should use retain assignment when re-enabling table(s). > For initial discussion, see HBASE-6143 > Since patches for 0.94 and trunk may turn out to be very different, this JIRA > is for fixing the issue in 0.94 branch. > Thanks Lars, Elliot, Vladmir and Stack for providing opinions on this subject. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805948#comment-13805948 ] Hudson commented on HBASE-9748: --- SUCCESS: Integrated in HBase-TRUNK #4648 (See [https://builds.apache.org/job/HBase-TRUNK/4648/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535812) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9696) Master recovery ignores online merge znode
[ https://issues.apache.org/jira/browse/HBASE-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805945#comment-13805945 ] Hudson commented on HBASE-9696: --- SUCCESS: Integrated in HBase-TRUNK #4648 (See [https://builds.apache.org/job/HBase-TRUNK/4648/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535812) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java > Master recovery ignores online merge znode > -- > > Key: HBASE-9696 > URL: https://issues.apache.org/jira/browse/HBASE-9696 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0, 0.96.0 > > Attachments: 0.96-9696_v3.2.patch, 0.96-9696_v3.3.patch, > trunk-9696.patch, trunk-9696_v2.1.patch, trunk-9696_v2.patch, > trunk-9696_v3.1.patch, trunk-9696_v3.2.patch, trunk-9696_v3.3.patch, > trunk-9696_v3.patch, trunk-9696_v3.patch > > > The online merge znode uses the new region to be created. When master > restarts, the new region is still unknown if the merging is not completed. > Therefore the znode is ignored, which should not. That means the two merging > regions could be moved around. This could cause some data loss if we are not > luck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9843) Various fixes in client code
[
https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805923#comment-13805923
]
Hadoop QA commented on HBASE-9843:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610408/9843-trunk.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7638//console
This message is automatically generated.
> Various fixes in client code
>
>
> Key: HBASE-9843
> URL: https://issues.apache.org/jira/browse/HBASE-9843
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.96.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 0.98.0, 0.96.1
>
> Attachments: 9843-trunk.v2.patch, 9843-trunk.v3.patch
>
>
> This mainly fixes issues when we had "long" errors, for example a multi
> blocked when trying to obtain a lock that was finally failing after 60s.
> Previously we were trying only for 5 minutes. We now do all the tries. I've
> fixed stuff around this area to make it work.
> There is also more logs.
> I've changed the back off array. With the default pause of 100ms, even after
> 20 tries we still retry every 10s.
> I've also changed the max per RS to something minimal. If the cluster is not
> in a very good state it's less aggressive. It seems to be a better default.
> I've done two tests:
> - on a small; homogeneous cluster, I had the same performances
> - on a bigger, but heterogeneous cluster it was twice as fast.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9080) Retain assignment should be used when re-enabling table(s)
[ https://issues.apache.org/jira/browse/HBASE-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805916#comment-13805916 ] Hudson commented on HBASE-9080: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535843) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Retain assignment should be used when re-enabling table(s) > -- > > Key: HBASE-9080 > URL: https://issues.apache.org/jira/browse/HBASE-9080 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.10 >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.94.11 > > Attachments: 9080-0.94.txt, 9080-0.94-v2.txt > > > Region servers are picked in round robin fashion when a table is re-enabled. > This results in poor data locality. > We should use retain assignment when re-enabling table(s). > For initial discussion, see HBASE-6143 > Since patches for 0.94 and trunk may turn out to be very different, this JIRA > is for fixing the issue in 0.94 branch. > Thanks Lars, Elliot, Vladmir and Stack for providing opinions on this subject. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805917#comment-13805917 ] Hudson commented on HBASE-9748: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535812) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535771) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805914#comment-13805914
]
Hudson commented on HBASE-7525:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/])
HBASE-7525 A canary monitoring program specifically for regionserver
(takeshi.miao) (eclark: rev 1535846)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9838) Fix javadoc warning in ImportTsv#TsvParser ctor
[
https://issues.apache.org/jira/browse/HBASE-9838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805915#comment-13805915
]
Hudson commented on HBASE-9838:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/])
HBASE-9838 Fix javadoc warning in ImportTsv#TsvParser ctor (Ram) (tedyu: rev
1535751)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
> Fix javadoc warning in ImportTsv#TsvParser ctor
> ---
>
> Key: HBASE-9838
> URL: https://issues.apache.org/jira/browse/HBASE-9838
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 9838.txt, HBASE-9767_addendum_1.patch,
> HBASE-9767_addendum.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/7623/artifact/trunk/patchprocess/patchJavadocWarnings.txt
> :
> {code}
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java:123:
> warning - @param argument "tagSeperatorStr" is not a parameter name.
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805918#comment-13805918 ] Hudson commented on HBASE-9837: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535843) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9696) Master recovery ignores online merge znode
[ https://issues.apache.org/jira/browse/HBASE-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805913#comment-13805913 ] Hudson commented on HBASE-9696: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535812) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535771) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Master recovery ignores online merge znode > -- > > Key: HBASE-9696 > URL: https://issues.apache.org/jira/browse/HBASE-9696 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0, 0.96.0 > > Attachments: 0.96-9696_v3.2.patch, 0.96-9696_v3.3.patch, > trunk-9696.patch, trunk-9696_v2.1.patch, trunk-9696_v2.patch, > trunk-9696_v3.1.patch, trunk-9696_v3.2.patch, trunk-9696_v3.3.patch, > trunk-9696_v3.patch, trunk-9696_v3.patch > > > The online merge znode uses the new region to be created. When master > restarts, the new region is still unknown if the merging is not completed. > Therefore the znode is ignored, which should not. That means the two merging > regions could be moved around. This could cause some data loss if we are not > luck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9807) block encoder unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805912#comment-13805912
]
Hudson commented on HBASE-9807:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #811 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/811/])
HBASE-9807 block encoder unnecessarily copies the key for each reseek (larsh:
rev 1535779)
*
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
*
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
*
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> block encoder unnecessarily copies the key for each reseek
> --
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates two ByteBuffers in ScannerV2 and makes a deep copy of
> the key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805906#comment-13805906 ] Hudson commented on HBASE-9837: --- SUCCESS: Integrated in hbase-0.96 #162 (See [https://builds.apache.org/job/hbase-0.96/162/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535841) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805903#comment-13805903
]
Hudson commented on HBASE-7525:
---
SUCCESS: Integrated in hbase-0.96 #162 (See
[https://builds.apache.org/job/hbase-0.96/162/])
HBASE-7525 A canary monitoring program specifically for regionserver
(takeshi.miao) (eclark: rev 1535847)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9080) Retain assignment should be used when re-enabling table(s)
[ https://issues.apache.org/jira/browse/HBASE-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805904#comment-13805904 ] Hudson commented on HBASE-9080: --- SUCCESS: Integrated in hbase-0.96 #162 (See [https://builds.apache.org/job/hbase-0.96/162/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535841) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Retain assignment should be used when re-enabling table(s) > -- > > Key: HBASE-9080 > URL: https://issues.apache.org/jira/browse/HBASE-9080 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.10 >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.94.11 > > Attachments: 9080-0.94.txt, 9080-0.94-v2.txt > > > Region servers are picked in round robin fashion when a table is re-enabled. > This results in poor data locality. > We should use retain assignment when re-enabling table(s). > For initial discussion, see HBASE-6143 > Since patches for 0.94 and trunk may turn out to be very different, this JIRA > is for fixing the issue in 0.94 branch. > Thanks Lars, Elliot, Vladmir and Stack for providing opinions on this subject. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805905#comment-13805905 ] Hudson commented on HBASE-9748: --- SUCCESS: Integrated in hbase-0.96 #162 (See [https://builds.apache.org/job/hbase-0.96/162/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535815) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9696) Master recovery ignores online merge znode
[ https://issues.apache.org/jira/browse/HBASE-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805902#comment-13805902 ] Hudson commented on HBASE-9696: --- SUCCESS: Integrated in hbase-0.96 #162 (See [https://builds.apache.org/job/hbase-0.96/162/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535815) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java > Master recovery ignores online merge znode > -- > > Key: HBASE-9696 > URL: https://issues.apache.org/jira/browse/HBASE-9696 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0, 0.96.0 > > Attachments: 0.96-9696_v3.2.patch, 0.96-9696_v3.3.patch, > trunk-9696.patch, trunk-9696_v2.1.patch, trunk-9696_v2.patch, > trunk-9696_v3.1.patch, trunk-9696_v3.2.patch, trunk-9696_v3.3.patch, > trunk-9696_v3.patch, trunk-9696_v3.patch > > > The online merge znode uses the new region to be created. When master > restarts, the new region is still unknown if the merging is not completed. > Therefore the znode is ignored, which should not. That means the two merging > regions could be moved around. This could cause some data loss if we are not > luck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9846) Integration test and LoadTestTool support for cell ACLs
[ https://issues.apache.org/jira/browse/HBASE-9846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9846: -- Issue Type: Sub-task (was: Improvement) Parent: HBASE-6222 > Integration test and LoadTestTool support for cell ACLs > --- > > Key: HBASE-9846 > URL: https://issues.apache.org/jira/browse/HBASE-9846 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.0 > > > Cell level ACLs should have an integration test and LoadTestTool support. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9846) Integration test and LoadTestTool support for cell ACLs
[ https://issues.apache.org/jira/browse/HBASE-9846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805901#comment-13805901 ] Andrew Purtell commented on HBASE-9846: --- What ACLs to include on what cells is dependent on test scenario. Might make sense to do this building on or in conjunction with HBASE-9824 so we can reach in and replace the KV generator. > Integration test and LoadTestTool support for cell ACLs > --- > > Key: HBASE-9846 > URL: https://issues.apache.org/jira/browse/HBASE-9846 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.0 > > > Cell level ACLs should have an integration test and LoadTestTool support. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9846) Integration test and LoadTestTool support for cell ACLs
Andrew Purtell created HBASE-9846: - Summary: Integration test and LoadTestTool support for cell ACLs Key: HBASE-9846 URL: https://issues.apache.org/jira/browse/HBASE-9846 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.0 Cell level ACLs should have an integration test and LoadTestTool support. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-9715) Backport -in_memory option support for LoadTestTool from trunk
[ https://issues.apache.org/jira/browse/HBASE-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-9715. --- Resolution: Fixed Fix Version/s: 0.94.14 Forgot to commit this earlier. > Backport -in_memory option support for LoadTestTool from trunk > -- > > Key: HBASE-9715 > URL: https://issues.apache.org/jira/browse/HBASE-9715 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.13 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 0.94.14 > > Attachments: 9715.patch > > > Backport the -in_memory option for LoadTestTool from trunk so it can create > tables with IN_MEMORY set to true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805893#comment-13805893 ] Himanshu Vashishtha commented on HBASE-8741: mvn clean test -Dtest=Test*HLog*,Test*WAL* -Dhadoop.profile=2.0 passes. > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.2.2.patch, > HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9845) Add MR support for cell ACLs
Andrew Purtell created HBASE-9845: - Summary: Add MR support for cell ACLs Key: HBASE-9845 URL: https://issues.apache.org/jira/browse/HBASE-9845 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.0 We should support adding cell ACLs through TableOutputFormat and HFileOutputFormat. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-trunk-v6.2.2.patch Fixed the find bugs warning; it is just an api usage change and doesn't make any changes functionality wise. > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.2.2.patch, > HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8543) fix coverage org.apache.hadoop.hbase.rest.client
[
https://issues.apache.org/jira/browse/HBASE-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805864#comment-13805864
]
Enis Soztutar commented on HBASE-8543:
--
A couple of comments:
- @Category(SmallTests.class) seems ok since we are not booting a cluster.
- Can you copy paste the license headers for the new files from some other
java file. We are using the version which is limited to 80 chars per line
mostly.
- TestRemoteHTableRetries.MAX_TIME should also be MAX_TIME = SLEEP_TIME *
(RETRIES - 1) ?
- We should not need this:
{code}
-assertEquals(1, results[0].size());
-assertEquals(3, results[1].size());
+assertEquals(2, results[0].size());
+assertEquals(4, results[1].size());
{code}
It seems that the individual test methods are not independent. Can you please
make it so that each method uses it's own table, so that the test results won't
depend on run order.
Otherwise looks good.
> fix coverage org.apache.hadoop.hbase.rest.client
>
>
> Key: HBASE-8543
> URL: https://issues.apache.org/jira/browse/HBASE-8543
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.94.8, 0.95.2
>Reporter: Aleksey Gorshkov
>Assignee: Andrey Klochkov
> Attachments: HBASE-8543-0.94--N2.patch, HBASE-8543-0.94.patch,
> HBASE-8543-trunk--N2.patch, HBASE-8543-trunk--n3.patch,
> HBASE-8543-trunk--n4.patch, HBASE-8543-trunk--n5.patch, HBASE-8543-trunk.patch
>
>
> fix coverage org.apache.hadoop.hbase.rest.client
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805858#comment-13805858 ] Himanshu Vashishtha commented on HBASE-8741: Hmm, wonder why I didn't see on my local. The three differences are: 1) RV org.apache.hadoop.hbase.tool.Canary$RegionMonitor.generateMonitorTables(String[]) forgets to throw new org.apache.hadoop.hbase.TableNotFoundException(String) 2) WMI Method org.apache.hadoop.hbase.tool.Canary$RegionServerMonitor.doFilterRegionServerByName(Map) makes inefficient use of keySet iterator instead of entrySet iterator 3) Method org.apache.hadoop.hbase.regionserver.wal.FSHLog.isAllRegionsFlushed(Map, Map, Map) makes inefficient use of keySet iterator instead of entrySet iterator The last one is related to the patch, the other two looks from HBASE-7525. I changed the patch to make findbugs happy. Unit testing it, will upload it shortly. > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.patch, > HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, > HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, > HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805857#comment-13805857
]
Hadoop QA commented on HBASE-9808:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610386/HBASE-9808.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7637//console
This message is automatically generated.
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
[jira] [Updated] (HBASE-7544) Transparent table/CF encryption
[
https://issues.apache.org/jira/browse/HBASE-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-7544:
--
Attachment: 7544.patch
Changes in latest patch:
- Support random HFile keying.
- Check the results of key unwrapping with a CRC.
- Introduce a new configuration value for holding an alternate master key
alias. If the current master key fails to unwrap, and an alternate is
available, try it. Allows for gradual master key rotation.
- Plumb configuration down to the HFileV3 reader so we can avoid parsing the
site file when creating a reader.
- Update KeyProviderForTesting for verifying configuration plumbing.
- Additional test cases and changes to existing test cases to confirm new
functionality.
No more changes are planned.
> Transparent table/CF encryption
> ---
>
> Key: HBASE-7544
> URL: https://issues.apache.org/jira/browse/HBASE-7544
> Project: HBase
> Issue Type: New Feature
> Components: HFile, io
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0
>
> Attachments: 7544p1.patch, 7544p1.patch, 7544p2.patch, 7544p2.patch,
> 7544p3.patch, 7544p3.patch, 7544p4.patch, 7544.patch, 7544.patch, 7544.patch,
> historical-7544.patch, historical-7544.pdf, historical-shell.patch
>
>
> Introduce transparent encryption of HBase on disk data.
> Depends on a separate contribution of an encryption codec framework to Hadoop
> core and an AES-NI (native code) codec. This is work done in the context of
> MAPREDUCE-4491 but I'd gather there will be additional JIRAs for common and
> HDFS parts of it.
> Requirements:
> - Transparent encryption at the CF or table level
> - Protect against all data leakage from files at rest
> - Two-tier key architecture for consistency with best practices for this
> feature in the RDBMS world
> - Built-in key management
> - Flexible and non-intrusive key rotation
> - Mechanisms not exposed to or modifiable by users
> - Hardware security module integration (via Java KeyStore)
> - HBCK support for transparently encrypted files (+ plugin architecture for
> HBCK)
> Additional goals:
> - Shell support for administrative functions
> - Avoid performance impact for the null crypto codec case
> - Play nicely with other changes underway: in HFile, block coding, etc.
> We're aiming for rough parity with Oracle's transparent tablespace encryption
> feature, described in
> http://www.oracle.com/technetwork/database/owp-security-advanced-security-11gr-133411.pdf
> as
> {quote}
> “Transparent Data Encryption uses a 2-tier key architecture for flexible and
> non-intrusive key rotation and least operational and performance impact: Each
> application table with at least one encrypted column has its own table key,
> which is applied to all encrypted columns in that table. Equally, each
> encrypted tablespace has its own tablespace key. Table keys are stored in the
> data dictionary of the database, while tablespace keys are stored in the
> header of the tablespace and additionally, the header of each underlying OS
> file that makes up the tablespace. Each of these keys is encrypted with the
> TDE master encryption key, which is stored outside of the database in an
> external security module: either the Oracle Wallet (a PKCS#12 formatted file
> that is encrypted using a passphrase supplied either by the designated
> security administrator or DBA during setup), or a Hardware Security Module
> (HSM) device for higher assurance […]”
> {quote}
> Further design details forthcoming in a design document and patch as soon as
> we have all of the clearances in place.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805850#comment-13805850 ] Hudson commented on HBASE-9748: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #102 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/102/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535815) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535789) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9807) block encoder unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805846#comment-13805846
]
Hudson commented on HBASE-9807:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #102 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/102/])
HBASE-9807 block encoder unnecessarily copies the key for each reseek (larsh:
rev 1535782)
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
*
/hbase/branches/0.96/hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> block encoder unnecessarily copies the key for each reseek
> --
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates two ByteBuffers in ScannerV2 and makes a deep copy of
> the key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9080) Retain assignment should be used when re-enabling table(s)
[ https://issues.apache.org/jira/browse/HBASE-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805849#comment-13805849 ] Hudson commented on HBASE-9080: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #102 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/102/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535841) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Retain assignment should be used when re-enabling table(s) > -- > > Key: HBASE-9080 > URL: https://issues.apache.org/jira/browse/HBASE-9080 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.10 >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.94.11 > > Attachments: 9080-0.94.txt, 9080-0.94-v2.txt > > > Region servers are picked in round robin fashion when a table is re-enabled. > This results in poor data locality. > We should use retain assignment when re-enabling table(s). > For initial discussion, see HBASE-6143 > Since patches for 0.94 and trunk may turn out to be very different, this JIRA > is for fixing the issue in 0.94 branch. > Thanks Lars, Elliot, Vladmir and Stack for providing opinions on this subject. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805851#comment-13805851 ] Hudson commented on HBASE-9837: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #102 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/102/]) HBASE-9837 Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)' (tedyu: rev 1535841) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805848#comment-13805848
]
Hudson commented on HBASE-7525:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #102 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/102/])
HBASE-7525 A canary monitoring program specifically for regionserver
(takeshi.miao) (eclark: rev 1535847)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9696) Master recovery ignores online merge znode
[ https://issues.apache.org/jira/browse/HBASE-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805847#comment-13805847 ] Hudson commented on HBASE-9696: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #102 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/102/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 - ADDENDUM (jxiang: rev 1535815) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535789) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Master recovery ignores online merge znode > -- > > Key: HBASE-9696 > URL: https://issues.apache.org/jira/browse/HBASE-9696 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0, 0.96.0 > > Attachments: 0.96-9696_v3.2.patch, 0.96-9696_v3.3.patch, > trunk-9696.patch, trunk-9696_v2.1.patch, trunk-9696_v2.patch, > trunk-9696_v3.1.patch, trunk-9696_v3.2.patch, trunk-9696_v3.3.patch, > trunk-9696_v3.patch, trunk-9696_v3.patch > > > The online merge znode uses the new region to be created. When master > restarts, the new region is still unknown if the merging is not completed. > Therefore the znode is ignored, which should not. That means the two merging > regions could be moved around. This could cause some data loss if we are not > luck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805839#comment-13805839 ] Nicolas Liochon commented on HBASE-9843: v3 with: - check for null when throwing the exception - Elliott's comment taken into account - I haven't found the issues for the warning, I fixed two trivial ones to decrease the counter. > Various fixes in client code > > > Key: HBASE-9843 > URL: https://issues.apache.org/jira/browse/HBASE-9843 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.98.0, 0.96.1 > > Attachments: 9843-trunk.v2.patch, 9843-trunk.v3.patch > > > This mainly fixes issues when we had "long" errors, for example a multi > blocked when trying to obtain a lock that was finally failing after 60s. > Previously we were trying only for 5 minutes. We now do all the tries. I've > fixed stuff around this area to make it work. > There is also more logs. > I've changed the back off array. With the default pause of 100ms, even after > 20 tries we still retry every 10s. > I've also changed the max per RS to something minimal. If the cluster is not > in a very good state it's less aggressive. It seems to be a better default. > I've done two tests: > - on a small; homogeneous cluster, I had the same performances > - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9843: --- Status: Open (was: Patch Available) > Various fixes in client code > > > Key: HBASE-9843 > URL: https://issues.apache.org/jira/browse/HBASE-9843 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.98.0, 0.96.1 > > Attachments: 9843-trunk.v2.patch, 9843-trunk.v3.patch > > > This mainly fixes issues when we had "long" errors, for example a multi > blocked when trying to obtain a lock that was finally failing after 60s. > Previously we were trying only for 5 minutes. We now do all the tries. I've > fixed stuff around this area to make it work. > There is also more logs. > I've changed the back off array. With the default pause of 100ms, even after > 20 tries we still retry every 10s. > I've also changed the max per RS to something minimal. If the cluster is not > in a very good state it's less aggressive. It seems to be a better default. > I've done two tests: > - on a small; homogeneous cluster, I had the same performances > - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9843: --- Status: Patch Available (was: Open) > Various fixes in client code > > > Key: HBASE-9843 > URL: https://issues.apache.org/jira/browse/HBASE-9843 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.98.0, 0.96.1 > > Attachments: 9843-trunk.v2.patch, 9843-trunk.v3.patch > > > This mainly fixes issues when we had "long" errors, for example a multi > blocked when trying to obtain a lock that was finally failing after 60s. > Previously we were trying only for 5 minutes. We now do all the tries. I've > fixed stuff around this area to make it work. > There is also more logs. > I've changed the back off array. With the default pause of 100ms, even after > 20 tries we still retry every 10s. > I've also changed the max per RS to something minimal. If the cluster is not > in a very good state it's less aggressive. It seems to be a better default. > I've done two tests: > - on a small; homogeneous cluster, I had the same performances > - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9843: --- Attachment: 9843-trunk.v3.patch > Various fixes in client code > > > Key: HBASE-9843 > URL: https://issues.apache.org/jira/browse/HBASE-9843 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.98.0, 0.96.1 > > Attachments: 9843-trunk.v2.patch, 9843-trunk.v3.patch > > > This mainly fixes issues when we had "long" errors, for example a multi > blocked when trying to obtain a lock that was finally failing after 60s. > Previously we were trying only for 5 minutes. We now do all the tries. I've > fixed stuff around this area to make it work. > There is also more logs. > I've changed the back off array. With the default pause of 100ms, even after > 20 tries we still retry every 10s. > I've also changed the max per RS to something minimal. If the cluster is not > in a very good state it's less aggressive. It seems to be a better default. > I've done two tests: > - on a small; homogeneous cluster, I had the same performances > - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9792) Region states should update last assignments when a region is opened.
[ https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805812#comment-13805812 ] Sergey Shelukhin commented on HBASE-9792: - Patch makes sense on code level, although I don;t think I fully understand OFFLINE-related changes > Region states should update last assignments when a region is opened. > - > > Key: HBASE-9792 > URL: https://issues.apache.org/jira/browse/HBASE-9792 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-9792.patch, trunk-9792_v2.patch > > > Currently, we update a region's last assignment region server when the region > is online. We should do this sooner, when the region is moved to OPEN state. > CM could kill this region server before we delete the znode and online the > region. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9841) Provide tracking URL for cluster which is immune to master failover
[ https://issues.apache.org/jira/browse/HBASE-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805808#comment-13805808 ] Ted Yu commented on HBASE-9841: --- MasterAddressTracker has method, getMasterAddress(). However MasterAddressTracker is marked private. Which public class / method can provide this information ? > Provide tracking URL for cluster which is immune to master failover > --- > > Key: HBASE-9841 > URL: https://issues.apache.org/jira/browse/HBASE-9841 > Project: HBase > Issue Type: Task >Reporter: Ted Yu > > Currently each master provides web UI whose URL would not survive master > failure. > For downstream project, it is desirable to have a tracking URL for the > cluster which is always functional regardless which machine runs the active > master. > One solution is to run light weight web server on designated server which > serves the URL on well known port. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9843) Various fixes in client code
[
https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805802#comment-13805802
]
Hadoop QA commented on HBASE-9843:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610376/9843-trunk.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7636//console
This message is automatically generated.
> Various fixes in client code
>
>
> Key: HBASE-9843
> URL: https://issues.apache.org/jira/browse/HBASE-9843
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.96.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 0.98.0, 0.96.1
>
> Attachments: 9843-trunk.v2.patch
>
>
> This mainly fixes issues when we had "long" errors, for example a multi
> blocked when trying to obtain a lock that was finally failing after 60s.
> Previously we were trying only for 5 minutes. We now do all the tries. I've
> fixed stuff around this area to make it work.
> There is also more logs.
> I've changed the back off array. With the default pause of 100ms, even after
> 20 tries we still retry every 10s.
> I've also changed the max per RS to something minimal. If the cluster is not
> in a very good state it's less aggressive. It seems to be a better default.
> I've done two tests:
> - on a small; homogeneous cluster, I had the same performances
> - on a bigger, but heterogeneous cluster it was twice as fast.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805796#comment-13805796
]
Ted Yu edited comment on HBASE-9836 at 10/25/13 11:07 PM:
--
Another approach would be similar to the one used by
TestIOFencing#testFencingAroundCompactionAfterWALSync where HRegion is
subclassed and CountDownLatch is added to serve as the barrier.
Would that be acceptable ?
was (Author: yuzhih...@gmail.com):
Another approach would be similar to the one used by
TestIOFencing#testFencingAroundCompactionAfterWALSync where HRegion is
subclassed where CountDownLatch is added to serve as the barrier.
Would that be acceptable ?
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt, 9836-v4.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 2 store
> files, 0 compacting, 2 eligible, 10 blocking
> 2013-10-24 18:08:10,547 DEBUG [pool-1-thread-1] catalog.CatalogTracker(209):
> Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4be179
> 2013-10-24 18:08:10,549 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.ExploringCompactionPolicy(112): Exploring compaction algorithm
> has selected 2 files of size 1999 starting at candidate #0 after considering
> 1 permutations with 1 in ratio
> 2013-10-24 18:08:10,551 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(1329):
> e96920e43ea374ba1bd559df115870cf - A: Initiating major compaction
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> regionserver.HRegion(1294): Starting compaction on A in region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(982):
> Starting compaction of 2 file(s) in A of
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> into
> tmpdir=hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/.tmp,
> totalSize=2.0k
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/44f87b94732149c08f20bdba00dd7140,
> keycount=1, bloomtype=ROW, size=992.0, encoding=NONE, seqNum=3,
> earliestPutTs=1382638089528
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/0b2e580cbda246718bbf64c21e81cd18,
> keycount=1, bloomtype=ROW, size=1007.0, encoding=NONE, seqNum=5,
> earliestPutTs=1382638090053
> 2013-10-24 18:08:10,564 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] util.FSUtils(305): DFS
> Client does not support most favored nodes create; using default create
> ...
[jira] [Commented] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805796#comment-13805796
]
Ted Yu commented on HBASE-9836:
---
Another approach would be similar to the one used by
TestIOFencing#testFencingAroundCompactionAfterWALSync where HRegion is
subclassed where CountDownLatch is added to serve as the barrier.
Would that be acceptable ?
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt, 9836-v4.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 2 store
> files, 0 compacting, 2 eligible, 10 blocking
> 2013-10-24 18:08:10,547 DEBUG [pool-1-thread-1] catalog.CatalogTracker(209):
> Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4be179
> 2013-10-24 18:08:10,549 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.ExploringCompactionPolicy(112): Exploring compaction algorithm
> has selected 2 files of size 1999 starting at candidate #0 after considering
> 1 permutations with 1 in ratio
> 2013-10-24 18:08:10,551 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(1329):
> e96920e43ea374ba1bd559df115870cf - A: Initiating major compaction
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> regionserver.HRegion(1294): Starting compaction on A in region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(982):
> Starting compaction of 2 file(s) in A of
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> into
> tmpdir=hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/.tmp,
> totalSize=2.0k
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/44f87b94732149c08f20bdba00dd7140,
> keycount=1, bloomtype=ROW, size=992.0, encoding=NONE, seqNum=3,
> earliestPutTs=1382638089528
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/0b2e580cbda246718bbf64c21e81cd18,
> keycount=1, bloomtype=ROW, size=1007.0, encoding=NONE, seqNum=5,
> earliestPutTs=1382638090053
> 2013-10-24 18:08:10,564 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] util.FSUtils(305): DFS
> Client does not support most favored nodes create; using default create
> ...
> Potentially hanging thread: RS:0;asf002:58174-smallCompactions-1382638090545
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.ipc.Client.call(Client.java:1099)
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> $Proxy9.complete(Unknown
[jira] [Comment Edited] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805748#comment-13805748 ] Enis Soztutar edited comment on HBASE-8369 at 10/25/13 10:11 PM: - Uploaded the slides about this issue here: http://www.slideshare.net/enissoz/mapreduce-over-snapshots. It also contains some numbers for performaance comparison. Here are the raw numbers (in MB/s) ||Data size|| 6.6 G|| 13.2G|| 19.8 G||26.4 G|| |StoreFileCount per region| 3| 6| 9| 12| |Scan(MB/s) |8.2|7.6|11.2 |7.2| |SnapshotScan(MB/s) |60.8 |59.5 |55.3 |46.7| |ScanMR(MB/s) |75.9 |80.4 |82.3 |140.7| |SnapshotScanMR(MB/s) |198.6 |275.6 |311.6 |329.4| Main takeaway, seems to be the single scanner speeds improve 5-6x, from 11MB/s to 55MB/s. That is also half of raw disk speed (for a single disk). Do not read much into MR test speed improvements when store file increases. That is due to job launch costs taking relatively smaller percentage when data sizes increase. was (Author: enis): Uploaded the slides about this issue here: http://www.slideshare.net/enissoz/mapreduce-over-snapshots. It also contains some numbers for performaance comparison. Here are the raw numbers (in MB/s) ||Data size|| 6.6 G|| 13.2G|| 19.8 G||26.4 G|| |StoreFileCount per region| 3| 6| 9| 12| |Scan |8.2|7.6|11.2 |7.2| |SnapshotScan |60.8 |59.5 |55.3 |46.7| |ScanMR |75.9 |80.4 |82.3 |140.7| |SnapshotScanMR |198.6 |275.6 |311.6 |329.4| Main takeaway, seems to be the single scanner speeds improve 5-6x, from 11MB/s to 55MB/s. That is also half of raw disk speed (for a single disk). Do not read much into MR test speed improvements when store file increases. That is due to job launch costs taking relatively smaller percentage when data sizes increase. > MapReduce over snapshot files > - > > Key: HBASE-8369 > URL: https://issues.apache.org/jira/browse/HBASE-8369 > Project: HBase > Issue Type: New Feature > Components: mapreduce, snapshots >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.98.0 > > Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, > HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, > HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, > HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch > > > The idea is to add an InputFormat, which can run the mapreduce job over > snapshot files directly bypassing hbase server layer. The IF is similar in > usage to TableInputFormat, taking a Scan object from the user, but instead of > running from an online table, it runs from a table snapshot. We do one split > per region in the snapshot, and open an HRegion inside the RecordReader. A > RegionScanner is used internally for doing the scan without any HRegionServer > bits. > Users have been asking and searching for ways to run MR jobs by reading > directly from hfiles, so this allows new use cases if reading from stale data > is ok: > - Take snapshots periodically, and run MR jobs only on snapshots. > - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster > without HBase cluster. > - (Future use case) Combine snapshot data with online hbase data: Scan from > yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805750#comment-13805750
]
Gustavo Anatoly commented on HBASE-9808:
Ok, I will fix it tonight and resubmit the patch.
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805748#comment-13805748 ] Enis Soztutar commented on HBASE-8369: -- Uploaded the slides about this issue here: http://www.slideshare.net/enissoz/mapreduce-over-snapshots. It also contains some numbers for performaance comparison. Here are the raw numbers (in MB/s) ||Data size|| 6.6 G|| 13.2G|| 19.8 G||26.4 G|| |StoreFileCount per region| 3| 6| 9| 12| |Scan |8.2|7.6|11.2 |7.2| |SnapshotScan |60.8 |59.5 |55.3 |46.7| |ScanMR |75.9 |80.4 |82.3 |140.7| |SnapshotScanMR |198.6 |275.6 |311.6 |329.4| Main takeaway, seems to be the single scanner speeds improve 5-6x, from 11MB/s to 55MB/s. That is also half of raw disk speed (for a single disk). Do not read much into MR test speed improvements when store file increases. That is due to job launch costs taking relatively smaller percentage when data sizes increase. > MapReduce over snapshot files > - > > Key: HBASE-8369 > URL: https://issues.apache.org/jira/browse/HBASE-8369 > Project: HBase > Issue Type: New Feature > Components: mapreduce, snapshots >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.98.0 > > Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, > HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, > HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, > HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch > > > The idea is to add an InputFormat, which can run the mapreduce job over > snapshot files directly bypassing hbase server layer. The IF is similar in > usage to TableInputFormat, taking a Scan object from the user, but instead of > running from an online table, it runs from a table snapshot. We do one split > per region in the snapshot, and open an HRegion inside the RecordReader. A > RegionScanner is used internally for doing the scan without any HRegionServer > bits. > Users have been asking and searching for ways to run MR jobs by reading > directly from hfiles, so this allows new use cases if reading from stale data > is ok: > - Take snapshots periodically, and run MR jobs only on snapshots. > - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster > without HBase cluster. > - (Future use case) Combine snapshot data with online hbase data: Scan from > yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805745#comment-13805745 ] Ted Yu commented on HBASE-8741: --- When I compared https://builds.apache.org/job/PreCommit-HBASE-Build/7635/artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html with https://builds.apache.org/job/PreCommit-HBASE-Build/7629/artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html, I saw 1 more Correctness Warning and 2 more Performance Warnings. > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.patch, > HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, > HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, > HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805740#comment-13805740
]
Hadoop QA commented on HBASE-8741:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610371/HBASE-8741-trunk-v6.2.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7635//console
This message is automatically generated.
> Scope sequenceid to the region rather than regionserver (WAS: Mutations on
> Regions in recovery mode might have same sequenceIDs)
>
>
> Key: HBASE-8741
> URL: https://issues.apache.org/jira/browse/HBASE-8741
> Project: HBase
> Issue Type: Bug
> Components: MTTR
>Affects Versions: 0.95.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.98.0
>
> Attachments: HBASE-8741-trunk-v6.1-rebased.patch,
> HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.patch,
> HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch,
> HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch,
> HBASE-8741-v5-again.patch, HBASE-8741-v5.patch
>
>
> Currently, when opening a region, we find the maximum sequence ID from all
> its HFiles and then set the LogSequenceId of the log (in case the later is at
> a small value). This works good in recovered.edits case as we are not writing
> to the region until we have replayed all of its previous edits.
> With distributed log replay, if we want to enable writes while a region is
> under recovery, we need to make sure that the logSequenceId > maximum
> logSequenceId of the old regionserver. Otherwise, we might have a situation
> where new edits have same (or smaller) sequenceIds.
> We can store region level information in the WALTrailer, than this scenario
> could be avoided by:
> a) reading the trailer of the "last completed" file, i.e., last wal file
> which has a trailer and,
> b) completely reading the last wal file (this file would not have the
> trailer, so it needs to be read completely).
> In future, if we switch to multi wal file, we could read the trailer
[jira] [Comment Edited] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805733#comment-13805733
]
Ted Yu edited comment on HBASE-9808 at 10/25/13 9:52 PM:
-
Looking at svn history for
src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java , it was
created in HBASE-2542
I went through the patch once.
{code}
-import org.apache.hadoop.hbase.rest.client.Client;
...
-import org.apache.hadoop.hbase.rest.client.RemoteAdmin;
-import org.apache.hadoop.hbase.rest.client.RemoteHTable;
{code}
The above classes from rest package should be used, instead of
org.apache.hadoop.hbase.client.HBaseAdmin, etc.
was (Author: yuzhih...@gmail.com):
Looking at svn history for
src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java , it was
created in HBASE-2542
I went through the patch once. Looks Okay.
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805733#comment-13805733
]
Ted Yu commented on HBASE-9808:
---
Looking at svn history for
src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java , it was
created in HBASE-2542
I went through the patch once. Looks Okay.
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9808:
--
Status: Patch Available (was: Open)
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805718#comment-13805718
]
Enis Soztutar commented on HBASE-8143:
--
For MR especially, deployments usually add the whole hadoop conf dir to the
classpath, no? I think bigtop also does this. In this case, we would like to
take the value from hdfs-site.
bq. What if it is not the default and still too large or if the default value
changes (we can't read hdfs-side configs)?
Luckily, dfs.client.read.shortcircuit.buffer.size is not set in
hdfs-default.xml. From the FSUtils code, can we call conf.setIfUnset(), would
that work?
> HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
> --
>
> Key: HBASE-8143
> URL: https://issues.apache.org/jira/browse/HBASE-8143
> Project: HBase
> Issue Type: Bug
> Components: hadoop2
>Affects Versions: 0.98.0, 0.94.7, 0.95.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: 8143doc.txt, 8143.hbase-default.xml.txt,
> OpenFileTest.java
>
>
> We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that
> the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some
> time, this causes OOM for the RSs.
> Upon further investigation, I've found out that we end up with 200 regions,
> each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal
> allocates DirectBuffers, which is unlike HDFS 1 where there is no direct
> buffer allocation.
> It seems that there is no guards against the memory used by local buffers in
> hdfs 2, and having a large number of open files causes multiple GB of memory
> to be consumed from the RS process.
> This issue is to further investigate what is going on. Whether we can limit
> the memory usage in HDFS, or HBase, and/or document the setup.
> Possible mitigation scenarios are:
> - Turn off SSR for Hadoop 2
> - Ensure that there is enough unallocated memory for the RS based on
> expected # of store files
> - Ensure that there is lower number of regions per region server (hence
> number of open files)
> Stack trace:
> {code}
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at
> org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:315)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1261)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
> a
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805711#comment-13805711
]
Gustavo Anatoly commented on HBASE-9808:
Ted,
I finished the task, but I have a question. Why we have two classes doing the
same behavior? And could you please review the patch? I did some tests, but you
have others test that I should run?
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo Anatoly updated HBASE-9808:
---
Attachment: HBASE-9808.patch
> org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
> org.apache.hadoop.hbase.PerformanceEvaluation
>
>
> Key: HBASE-9808
> URL: https://issues.apache.org/jira/browse/HBASE-9808
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
> Attachments: HBASE-9808.patch
>
>
> Here is list of JIRAs whose fixes might have gone into
> rest.PerformanceEvaluation :
> {code}
>
> r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
> HBASE-9663 PerformanceEvaluation does not properly honor specified table name
> parameter
>
> r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
> HBASE-9662 PerformanceEvaluation input do not handle tags properties
>
> r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
> HBASE-8496 - Implement tags and the internals of how a tag should look like
> (Ram)
>
> r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
> HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
> dependency to MiniDFSCluster
>
> r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
> HBASE-9521 clean clearBufferOnFail behavior and deprecate it
>
> r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
> HBASE-9330 Refactor PE to create HTable the correct way
> {code}
> Long term, we may consider consolidating the two PerformanceEvaluation
> classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805703#comment-13805703 ] Ted Yu commented on HBASE-8397: --- Ran TestMasterStatistics using security profile on Mac but didn't reproduce the test failure. > improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics > (0.94) > --- > > Key: HBASE-8397 > URL: https://issues.apache.org/jira/browse/HBASE-8397 > Project: HBase > Issue Type: Test >Affects Versions: 0.94.8 >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky > Fix For: 0.94.14 > > Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, > HBASE-8397-0.94--N6.patch > > > The patch suggests new test that elevates unit-test coverage of package > org.apache.hadoop.hbase.master.metrics . > The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9843) Various fixes in client code
[
https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805678#comment-13805678
]
Elliott Clark commented on HBASE-9843:
--
bq.+ Runnable runnable = Trace.wrap("#" + id + ",
AsyncProcess.sendMultiAction", new Runnable() {
The description in the trace should always stay the same so that aggregation of
the traces by name is easy. We can add the async processes id in an annotation
if that is needed.
> Various fixes in client code
>
>
> Key: HBASE-9843
> URL: https://issues.apache.org/jira/browse/HBASE-9843
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.96.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 0.98.0, 0.96.1
>
> Attachments: 9843-trunk.v2.patch
>
>
> This mainly fixes issues when we had "long" errors, for example a multi
> blocked when trying to obtain a lock that was finally failing after 60s.
> Previously we were trying only for 5 minutes. We now do all the tries. I've
> fixed stuff around this area to make it work.
> There is also more logs.
> I've changed the back off array. With the default pause of 100ms, even after
> 20 tries we still retry every 10s.
> I've also changed the max per RS to something minimal. If the cluster is not
> in a very good state it's less aggressive. It seems to be a better default.
> I've done two tests:
> - on a small; homogeneous cluster, I had the same performances
> - on a bigger, but heterogeneous cluster it was twice as fast.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805668#comment-13805668 ] Hudson commented on HBASE-9748: --- SUCCESS: Integrated in HBase-TRUNK #4647 (See [https://builds.apache.org/job/HBase-TRUNK/4647/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535771) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9696) Master recovery ignores online merge znode
[ https://issues.apache.org/jira/browse/HBASE-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805667#comment-13805667 ] Hudson commented on HBASE-9696: --- SUCCESS: Integrated in HBase-TRUNK #4647 (See [https://builds.apache.org/job/HBase-TRUNK/4647/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535771) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Master recovery ignores online merge znode > -- > > Key: HBASE-9696 > URL: https://issues.apache.org/jira/browse/HBASE-9696 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0, 0.96.0 > > Attachments: 0.96-9696_v3.2.patch, 0.96-9696_v3.3.patch, > trunk-9696.patch, trunk-9696_v2.1.patch, trunk-9696_v2.patch, > trunk-9696_v3.1.patch, trunk-9696_v3.2.patch, trunk-9696_v3.3.patch, > trunk-9696_v3.patch, trunk-9696_v3.patch > > > The online merge znode uses the new region to be created. When master > restarts, the new region is still unknown if the merging is not completed. > Therefore the znode is ignored, which should not. That means the two merging > regions could be moved around. This could cause some data loss if we are not > luck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9807) block encoder unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805666#comment-13805666
]
Hudson commented on HBASE-9807:
---
SUCCESS: Integrated in HBase-TRUNK #4647 (See
[https://builds.apache.org/job/HBase-TRUNK/4647/])
HBASE-9807 block encoder unnecessarily copies the key for each reseek (larsh:
rev 1535779)
*
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
*
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
*
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> block encoder unnecessarily copies the key for each reseek
> --
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates two ByteBuffers in ScannerV2 and makes a deep copy of
> the key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805664#comment-13805664
]
Hadoop QA commented on HBASE-9836:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610369/9836-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7634//console
This message is automatically generated.
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt, 9836-v4.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 2 store
> files, 0
[jira] [Commented] (HBASE-9696) Master recovery ignores online merge znode
[ https://issues.apache.org/jira/browse/HBASE-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805656#comment-13805656 ] Hudson commented on HBASE-9696: --- FAILURE: Integrated in hbase-0.96 #161 (See [https://builds.apache.org/job/hbase-0.96/161/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535789) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Master recovery ignores online merge znode > -- > > Key: HBASE-9696 > URL: https://issues.apache.org/jira/browse/HBASE-9696 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0, 0.96.0 > > Attachments: 0.96-9696_v3.2.patch, 0.96-9696_v3.3.patch, > trunk-9696.patch, trunk-9696_v2.1.patch, trunk-9696_v2.patch, > trunk-9696_v3.1.patch, trunk-9696_v3.2.patch, trunk-9696_v3.3.patch, > trunk-9696_v3.patch, trunk-9696_v3.patch > > > The online merge znode uses the new region to be created. When master > restarts, the new region is still unknown if the merging is not completed. > Therefore the znode is ignored, which should not. That means the two merging > regions could be moved around. This could cause some data loss if we are not > luck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9807) block encoder unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805655#comment-13805655
]
Hudson commented on HBASE-9807:
---
FAILURE: Integrated in hbase-0.96 #161 (See
[https://builds.apache.org/job/hbase-0.96/161/])
HBASE-9807 block encoder unnecessarily copies the key for each reseek (larsh:
rev 1535782)
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
*
/hbase/branches/0.96/hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> block encoder unnecessarily copies the key for each reseek
> --
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates two ByteBuffers in ScannerV2 and makes a deep copy of
> the key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805657#comment-13805657 ] Hudson commented on HBASE-9748: --- FAILURE: Integrated in hbase-0.96 #161 (See [https://builds.apache.org/job/hbase-0.96/161/]) HBASE-9748 Address outstanding comments raised for HBASE-9696 (jxiang: rev 1535789) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerNoMaster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7525:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to 0.96 and 0.98
Thanks for the patch [~takeshi.miao]
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7525:
-
Assignee: takeshi.miao (was: Elliott Clark)
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805640#comment-13805640 ] Ted Yu commented on HBASE-9837: --- Integrated to 0.96 and trunk. Thanks for the reviews. > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9843: --- Status: Patch Available (was: Open) > Various fixes in client code > > > Key: HBASE-9843 > URL: https://issues.apache.org/jira/browse/HBASE-9843 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.98.0, 0.96.1 > > Attachments: 9843-trunk.v2.patch > > > This mainly fixes issues when we had "long" errors, for example a multi > blocked when trying to obtain a lock that was finally failing after 60s. > Previously we were trying only for 5 minutes. We now do all the tries. I've > fixed stuff around this area to make it work. > There is also more logs. > I've changed the back off array. With the default pause of 100ms, even after > 20 tries we still retry every 10s. > I've also changed the max per RS to something minimal. If the cluster is not > in a very good state it's less aggressive. It seems to be a better default. > I've done two tests: > - on a small; homogeneous cluster, I had the same performances > - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7525:
-
Assignee: Elliott Clark
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9843: --- Attachment: 9843-trunk.v2.patch > Various fixes in client code > > > Key: HBASE-9843 > URL: https://issues.apache.org/jira/browse/HBASE-9843 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.98.0, 0.96.1 > > Attachments: 9843-trunk.v2.patch > > > This mainly fixes issues when we had "long" errors, for example a multi > blocked when trying to obtain a lock that was finally failing after 60s. > Previously we were trying only for 5 minutes. We now do all the tries. I've > fixed stuff around this area to make it work. > There is also more logs. > I've changed the back off array. With the default pause of 100ms, even after > 20 tries we still retry every 10s. > I've also changed the max per RS to something minimal. If the cluster is not > in a very good state it's less aggressive. It seems to be a better default. > I've done two tests: > - on a small; homogeneous cluster, I had the same performances > - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805634#comment-13805634 ] Elliott Clark commented on HBASE-9837: -- +1 we should get this in. > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7525) A canary monitoring program specifically for regionserver
[
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805632#comment-13805632
]
Elliott Clark commented on HBASE-7525:
--
+1 there are some docs that should be added. But I can add those in a new jira.
> A canary monitoring program specifically for regionserver
> -
>
> Key: HBASE-7525
> URL: https://issues.apache.org/jira/browse/HBASE-7525
> Project: HBase
> Issue Type: New Feature
> Components: monitoring
>Affects Versions: 0.94.0
>Reporter: takeshi.miao
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch,
> HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch,
> HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch,
> HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch,
> RegionServerCanary.java
>
>
> *Motivation*
> This ticket is to provide a canary monitoring tool specifically for
> HRegionserver, details as follows
> 1. This tool is required by operation team due to they thought that the
> canary for each region of a HBase is too many for them, so I implemented this
> coarse-granular one based on the original o.a.h.h.tool.Canary for them
> 2. And this tool is implemented by multi-threading, which means the each Get
> request sent by a thread. the reason I use this way is due to we suffered the
> region server hung issue by now the root cause is still not clear. so this
> tool can help operation team to detect hung region server if any.
> *example*
> 1. the tool docs
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
> Usage: [opts] [regionServerName 1 [regionServrName 2...]]
> regionServerName - FQDN serverName, can use linux command:hostname -f to
> check your serverName
> where [-opts] are:
>-help Show this help and exit.
>-eUse regionServerName as regular expression
> which means the regionServerName is regular expression pattern
>-f stop whole program if first error occurs, default is true
>-t timeout for a check, default is 60 (milisecs)
>-daemonContinuous check at defined intervals.
>-interval Interval between checks (sec)
> 2. Will send a request to each regionserver in a HBase cluster
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
> 3. Will send a request to a regionserver by given name
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
> 4. Will send a request to regionserver(s) by given regular-expression
> /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e
> rs1.domainname.pattern
> // another example
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e
> tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
> 5. Will send a request to a regionserver and also set a timeout limit for
> this test
> // query regionserver:rs1.domainname with timeout limit 10sec
> // -f false, means that will not exit this program even test failed
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1
> rs1.domainname
> // echo "1" if timeout
> echo "$?"
> 6. Will run as daemon mode, which means it will send request to each
> regionserver periodically
> ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805623#comment-13805623 ] Ted Yu commented on HBASE-9837: --- HBASE-9080 was integrated on July 30th. At that time, HBASE-6143 was still being discussed. > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9837) Forward port HBASE-9080 'Retain assignment should be used when re-enabling table(s)'
[ https://issues.apache.org/jira/browse/HBASE-9837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805620#comment-13805620 ] stack commented on HBASE-9837: -- +1 for 0.96 and trunk (Why not do this for all branches in hbase-9080?) > Forward port HBASE-9080 'Retain assignment should be used when re-enabling > table(s)' > > > Key: HBASE-9837 > URL: https://issues.apache.org/jira/browse/HBASE-9837 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.0, 0.96.1 > > Attachments: 9837-v1.txt > > > HBASE-6143 still has some non-trivial work to do (according to Elliott). > This issue is about forward porting HBASE-9080 'Retain assignment should be > used when re-enabling table(s)' to 0.96 and trunk. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9844) zookeepers.sh - ZKServerTool log permission issue
Sebastien Barrier created HBASE-9844: Summary: zookeepers.sh - ZKServerTool log permission issue Key: HBASE-9844 URL: https://issues.apache.org/jira/browse/HBASE-9844 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.94.12 Environment: Linux Reporter: Sebastien Barrier Priority: Minor The zookeepers.sh script exec the following command during it's process /usr/local/hbase/bin/hbase org.apache.hadoop.hbase.zookeeper.ZKServerTool before doing this it also change of directory to the hbase binary for example 'cd /usr/local/hbase/bin' if the permissions of the directory are differents from the user running the ZKServerTool for example hadoop user and root for the directory there's the following error because it try to create a log file (hadoop.log) in the current directory log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: ./hadoop.log (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:212) at java.io.FileOutputStream.(FileOutputStream.java:136) the log should be written in HBASE_LOG_DIR and not in the current directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805607#comment-13805607
]
Himanshu Vashishtha commented on HBASE-8741:
The above link to findbugs failure gives 404; I ran test-patch script and it
gives +1 to findbugs.
{code}
./dev-support/test-patch.sh ~/patches/HBASE-8741-trunk-v6.2.1.patch
{code}
Output:
{code}
{color:red}-1 overall{color}.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version ) warnings.
{color:red}-1 release audit{color}. The applied patch generated 9 release
audit warnings (more than the trunk's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{code}
Let me re-add it here.
> Scope sequenceid to the region rather than regionserver (WAS: Mutations on
> Regions in recovery mode might have same sequenceIDs)
>
>
> Key: HBASE-8741
> URL: https://issues.apache.org/jira/browse/HBASE-8741
> Project: HBase
> Issue Type: Bug
> Components: MTTR
>Affects Versions: 0.95.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.98.0
>
> Attachments: HBASE-8741-trunk-v6.1-rebased.patch,
> HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.patch,
> HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch,
> HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch,
> HBASE-8741-v5-again.patch, HBASE-8741-v5.patch
>
>
> Currently, when opening a region, we find the maximum sequence ID from all
> its HFiles and then set the LogSequenceId of the log (in case the later is at
> a small value). This works good in recovered.edits case as we are not writing
> to the region until we have replayed all of its previous edits.
> With distributed log replay, if we want to enable writes while a region is
> under recovery, we need to make sure that the logSequenceId > maximum
> logSequenceId of the old regionserver. Otherwise, we might have a situation
> where new edits have same (or smaller) sequenceIds.
> We can store region level information in the WALTrailer, than this scenario
> could be avoided by:
> a) reading the trailer of the "last completed" file, i.e., last wal file
> which has a trailer and,
> b) completely reading the last wal file (this file would not have the
> trailer, so it needs to be read completely).
> In future, if we switch to multi wal file, we could read the trailer for all
> completed WAL files, and reading the remaining incomplete files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-trunk-v6.2.1.patch > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.2.1.patch, HBASE-8741-trunk-v6.patch, > HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, > HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, > HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9659) some integration tests can no longer be run using maven
[ https://issues.apache.org/jira/browse/HBASE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805605#comment-13805605 ] stack commented on HBASE-9659: -- +1 on backport to 0.96 Thanks [~sershe] > some integration tests can no longer be run using maven > --- > > Key: HBASE-9659 > URL: https://issues.apache.org/jira/browse/HBASE-9659 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.98.0 > > Attachments: HBASE-9659-v0.patch > > > When I run mvn test (or verify) -Dtest=IntegrationTestIngest, the test fails > instantly, seemingly because initialization doesn't run. I am assuming junit > is not picking before-after methods from the superclass, could be some other > issue. > Also, if it does run, it won't be very useful because it runs with calm > monkey by default. > We need to detect being run locally rather than as AbstractHBaseTool > (probably any time JUnit-annotated methods like Before are called), and set > up a different chaos monkey, such as an old default one. > May also apply to other tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805604#comment-13805604
]
stack commented on HBASE-9836:
--
I would rather remove the test than add this 'fix'
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt, 9836-v4.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 2 store
> files, 0 compacting, 2 eligible, 10 blocking
> 2013-10-24 18:08:10,547 DEBUG [pool-1-thread-1] catalog.CatalogTracker(209):
> Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4be179
> 2013-10-24 18:08:10,549 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.ExploringCompactionPolicy(112): Exploring compaction algorithm
> has selected 2 files of size 1999 starting at candidate #0 after considering
> 1 permutations with 1 in ratio
> 2013-10-24 18:08:10,551 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(1329):
> e96920e43ea374ba1bd559df115870cf - A: Initiating major compaction
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> regionserver.HRegion(1294): Starting compaction on A in region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(982):
> Starting compaction of 2 file(s) in A of
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> into
> tmpdir=hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/.tmp,
> totalSize=2.0k
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/44f87b94732149c08f20bdba00dd7140,
> keycount=1, bloomtype=ROW, size=992.0, encoding=NONE, seqNum=3,
> earliestPutTs=1382638089528
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/0b2e580cbda246718bbf64c21e81cd18,
> keycount=1, bloomtype=ROW, size=1007.0, encoding=NONE, seqNum=5,
> earliestPutTs=1382638090053
> 2013-10-24 18:08:10,564 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] util.FSUtils(305): DFS
> Client does not support most favored nodes create; using default create
> ...
> Potentially hanging thread: RS:0;asf002:58174-smallCompactions-1382638090545
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.ipc.Client.call(Client.java:1099)
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> $Proxy9.complete(Unknown Source)
> sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> j
[jira] [Updated] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9836:
--
Attachment: 9836-v4.txt
Patch v4 fixes the findbugs warning.
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt, 9836-v4.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 2 store
> files, 0 compacting, 2 eligible, 10 blocking
> 2013-10-24 18:08:10,547 DEBUG [pool-1-thread-1] catalog.CatalogTracker(209):
> Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4be179
> 2013-10-24 18:08:10,549 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.ExploringCompactionPolicy(112): Exploring compaction algorithm
> has selected 2 files of size 1999 starting at candidate #0 after considering
> 1 permutations with 1 in ratio
> 2013-10-24 18:08:10,551 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(1329):
> e96920e43ea374ba1bd559df115870cf - A: Initiating major compaction
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> regionserver.HRegion(1294): Starting compaction on A in region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(982):
> Starting compaction of 2 file(s) in A of
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> into
> tmpdir=hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/.tmp,
> totalSize=2.0k
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/44f87b94732149c08f20bdba00dd7140,
> keycount=1, bloomtype=ROW, size=992.0, encoding=NONE, seqNum=3,
> earliestPutTs=1382638089528
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/0b2e580cbda246718bbf64c21e81cd18,
> keycount=1, bloomtype=ROW, size=1007.0, encoding=NONE, seqNum=5,
> earliestPutTs=1382638090053
> 2013-10-24 18:08:10,564 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] util.FSUtils(305): DFS
> Client does not support most favored nodes create; using default create
> ...
> Potentially hanging thread: RS:0;asf002:58174-smallCompactions-1382638090545
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.ipc.Client.call(Client.java:1099)
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> $Proxy9.complete(Unknown Source)
> sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.ja
[jira] [Updated] (HBASE-9792) Region states should update last assignments when a region is opened.
[ https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9792: --- Status: Open (was: Patch Available) > Region states should update last assignments when a region is opened. > - > > Key: HBASE-9792 > URL: https://issues.apache.org/jira/browse/HBASE-9792 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-9792.patch, trunk-9792_v2.patch > > > Currently, we update a region's last assignment region server when the region > is online. We should do this sooner, when the region is moved to OPEN state. > CM could kill this region server before we delete the znode and online the > region. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9765) TestImportTsv.testBulkOutputWithTsvImporterTextMapper() may not be working correctly
[
https://issues.apache.org/jira/browse/HBASE-9765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805588#comment-13805588
]
Nick Dimiduk commented on HBASE-9765:
-
Thanks for the catch, Ram.
I'm +1 for backporting this to 0.96. It looks like the test method doesn't
exist on 0.94.
> TestImportTsv.testBulkOutputWithTsvImporterTextMapper() may not be working
> correctly
>
>
> Key: HBASE-9765
> URL: https://issues.apache.org/jira/browse/HBASE-9765
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.2, 0.96.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-9765.patch
>
>
> The testcase defines a
> {code}
> "-D" + ImportTsv.SEPARATOR_CONF_KEY + "=,",
> {code}
> but the data line has
> {code}
> String data = "KEY\u001bVALUE4\u001bVALUE8\n";
> {code}
> This may not be working correctly. Need to change the seperator.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805584#comment-13805584 ] Jimmy Xiang commented on HBASE-9748: Good catch. I checked in an addendum. Thanks. > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9807) block encoder unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805579#comment-13805579
]
Hudson commented on HBASE-9807:
---
SUCCESS: Integrated in HBase-0.94 #1184 (See
[https://builds.apache.org/job/HBase-0.94/1184/])
HBASE-9807 block encoder unnecessarily copies the key for each reseek (larsh:
rev 1535781)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> block encoder unnecessarily copies the key for each reseek
> --
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates two ByteBuffers in ScannerV2 and makes a deep copy of
> the key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9792) Region states should update last assignments when a region is opened.
[
https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805575#comment-13805575
]
Hadoop QA commented on HBASE-9792:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610341/trunk-9792_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.master.TestAssignmentManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7632//console
This message is automatically generated.
> Region states should update last assignments when a region is opened.
> -
>
> Key: HBASE-9792
> URL: https://issues.apache.org/jira/browse/HBASE-9792
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: trunk-9792.patch, trunk-9792_v2.patch
>
>
> Currently, we update a region's last assignment region server when the region
> is online. We should do this sooner, when the region is moved to OPEN state.
> CM could kill this region server before we delete the znode and online the
> region.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[
https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805572#comment-13805572
]
Ted Yu commented on HBASE-9748:
---
Looks like the following javadoc warning is related to this issue
(https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/patchJavadocWarnings.txt):
{code}
[WARNING] Javadoc Warnings
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java:440:
warning - @param argument "sn" is not a parameter name.
{code}
> Address outstanding comments raised for HBASE-9696
> --
>
> Key: HBASE-9748
> URL: https://issues.apache.org/jira/browse/HBASE-9748
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9748.patch, trunk-9748_v2.1.patch,
> trunk-9748_v2.2.patch, trunk-9748_v2.patch
>
>
> This is a follow-up issue of HBASE-9696.
> There are some outstanding review comments in
> https://reviews.apache.org/r/14470/ that haven't been addressed. I will
> address them later in this jira.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9659) some integration tests can no longer be run using maven
[ https://issues.apache.org/jira/browse/HBASE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805571#comment-13805571 ] Sergey Shelukhin commented on HBASE-9659: - Actually the issue was not that, some methods were just not marked for the test > some integration tests can no longer be run using maven > --- > > Key: HBASE-9659 > URL: https://issues.apache.org/jira/browse/HBASE-9659 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.98.0 > > Attachments: HBASE-9659-v0.patch > > > When I run mvn test (or verify) -Dtest=IntegrationTestIngest, the test fails > instantly, seemingly because initialization doesn't run. I am assuming junit > is not picking before-after methods from the superclass, could be some other > issue. > Also, if it does run, it won't be very useful because it runs with calm > monkey by default. > We need to detect being run locally rather than as AbstractHBaseTool > (probably any time JUnit-annotated methods like Before are called), and set > up a different chaos monkey, such as an old default one. > May also apply to other tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805567#comment-13805567
]
Hadoop QA commented on HBASE-9836:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610354/9836-v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7633//console
This message is automatically generated.
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioB
[jira] [Commented] (HBASE-9840) Large scans and BlockCache evictions problems
[ https://issues.apache.org/jira/browse/HBASE-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805563#comment-13805563 ] Jean-Daniel Cryans commented on HBASE-9840: --- bq. When all three buckets become full all new blocks are inserted at 0.25 split in a queue Yes, but like I am saying 2Q has its own configs that you can't change at runtime and you will hit another edgy use case that isn't able to use all the cache. > Large scans and BlockCache evictions problems > - > > Key: HBASE-9840 > URL: https://issues.apache.org/jira/browse/HBASE-9840 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > I just ran into a scenario that baffled me first, but after some reflection > makes sense. I ran a very large scan that filled up most of the block cache > with my scan's data. I ran that scan a few times. > That I ran a much smaller scan, and this scan will never get all its blocks > cached if it does not fit entirely into the remaining BlockCache; regardless > how I often I run it! > The reason is that the blocks of the first large scan were all promoted. > Since the 2nd scan did not fully fit into the cache all blocks are > round-robin evicted as I rerun the scan. Thus those blocks will never get > accessed more than once before they get evicted again. > Since promoted blocks are not demoted the large scan's block will never be > evicted unless we have another small enough scan/get that can promote its > blocks. > Not sure what the proper solution is, but it seems only a LRU cache that can > expire blocks over time would solve this. > Granted, this is a pretty special case. > Edit: My usual spelling digressions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9840) Large scans and BlockCache evictions problems
[ https://issues.apache.org/jira/browse/HBASE-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805527#comment-13805527 ] Vladimir Rodionov commented on HBASE-9840: -- When all three buckets become full all new blocks are inserted at 0.25 split in a queue. Always, because : 1. cache entries never expire. 2. all buckets manage eviction independently. > Large scans and BlockCache evictions problems > - > > Key: HBASE-9840 > URL: https://issues.apache.org/jira/browse/HBASE-9840 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > I just ran into a scenario that baffled me first, but after some reflection > makes sense. I ran a very large scan that filled up most of the block cache > with my scan's data. I ran that scan a few times. > That I ran a much smaller scan, and this scan will never get all its blocks > cached if it does not fit entirely into the remaining BlockCache; regardless > how I often I run it! > The reason is that the blocks of the first large scan were all promoted. > Since the 2nd scan did not fully fit into the cache all blocks are > round-robin evicted as I rerun the scan. Thus those blocks will never get > accessed more than once before they get evicted again. > Since promoted blocks are not demoted the large scan's block will never be > evicted unless we have another small enough scan/get that can promote its > blocks. > Not sure what the proper solution is, but it seems only a LRU cache that can > expire blocks over time would solve this. > Granted, this is a pretty special case. > Edit: My usual spelling digressions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9748: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Integrated into trunk and 0.96. Thanks. > Address outstanding comments raised for HBASE-9696 > -- > > Key: HBASE-9748 > URL: https://issues.apache.org/jira/browse/HBASE-9748 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: trunk-9748.patch, trunk-9748_v2.1.patch, > trunk-9748_v2.2.patch, trunk-9748_v2.patch > > > This is a follow-up issue of HBASE-9696. > There are some outstanding review comments in > https://reviews.apache.org/r/14470/ that haven't been addressed. I will > address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805521#comment-13805521 ] Himanshu Vashishtha commented on HBASE-8741: Rb'ed it: https://reviews.apache.org/r/14934/ Let me take a look at findbugs now. Thanks. > Scope sequenceid to the region rather than regionserver (WAS: Mutations on > Regions in recovery mode might have same sequenceIDs) > > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBASE-8741-trunk-v6.1-rebased.patch, > HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9659) some integration tests can no longer be run using maven
[ https://issues.apache.org/jira/browse/HBASE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805520#comment-13805520 ] Elliott Clark commented on HBASE-9659: -- I would say yea, a test clean up seems like a good backport to me. It's too bad that JUnit/surefire aren't picking up the before and after methods. That's really weird. > some integration tests can no longer be run using maven > --- > > Key: HBASE-9659 > URL: https://issues.apache.org/jira/browse/HBASE-9659 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.98.0 > > Attachments: HBASE-9659-v0.patch > > > When I run mvn test (or verify) -Dtest=IntegrationTestIngest, the test fails > instantly, seemingly because initialization doesn't run. I am assuming junit > is not picking before-after methods from the superclass, could be some other > issue. > Also, if it does run, it won't be very useful because it runs with calm > monkey by default. > We need to detect being run locally rather than as AbstractHBaseTool > (probably any time JUnit-annotated methods like Before are called), and set > up a different chaos monkey, such as an old default one. > May also apply to other tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9836) Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure
[
https://issues.apache.org/jira/browse/HBASE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9836:
--
Attachment: 9836-v3.txt
Patch v3 introduces CountDownLatch which would only be instantiated by
TestRegionObserverScannerOpenHook
Since TestRegionObserverScannerOpenHook is in coprocessor package, the
getCompactionStateChangeLatch() has to be public.
TestRegionObserverScannerOpenHook and TestHeapSize passed.
> Intermittent
> TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking
> failure
> ---
>
> Key: HBASE-9836
> URL: https://issues.apache.org/jira/browse/HBASE-9836
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9836-v1.txt, 9836-v3.txt
>
>
> Here were two recent examples:
> https://builds.apache.org/job/hbase-0.96-hadoop2/99/testReport/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> https://builds.apache.org/job/PreCommit-HBASE-Build/7616/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionObserverScannerOpenHook/testRegionObserverCompactionTimeStacking/
> From the second:
> {code}
> 2013-10-24 18:08:10,080 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegionServer(3672): Flushing
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> ...
> 2013-10-24 18:08:10,544 INFO [Priority.RpcServer.handler=1,port=58174]
> regionserver.HRegion(1645): Finished memstore flush of ~128.0/128,
> currentsize=0.0/0 for region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> in 464ms, sequenceid=5, compaction requested=true
> 2013-10-24 18:08:10,546 DEBUG [Priority.RpcServer.handler=1,port=58174]
> regionserver.CompactSplitThread(319): Small Compaction requested: system;
> Because: Compaction through user triggered flush; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
> 2013-10-24 18:08:10,547 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 2 store
> files, 0 compacting, 2 eligible, 10 blocking
> 2013-10-24 18:08:10,547 DEBUG [pool-1-thread-1] catalog.CatalogTracker(209):
> Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4be179
> 2013-10-24 18:08:10,549 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.ExploringCompactionPolicy(112): Exploring compaction algorithm
> has selected 2 files of size 1999 starting at candidate #0 after considering
> 1 permutations with 1 in ratio
> 2013-10-24 18:08:10,551 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(1329):
> e96920e43ea374ba1bd559df115870cf - A: Initiating major compaction
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> regionserver.HRegion(1294): Starting compaction on A in region
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> 2013-10-24 18:08:10,551 INFO
> [RS:0;asf002:58174-smallCompactions-1382638090545] regionserver.HStore(982):
> Starting compaction of 2 file(s) in A of
> testRegionObserverCompactionTimeStacking,,1382638088230.e96920e43ea374ba1bd559df115870cf.
> into
> tmpdir=hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/.tmp,
> totalSize=2.0k
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/44f87b94732149c08f20bdba00dd7140,
> keycount=1, bloomtype=ROW, size=992.0, encoding=NONE, seqNum=3,
> earliestPutTs=1382638089528
> 2013-10-24 18:08:10,552 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545]
> compactions.Compactor(168): Compacting
> hdfs://localhost:49506/user/jenkins/hbase/data/default/testRegionObserverCompactionTimeStacking/e96920e43ea374ba1bd559df115870cf/A/0b2e580cbda246718bbf64c21e81cd18,
> keycount=1, bloomtype=ROW, size=1007.0, encoding=NONE, seqNum=5,
> earliestPutTs=1382638090053
> 2013-10-24 18:08:10,564 DEBUG
> [RS:0;asf002:58174-smallCompactions-1382638090545] util.FSUtils(305): DFS
> Client does not support most favored nodes create; using default create
> ...
> Potentially hanging thread: RS:0;asf002:58174-smallCompactions-1382638090545
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.ipc.Client.call(Client.java:1099)
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>
[jira] [Commented] (HBASE-9807) block encoder unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805510#comment-13805510
]
Matt Corgan commented on HBASE-9807:
Sorry i didn't have any comments there. For prefix-tree to benefit, we have to
use the CellComparator, and pass Cell references up through these interfaces.
> block encoder unnecessarily copies the key for each reseek
> --
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates two ByteBuffers in ScannerV2 and makes a deep copy of
> the key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
