[jira] [Commented] (HBASE-9000) Linear reseek in Memstore
[ https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808781#comment-13808781 ] Chao Shi commented on HBASE-9000: - bq. Hah. I just something very similar, but actually in StoreScanner. Not quite ready, but assumes a seek within a row or to the next row is a near seek, and the tries next() a few time (I picked 10 as a default). Did you mean to call next at StoreScanner level? I'm not sure if it is possible to do this there, as it may not have necessary knowledge about the cost of next vs. reseek. For example, a next attempt that causes reading another uncached block from FS is probably worse than do a reseek. I may be wrong as I didn't read your implementation. Linear reseek in Memstore - Key: HBASE-9000 URL: https://issues.apache.org/jira/browse/HBASE-9000 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Shane Hogan Priority: Minor Fix For: 0.89-fb Attachments: hbase-9000-benchmark-program.patch, hbase-9000-port-fb.patch This is to address the linear reseek in MemStoreScanner. Currently reseek iterates over the kvset and the snapshot linearly by just calling next repeatedly. The new solution is to do this linear seek up to a configurable maximum amount of times then if the seek is not yet complete fall back to logarithmic seek. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9775) Client write path perf issues
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9775: - Attachment: 9775.rig.v2.patch Rebase for 0.96. You just run the main on TestClientNoCluster. After updating, no noticeable difference. We run up to 100 threads and stay there w/ near all in wait mode. Client write path perf issues - Key: HBASE-9775 URL: https://issues.apache.org/jira/browse/HBASE-9775 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0 Reporter: Elliott Clark Priority: Critical Attachments: 9775.rig.txt, 9775.rig.v2.patch, Charts Search Cloudera Manager - ITBLL.png, Charts Search Cloudera Manager.png, hbase-9775.patch, job_run.log, short_ycsb.png, ycsb_insert_94_vs_96.png, ycsb.png Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9045: -- Attachment: (was: HBASE-9045_V2.patch) Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9045: -- Status: Patch Available (was: Open) Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9045: -- Status: Open (was: Patch Available) Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9045: -- Attachment: HBASE-9045_V2.patch The Findbugs and Javadoc warnings seems unrelated to this patch. Resubmitting for QA. Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9838) Fix javadoc warning in ImportTsv#TsvParser ctor
[
https://issues.apache.org/jira/browse/HBASE-9838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-9838:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Hence resolving.
Fix javadoc warning in ImportTsv#TsvParser ctor
---
Key: HBASE-9838
URL: https://issues.apache.org/jira/browse/HBASE-9838
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 0.98.0
Attachments: 9838.txt, HBASE-9767_addendum_1.patch,
HBASE-9767_addendum.patch
From
https://builds.apache.org/job/PreCommit-HBASE-Build/7623/artifact/trunk/patchprocess/patchJavadocWarnings.txt
:
{code}
[WARNING] Javadoc Warnings
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java:123:
warning - @param argument tagSeperatorStr is not a parameter name.
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[
https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808845#comment-13808845
]
Hadoop QA commented on HBASE-9045:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610998/HBASE-9045_V2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7674//console
This message is automatically generated.
Support Dictionary based Tag compression in HFiles
--
Key: HBASE-9045
URL: https://issues.apache.org/jira/browse/HBASE-9045
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0
Attachments: HBASE-9045.patch, HBASE-9045_V2.patch
Along with the DataBlockEncoding algorithms, Dictionary based Tag compression
can be done
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8128) HTable#put improvements
[ https://issues.apache.org/jira/browse/HBASE-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808851#comment-13808851 ] Nicolas Liochon commented on HBASE-8128: [~jxiang] It's there (the code has changed a lot since, but there is still the doPut method for example). HTable#put improvements --- Key: HBASE-8128 URL: https://issues.apache.org/jira/browse/HBASE-8128 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.95.0, 0.95.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.94.7, 0.95.0, 0.95.2 Attachments: 8128.v1.patch 3 points: - When doing a single put, we're creating an object by calling Arrays.asList - we're doing a size check every 10 put. Not doing it seems simpler, better and allows to share some code between a single put and a list of puts. - we could call flushCommits on empty write buffer, especially for someone using a lot of HTable instead of using a pool, as it's called in close(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808880#comment-13808880 ] Anoop Sam John commented on HBASE-9045: --- Checking the findbugs report, I can not see any warnings from the code this patch touches. Seems unrelated. Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9835) Define C interface of HBase Client synchronous APIs
[ https://issues.apache.org/jira/browse/HBASE-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808935#comment-13808935 ] Aditya Kishore commented on HBASE-9835: --- Patch available at https://reviews.apache.org/r/15079/ Define C interface of HBase Client synchronous APIs --- Key: HBASE-9835 URL: https://issues.apache.org/jira/browse/HBASE-9835 Project: HBase Issue Type: Sub-task Components: Client Reporter: Aditya Kishore Assignee: Aditya Kishore Labels: C Creating this as a sub task of HBASE-1015 to define Define C language interface of HBase Client synchronous APIs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9861) Location does not have to be refreshed on regionTooBusy
Nicolas Liochon created HBASE-9861: -- Summary: Location does not have to be refreshed on regionTooBusy Key: HBASE-9861 URL: https://issues.apache.org/jira/browse/HBASE-9861 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Minor improvement. There is already some code to manage the exception, so I've change it to share between the classes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9861) Location does not have to be refreshed on regionTooBusy
[ https://issues.apache.org/jira/browse/HBASE-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9861: --- Status: Patch Available (was: Open) Location does not have to be refreshed on regionTooBusy --- Key: HBASE-9861 URL: https://issues.apache.org/jira/browse/HBASE-9861 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Attachments: 9861.v1.patch Minor improvement. There is already some code to manage the exception, so I've change it to share between the classes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9843) Various fixes in client code
[ https://issues.apache.org/jira/browse/HBASE-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9843: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Various fixes in client code Key: HBASE-9843 URL: https://issues.apache.org/jira/browse/HBASE-9843 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9843-trunk.v2.patch, 9843-trunk.v3.patch This mainly fixes issues when we had long errors, for example a multi blocked when trying to obtain a lock that was finally failing after 60s. Previously we were trying only for 5 minutes. We now do all the tries. I've fixed stuff around this area to make it work. There is also more logs. I've changed the back off array. With the default pause of 100ms, even after 20 tries we still retry every 10s. I've also changed the max per RS to something minimal. If the cluster is not in a very good state it's less aggressive. It seems to be a better default. I've done two tests: - on a small; homogeneous cluster, I had the same performances - on a bigger, but heterogeneous cluster it was twice as fast. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9861) Location does not have to be refreshed on regionTooBusy
[ https://issues.apache.org/jira/browse/HBASE-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9861: --- Attachment: 9861.v1.patch Location does not have to be refreshed on regionTooBusy --- Key: HBASE-9861 URL: https://issues.apache.org/jira/browse/HBASE-9861 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Attachments: 9861.v1.patch Minor improvement. There is already some code to manage the exception, so I've change it to share between the classes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9221) Provide interface for getting a User in the client
[ https://issues.apache.org/jira/browse/HBASE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809066#comment-13809066 ] Hudson commented on HBASE-9221: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #817 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/817/]) HBASE-9221: Provide interface for getting a User in the client: ADDENDUM (jyates: rev 1536944) * /hbase/trunk/hbase-client/pom.xml * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/security/UserProvider.java HBASE-9221: Provide interface for getting a User in the client (jyates: rev 1536937) * /hbase/trunk/hbase-client/pom.xml * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionKey.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/security * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/security/UserProvider.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestCallRunner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/HadoopSecurityEnabledUserProviderForTesting.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSecureLoadIncrementalHFilesSplitRecovery.java * /hbase/trunk/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java Provide interface for getting a User in the client -- Key: HBASE-9221 URL: https://issues.apache.org/jira/browse/HBASE-9221 Project: HBase Issue Type: Improvement Components: Client, hadoop2, security Affects Versions: 0.98.0, 0.95.2, 0.94.12 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: hbase-9221-0.94-v0.patch, hbase-9221-0.94-v1.patch, hbase-9221-0.94-v3.patch, hbase-9221-0.94-v4.patch, hbase-9221-0.96-trunk.addendum, hbase-9221-0.96-v0.patch, hbase-9221-0.96-v1.patch, hbase-9221-trunk-v0.patch, hbase-9221-trunk-v1.patch, hbase-9221-trunk-v2.patch, hbase-9221-trunk-v3.patch, hbase-9221-trunk-v4.patch, hbase-9221-trunk-v5.patch Sometimes, users will want to provide their own User class depending on the type of security they will support in their local environment. For instance, running Hadoop1 vs Hadoop2 vs CDH means potentially different ways of getting the UserGroupInformation. This issue abstracts out the mechanism by which we obtain an o.a.h.hbase.security.User to a UserProvider. This UserProvider can then be extented as a Hadoop 1/2 shim as well as supporting custom authentication code. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809131#comment-13809131 ] Andrew Purtell commented on HBASE-9045: --- +1 on patch v2 Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9861) Location does not have to be refreshed on regionTooBusy
[
https://issues.apache.org/jira/browse/HBASE-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809155#comment-13809155
]
Hadoop QA commented on HBASE-9861:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611056/9861.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7675//console
This message is automatically generated.
Location does not have to be refreshed on regionTooBusy
---
Key: HBASE-9861
URL: https://issues.apache.org/jira/browse/HBASE-9861
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.96.1
Attachments: 9861.v1.patch
Minor improvement. There is already some code to manage the exception, so
I've change it to share between the classes.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809161#comment-13809161 ] Anoop Sam John commented on HBASE-9045: --- Thanks Andy. Will commit tomorrow morning IST unless objections Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9861) Location does not have to be refreshed on regionTooBusy
[
https://issues.apache.org/jira/browse/HBASE-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809309#comment-13809309
]
stack commented on HBASE-9861:
--
Makes sense.
Could we get stuck here if new location is bad? Or will the new location be
the old location next time through?
+ if (oldLocation == null ||
!source.getServerName().equals(oldLocation.getServerName())) {
+// There is no such location in the cache (it's been removed already)
or
+// the cache has already been refreshed with a different location. =
nothing to do
This is great
+if (cause instanceof RegionTooBusyException || cause instanceof
RegionOpeningException) {
+ // We know that the region is still on this region server
+ return;
+}
Fix this comment:
+ * Look for an excpetion we know in the remove exception:
findException looks to duplicate a bunch of exception parsing that we do in
multiple places. We should do a walk through one of these days to clean it all
up.
oh, I take it back. You are doing cleanup. How'd we have that dupe of
exception parsing in RegionMoved and RegionOpening?
+1 caveat the little concern up top.
Nice [~nkeywal]
Location does not have to be refreshed on regionTooBusy
---
Key: HBASE-9861
URL: https://issues.apache.org/jira/browse/HBASE-9861
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.96.1
Attachments: 9861.v1.patch
Minor improvement. There is already some code to manage the exception, so
I've change it to share between the classes.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9861) Location does not have to be refreshed on regionTooBusy
[ https://issues.apache.org/jira/browse/HBASE-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809323#comment-13809323 ] Nicolas Liochon commented on HBASE-9861: bq. Could we get stuck here if new location is bad? Or will the new location be the old location next time through? We won't get stuck, as if it's wrong we will try it and change it. So the worse case is consuming a try (which is not great, but going to meta again may not help anyway). bq. + * Look for an excpetion we know in the remove exception: Ok, will do on commit (I will check the findbug as well) Thanks, Stack. Location does not have to be refreshed on regionTooBusy --- Key: HBASE-9861 URL: https://issues.apache.org/jira/browse/HBASE-9861 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Attachments: 9861.v1.patch Minor improvement. There is already some code to manage the exception, so I've change it to share between the classes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9000) Linear reseek in Memstore
[ https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809327#comment-13809327 ] Lars Hofhansl commented on HBASE-9000: -- Yeah I agree. It's actually better to do that at the MestoreScanner and StoreFileScanner level. Linear reseek in Memstore - Key: HBASE-9000 URL: https://issues.apache.org/jira/browse/HBASE-9000 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Shane Hogan Priority: Minor Fix For: 0.89-fb Attachments: hbase-9000-benchmark-program.patch, hbase-9000-port-fb.patch This is to address the linear reseek in MemStoreScanner. Currently reseek iterates over the kvset and the snapshot linearly by just calling next repeatedly. The new solution is to do this linear seek up to a configurable maximum amount of times then if the seek is not yet complete fall back to logarithmic seek. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9862) manage error per server and per region in the protobuffed client
Nicolas Liochon created HBASE-9862: -- Summary: manage error per server and per region in the protobuffed client Key: HBASE-9862 URL: https://issues.apache.org/jira/browse/HBASE-9862 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 The patch does not change anything else than the description says. The changes are about extracting the common paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9792) Region states should update last assignments when a region is opened.
[ https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809345#comment-13809345 ] Jimmy Xiang commented on HBASE-9792: Put patch v3 on RB: https://reviews.apache.org/r/15098/ Region states should update last assignments when a region is opened. - Key: HBASE-9792 URL: https://issues.apache.org/jira/browse/HBASE-9792 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: trunk-9792.patch, trunk-9792_v2.patch, trunk-9792_v3.patch Currently, we update a region's last assignment region server when the region is online. We should do this sooner, when the region is moved to OPEN state. CM could kill this region server before we delete the znode and online the region. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9047) Tool to handle finishing replication when the cluster is offline
[ https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809348#comment-13809348 ] Demai Ni commented on HBASE-9047: - I am looking at the error returned from Hadoop QA. The failed testcase TestZookeeper failed at admin.createTable(...) , I couldn't figure out the relationship with my patch, and I am also able to run it locally on my machine. Another interesting finding is that https://builds.apache.org/job/PreCommit-HBASE-Build/7671/changes shows another patch Hbase-9221, so maybe the failure is caused by a different code. the Findbugs warnings(https://builds.apache.org/job/PreCommit-HBASE-Build/7671//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html) show some problems in my code. I will fix them. Demai Tool to handle finishing replication when the cluster is offline Key: HBASE-9047 URL: https://issues.apache.org/jira/browse/HBASE-9047 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0 Attachments: HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch, HBASE-9047-trunk-v1.patch, HBASE-9047-trunk-v2.patch We're having a discussion on the mailing list about replicating the data on a cluster that was shut down in an offline fashion. The motivation could be that you don't want to bring HBase back up but still need that data on the slave. So I have this idea of a tool that would be running on the master cluster while it is down, although it could also run at any time. Basically it would be able to read the replication state of each master region server, finish replicating what's missing to all the slave, and then clear that state in zookeeper. The code that handles replication does most of that already, see ReplicationSourceManager and ReplicationSource. Basically when ReplicationSourceManager.init() is called, it will check all the queues in ZK and try to grab those that aren't attached to a region server. If the whole cluster is down, it will grab all of them. The beautiful thing here is that you could start that tool on all your machines and the load will be spread out, but that might not be a big concern if replication wasn't lagging since it would take a few seconds to finish replicating the missing data for each region server. I'm guessing when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source. FWIW the main difference in how replication is handled between Apache's HBase and Facebook's is that the latter is always done separately of HBase itself. This jira isn't about doing that. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9000) Linear reseek in Memstore
[ https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809366#comment-13809366 ] Ted Yu commented on HBASE-9000: --- 'reseek to next row' gets slower with patch. If linear.search.limit is lowered, would the slow down be less ? Linear reseek in Memstore - Key: HBASE-9000 URL: https://issues.apache.org/jira/browse/HBASE-9000 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Shane Hogan Priority: Minor Fix For: 0.89-fb Attachments: hbase-9000-benchmark-program.patch, hbase-9000-port-fb.patch This is to address the linear reseek in MemStoreScanner. Currently reseek iterates over the kvset and the snapshot linearly by just calling next repeatedly. The new solution is to do this linear seek up to a configurable maximum amount of times then if the seek is not yet complete fall back to logarithmic seek. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9816) Address review comments in HBASE-8496
[ https://issues.apache.org/jira/browse/HBASE-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9816: -- Attachment: HBASE-9816.patch Patch that addresses the review comments. Address review comments in HBASE-8496 - Key: HBASE-9816 URL: https://issues.apache.org/jira/browse/HBASE-9816 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9816.patch This JIRA would be used to address the review comments in HBASE-8496. Any more comments would be addressed and committed as part of this. There are already few comments from Stack on the RB. https://reviews.apache.org/r/13311/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9816) Address review comments in HBASE-8496
[ https://issues.apache.org/jira/browse/HBASE-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9816: -- Status: Patch Available (was: In Progress) Address review comments in HBASE-8496 - Key: HBASE-9816 URL: https://issues.apache.org/jira/browse/HBASE-9816 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9816.patch This JIRA would be used to address the review comments in HBASE-8496. Any more comments would be addressed and committed as part of this. There are already few comments from Stack on the RB. https://reviews.apache.org/r/13311/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9000) Linear reseek in Memstore
[
https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809383#comment-13809383
]
Ted Yu commented on HBASE-9000:
---
Some comments on the benchmark:
Please add Apache license header.
Add class javadoc for MemStoreReseekBenchmark.
{code}
+ private final Random random = new Random();
{code}
Can you use seed for the above call and log the value of the seed ?
{code}
+ private Listbyte[] rows; private Listbyte[] qualifiers;
{code}
Please put the above on two lines.
{code}
+ boolean ok = scanner.reseek(key);
+ if (!ok) {
+throw new AssertionError(!ok);
{code}
Please print the key which caused assertion.
Linear reseek in Memstore
-
Key: HBASE-9000
URL: https://issues.apache.org/jira/browse/HBASE-9000
Project: HBase
Issue Type: Improvement
Affects Versions: 0.89-fb
Reporter: Shane Hogan
Priority: Minor
Fix For: 0.89-fb
Attachments: hbase-9000-benchmark-program.patch,
hbase-9000-port-fb.patch
This is to address the linear reseek in MemStoreScanner. Currently reseek
iterates over the kvset and the snapshot linearly by just calling next
repeatedly. The new solution is to do this linear seek up to a configurable
maximum amount of times then if the seek is not yet complete fall back to
logarithmic seek.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9000) Linear reseek in Memstore
[ https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809402#comment-13809402 ] Lars Hofhansl commented on HBASE-9000: -- bq. 'reseek to next row' gets slower with patch. This is because the limit of 20 not optimal for this particular case. We'll never get it 100% right for all cases, what we need to do is to avoid the worst cases. In this case it'll be faster per op if the limit is increased to 30 (my guess) or lowered. Linear reseek in Memstore - Key: HBASE-9000 URL: https://issues.apache.org/jira/browse/HBASE-9000 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Shane Hogan Priority: Minor Fix For: 0.89-fb Attachments: hbase-9000-benchmark-program.patch, hbase-9000-port-fb.patch This is to address the linear reseek in MemStoreScanner. Currently reseek iterates over the kvset and the snapshot linearly by just calling next repeatedly. The new solution is to do this linear seek up to a configurable maximum amount of times then if the seek is not yet complete fall back to logarithmic seek. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8552) fix coverage org.apache.hadoop.hbase.rest.filter
[ https://issues.apache.org/jira/browse/HBASE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8552: - Resolution: Fixed Fix Version/s: 0.94.14 0.96.1 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~aklochkov] for the new test and [~te...@apache.org] for reviews! I've integrated the patch into 0.94, 0.96 and trunk branches. fix coverage org.apache.hadoop.hbase.rest.filter - Key: HBASE-8552 URL: https://issues.apache.org/jira/browse/HBASE-8552 Project: HBase Issue Type: Test Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8552-0.94.patch, HBASE-8552-trunk--n2.patch, HBASE-8552-trunk--n3.patch, HBASE-8552-trunk.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9816) Address review comments in HBASE-8496
[ https://issues.apache.org/jira/browse/HBASE-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809409#comment-13809409 ] stack commented on HBASE-9816: -- Stick it up on rb please [~ram_krish] Address review comments in HBASE-8496 - Key: HBASE-9816 URL: https://issues.apache.org/jira/browse/HBASE-9816 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9816.patch This JIRA would be used to address the review comments in HBASE-8496. Any more comments would be addressed and committed as part of this. There are already few comments from Stack on the RB. https://reviews.apache.org/r/13311/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8557) fix coverage org.apache.hadoop.hbase.rest.metrics
[ https://issues.apache.org/jira/browse/HBASE-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8557: - Assignee: Aleksey Gorshkov fix coverage org.apache.hadoop.hbase.rest.metrics - Key: HBASE-8557 URL: https://issues.apache.org/jira/browse/HBASE-8557 Project: HBase Issue Type: Test Affects Versions: 0.94.9 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: HBASE-8557-0.94.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8557) fix coverage org.apache.hadoop.hbase.rest.metrics
[ https://issues.apache.org/jira/browse/HBASE-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8557: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~aleksgor] for the new test! I've committed it into 0.94 branch. fix coverage org.apache.hadoop.hbase.rest.metrics - Key: HBASE-8557 URL: https://issues.apache.org/jira/browse/HBASE-8557 Project: HBase Issue Type: Test Affects Versions: 0.94.9 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: HBASE-8557-0.94.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HBASE-9854) initial documentation for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HBASE-9854: --- Assignee: Sergey Shelukhin initial documentation for stripe compactions Key: HBASE-9854 URL: https://issues.apache.org/jira/browse/HBASE-9854 Project: HBase Issue Type: Sub-task Components: Compaction Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Initial documentation for stripe compactions (distill from attached docs, make up to date, put somewhere like book) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[
https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809467#comment-13809467
]
ramkrishna.s.vasudevan commented on HBASE-9045:
---
Sorry for being late into this,
Just thinking on this do we always need to compress tag by tag or sometimes the
entire tag part can be compressed. In some cases compressing the entire thing
would be simple and would be better for that scneario I feel. That would imapct
the WAL compresssion also then. Can do it based on configuration?
We are having two flavours of compress and uncompress tags. I can get the
meaning when combined with the WAL tag compression.
{code}
Context that holds the dictionary for Tag compression and doing the
compress/uncompress. This
* will be used for compressing tags while writing into WALs.
{code}
Can change this javadoc in TagCompressionContext.
In what scenario can a byte buffer be not backed by an array?
{code}
if (in.hasArray())
{code}
Is it because in some cases we are passing just an empty bytebuffer? Looks
good overall.
Support Dictionary based Tag compression in HFiles
--
Key: HBASE-9045
URL: https://issues.apache.org/jira/browse/HBASE-9045
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0
Attachments: HBASE-9045.patch, HBASE-9045_V2.patch
Along with the DataBlockEncoding algorithms, Dictionary based Tag compression
can be done
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8557) fix coverage org.apache.hadoop.hbase.rest.metrics
[ https://issues.apache.org/jira/browse/HBASE-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8557: - Fix Version/s: 0.94.14 fix coverage org.apache.hadoop.hbase.rest.metrics - Key: HBASE-8557 URL: https://issues.apache.org/jira/browse/HBASE-8557 Project: HBase Issue Type: Test Affects Versions: 0.94.9 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 0.94.14 Attachments: HBASE-8557-0.94.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9822) IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
[
https://issues.apache.org/jira/browse/HBASE-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-9822:
-
Resolution: Fixed
Fix Version/s: 0.96.1
0.98.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks [~saint@gmail.com] for the review! I've committed the patch into
0.96 and trunk branch.
IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
---
Key: HBASE-9822
URL: https://issues.apache.org/jira/browse/HBASE-9822
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Trivial
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9822.patch
This test case failed in a secure deployment once with the following error.
It's due to a race condition between writers starts writes and table ACLs
propagation to region servers.
{code}
2013-10-14 13:03:32,185 ERROR [HBaseWriterThread_8]
util.MultiThreadedWriterBase: Failed to insert: 10 after 167ms; region
information: cached:
region=IntegrationTestLazyCfLoading,bffd,1381755808862.456a11d22693f7dc27763c32e55521a8.,
hostname=gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694,
seqNum=1; cache is up to date; errors: exception from
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal:60020 for
d3d9446802a44259755d38e6d163e820-10
E org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (table=IntegrationTestLazyCfLoading, family: essential:filter,
action=WRITE)
{code}
Writes were sent at 13:03:32,032
{code}
2013-10-14 13:03:32,032 WARN [htable-pool11-t1] client.AsyncProcess: Attempt
#1/35 failed for 1 ops on
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694 NOT
resubmitting.
{code}
While the permission propagation happened at 13:03:32,109 on region server
{code}
2013-10-14 13:03:32,109 DEBUG [regionserver60020-EventThread]
access.ZKPermissionWatcher: Updating permissions cache from node
IntegrationTestLazyCfLoading with data:
PBUF\x0AA\x0A\x06hrt_qa\x127\x08\x033\x0A'\x0A\x07default\x12\x1CIntegrationTestLazyCfLoading
\x00 \x01 \x02 \x03 \x04
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[
https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809492#comment-13809492
]
Anoop Sam John commented on HBASE-9045:
---
bq.Can do it based on configuration?
IMO deciding such a conf is hard. Tags might be internally used for different
use cases later. (Already there are some plans in other jiras) For a user to
know all these and deciding such a conf will be hard right? That is why I am
thinking that tag by tag compression will be safe. What do you say?
Will correct the javadoc. I missed that in this patch. Thanks for the finding.
bq.if (in.hasArray())
That is the recommendation from javadoc for using array()
{quote}
Invoke the hasArray method before invoking this
method in order to ensure that this buffer has an accessible backing
array.
{quote}
Support Dictionary based Tag compression in HFiles
--
Key: HBASE-9045
URL: https://issues.apache.org/jira/browse/HBASE-9045
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0
Attachments: HBASE-9045.patch, HBASE-9045_V2.patch
Along with the DataBlockEncoding algorithms, Dictionary based Tag compression
can be done
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809496#comment-13809496 ] Andrew Purtell commented on HBASE-9045: --- bq. do we always need to compress tag by tag or sometimes the entire tag part can be compressed. In some cases compressing the entire thing would be simple and would be better for that scneario I feel. That would imapct the WAL compresssion also then. Interesting point. For the case where there's just one tag on the cell, it's the same, and for cases where there are a number of cells with the exact same set of tags it would perform better. On the other hand, if cells have many common tags but the similarities don't coincide on any given cell then the dictionary will be inefficient compared to the per-tag approach. Probably the per-tag approach is better for the general case. Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809496#comment-13809496 ] Andrew Purtell edited comment on HBASE-9045 at 10/30/13 7:08 PM: - bq. do we always need to compress tag by tag or sometimes the entire tag part can be compressed. In some cases compressing the entire thing would be simple and would be better for that scneario I feel. That would imapct the WAL compresssion also then. Interesting point. For the case where there's just one tag on the cell, it's the same, and for cases where there are a number of cells with the exact same set of tags it would perform better. On the other hand, if cells have many common tags but the similarities don't coincide on any given cell then the dictionary will be inefficiently constructed compared to the per-tag approach. Probably the per-tag approach is better for the general case. was (Author: apurtell): bq. do we always need to compress tag by tag or sometimes the entire tag part can be compressed. In some cases compressing the entire thing would be simple and would be better for that scneario I feel. That would imapct the WAL compresssion also then. Interesting point. For the case where there's just one tag on the cell, it's the same, and for cases where there are a number of cells with the exact same set of tags it would perform better. On the other hand, if cells have many common tags but the similarities don't coincide on any given cell then the dictionary will be inefficient compared to the per-tag approach. Probably the per-tag approach is better for the general case. Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9816) Address review comments in HBASE-8496
[
https://issues.apache.org/jira/browse/HBASE-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809500#comment-13809500
]
Hadoop QA commented on HBASE-9816:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611122/HBASE-9816.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 54 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676//console
This message is automatically generated.
Address review comments in HBASE-8496
-
Key: HBASE-9816
URL: https://issues.apache.org/jira/browse/HBASE-9816
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0
Attachments: HBASE-9816.patch
This JIRA would be used to address the review comments in HBASE-8496. Any
more comments would be addressed and committed as part of this. There are
already few comments from Stack on the RB.
https://reviews.apache.org/r/13311/
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[ https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9045: -- Attachment: HBASE-9045_V3.patch Support Dictionary based Tag compression in HFiles -- Key: HBASE-9045 URL: https://issues.apache.org/jira/browse/HBASE-9045 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9045.patch, HBASE-9045_V2.patch, HBASE-9045_V3.patch Along with the DataBlockEncoding algorithms, Dictionary based Tag compression can be done -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9862) manage error per server and per region in the protobuffed client
[ https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9862: --- Attachment: 9862.v2.patch manage error per server and per region in the protobuffed client Key: HBASE-9862 URL: https://issues.apache.org/jira/browse/HBASE-9862 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9862.v2.patch The patch does not change anything else than the description says. The changes are about extracting the common paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9862) manage error per server and per region in the protobuffed client
[ https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9862: --- Status: Patch Available (was: Open) At the end, it includes a fix for the region name / RegionSpecifier deserialization in multi. manage error per server and per region in the protobuffed client Key: HBASE-9862 URL: https://issues.apache.org/jira/browse/HBASE-9862 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9862.v2.patch The patch does not change anything else than the description says. The changes are about extracting the common paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9047) Tool to handle finishing replication when the cluster is offline
[ https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Demai Ni updated HBASE-9047: Attachment: HBASE-9047-trunk-v3.patch update v3 for another Hadoop QA testing. v3 fixed the dodgy warnings about the static variable written from instance method Tool to handle finishing replication when the cluster is offline Key: HBASE-9047 URL: https://issues.apache.org/jira/browse/HBASE-9047 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0 Attachments: HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch, HBASE-9047-trunk-v1.patch, HBASE-9047-trunk-v2.patch, HBASE-9047-trunk-v3.patch We're having a discussion on the mailing list about replicating the data on a cluster that was shut down in an offline fashion. The motivation could be that you don't want to bring HBase back up but still need that data on the slave. So I have this idea of a tool that would be running on the master cluster while it is down, although it could also run at any time. Basically it would be able to read the replication state of each master region server, finish replicating what's missing to all the slave, and then clear that state in zookeeper. The code that handles replication does most of that already, see ReplicationSourceManager and ReplicationSource. Basically when ReplicationSourceManager.init() is called, it will check all the queues in ZK and try to grab those that aren't attached to a region server. If the whole cluster is down, it will grab all of them. The beautiful thing here is that you could start that tool on all your machines and the load will be spread out, but that might not be a big concern if replication wasn't lagging since it would take a few seconds to finish replicating the missing data for each region server. I'm guessing when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source. FWIW the main difference in how replication is handled between Apache's HBase and Facebook's is that the latter is always done separately of HBase itself. This jira isn't about doing that. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9863) Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs
Ted Yu created HBASE-9863:
-
Summary: Intermittently
TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs
Key: HBASE-9863
URL: https://issues.apache.org/jira/browse/HBASE-9863
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry sometimes
hung.
Here were two recent occurrences:
https://builds.apache.org/job/PreCommit-HBASE-Build/7676/console
https://builds.apache.org/job/PreCommit-HBASE-Build/7671/console
There were 9 occurrences of the following in both stack traces:
{code}
FifoRpcScheduler.handler1-thread-5 daemon prio=10 tid=0x09df8800 nid=0xc17
waiting for monitor entry [0x6fdf8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
- waiting to lock 0x7f69b5f0 (a
org.apache.hadoop.hbase.master.TableNamespaceManager)
at
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3146)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3105)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1743)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1782)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38221)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1983)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
{code}
The test hung here:
{code}
pool-1-thread-1 prio=10 tid=0x74f7b800 nid=0x5aa5 in Object.wait()
[0x74efe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xcc848348 (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1436)
- locked 0xcc848348 (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.createTable(MasterProtos.java:40372)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.createTable(HConnectionManager.java:1931)
at org.apache.hadoop.hbase.client.HBaseAdmin$2.call(HBaseAdmin.java:598)
at org.apache.hadoop.hbase.client.HBaseAdmin$2.call(HBaseAdmin.java:594)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
- locked 0x7faa26d0 (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
- locked 0x7faa26d0 (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3124)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:594)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:485)
at
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9047) Tool to handle finishing replication when the cluster is offline
[ https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809532#comment-13809532 ] Demai Ni commented on HBASE-9047: - the failure from previous Hadoop QA is the same as HBASE-9863 intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs Tool to handle finishing replication when the cluster is offline Key: HBASE-9047 URL: https://issues.apache.org/jira/browse/HBASE-9047 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0 Attachments: HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch, HBASE-9047-trunk-v1.patch, HBASE-9047-trunk-v2.patch, HBASE-9047-trunk-v3.patch We're having a discussion on the mailing list about replicating the data on a cluster that was shut down in an offline fashion. The motivation could be that you don't want to bring HBase back up but still need that data on the slave. So I have this idea of a tool that would be running on the master cluster while it is down, although it could also run at any time. Basically it would be able to read the replication state of each master region server, finish replicating what's missing to all the slave, and then clear that state in zookeeper. The code that handles replication does most of that already, see ReplicationSourceManager and ReplicationSource. Basically when ReplicationSourceManager.init() is called, it will check all the queues in ZK and try to grab those that aren't attached to a region server. If the whole cluster is down, it will grab all of them. The beautiful thing here is that you could start that tool on all your machines and the load will be spread out, but that might not be a big concern if replication wasn't lagging since it would take a few seconds to finish replicating the missing data for each region server. I'm guessing when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source. FWIW the main difference in how replication is handled between Apache's HBase and Facebook's is that the latter is always done separately of HBase itself. This jira isn't about doing that. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
stack created HBASE-9864: Summary: Notifications bus for use by cluster members keeping up-to-date on changes Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8552) fix coverage org.apache.hadoop.hbase.rest.filter
[ https://issues.apache.org/jira/browse/HBASE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809556#comment-13809556 ] Hudson commented on HBASE-8552: --- SUCCESS: Integrated in hbase-0.96 #170 (See [https://builds.apache.org/job/hbase-0.96/170/]) HBASE-8552: fix coverage org.apache.hadoop.hbase.rest.filter(by Andrey Klochkov) (jeffreyz: rev 1537198) * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java fix coverage org.apache.hadoop.hbase.rest.filter - Key: HBASE-8552 URL: https://issues.apache.org/jira/browse/HBASE-8552 Project: HBase Issue Type: Test Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8552-0.94.patch, HBASE-8552-trunk--n2.patch, HBASE-8552-trunk--n3.patch, HBASE-8552-trunk.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9862) manage error per server and per region in the protobuffed client
[
https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809560#comment-13809560
]
stack commented on HBASE-9862:
--
I tried it in my little harness and it doesn't break anything at least.
Patch application failed because you added something I had -- catching
Thowables.
+} catch (Throwable t) {
+ // This should not happen. Let's log retry anyway.
+ LOG.warn(# + id + , Caught throwable while calling. This is
unexpected. +
+ Retrying. Server is + loc.getServerName() + ,
tableName= + tableName, t);
+ receiveGlobalFailure(initialActions, multiAction, loc,
numAttempt, t,
+ errorsByServer);
return;
The above should be LOG.error because it is not supposed to happen (though it
did for me when making up my harness getting stuff wrong and probably for you
when you were refactoring -- it is too easy for exceptions to be suppressed in
this stuff).
We don't usually run w/ asserts:
+ assert responses != null;
.. perhaps in testing, I don't recall.
Let me mess some more w/ it in place before I give a +1... let me manufacture
the errors you address here.
manage error per server and per region in the protobuffed client
Key: HBASE-9862
URL: https://issues.apache.org/jira/browse/HBASE-9862
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9862.v2.patch
The patch does not change anything else than the description says. The
changes are about extracting the common paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9862) manage error per server and per region in the protobuffed client
[ https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809573#comment-13809573 ] Sergey Shelukhin commented on HBASE-9862: - iirc we used to run w/asserts in testing, at least in the past. Patch looks reasonable, assuming it's just a refactoring for the most part it should be good. One thing we could do in theory (in some other jira) is somehow wake sleepers that want to retry to particular regions, when we get updated cache location for that region. manage error per server and per region in the protobuffed client Key: HBASE-9862 URL: https://issues.apache.org/jira/browse/HBASE-9862 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9862.v2.patch The patch does not change anything else than the description says. The changes are about extracting the common paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9862) manage error per server and per region in the protobuffed client
[ https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809588#comment-13809588 ] Nicolas Liochon commented on HBASE-9862: For the assert, I've used them as a kind of documentation (I use IllegalState when I fear it could occur in production). When we run the tests they are activated. I can remove them, or replace them with an IllegalArgumentException. bq. assuming it's just a refactoring for the most part it should be good. Yes, the logic is not supposed to change (if I did it well :-) ). The changes are 90% code extraction. The remaining 10% took me a while... Thanks for the comments. manage error per server and per region in the protobuffed client Key: HBASE-9862 URL: https://issues.apache.org/jira/browse/HBASE-9862 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9862.v2.patch The patch does not change anything else than the description says. The changes are about extracting the common paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-7254) Refactor AccessController ZK-mediated permissions cache into a generic mechanism
[ https://issues.apache.org/jira/browse/HBASE-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7254: -- Component/s: (was: security) (was: Coprocessors) Affects Version/s: (was: 0.95.2) 0.98.0 Summary: Refactor AccessController ZK-mediated permissions cache into a generic mechanism (was: Consider replacing AccessController ZK-mediated permissions cache) This came up in a review for HBASE-7663. There are now two and potentially a third feature which implement their own ZK mediated distributed cache: the AccessController, namespaces, and the VisibilityController. We should have a common mechanism. I think there are a couple of reasons why it hasn't happened yet: it's easy to copy the AC's example, we're not sure a replacement would be better or work properly at first, and a bug that prevents proper functioning of security code could become an embarrasing CVE. It needs someone to sit down, clear other stuff, and focus. Refactor AccessController ZK-mediated permissions cache into a generic mechanism Key: HBASE-7254 URL: https://issues.apache.org/jira/browse/HBASE-7254 Project: HBase Issue Type: Task Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell After HBASE-5487 or HBASE-7212 goes in, we could replace the AccessController's permissions cache update via ZK using one of these more general frameworks, thus reducing functional duplication and code. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7254) Refactor AccessController ZK-mediated permissions cache into a generic mechanism
[ https://issues.apache.org/jira/browse/HBASE-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809589#comment-13809589 ] Andrew Purtell commented on HBASE-7254: --- Also my earlier objection to Curator is now removed, as noted on the other JIRA, because Curator has since entered and graduated ASF incubation to become Apache Curator. Refactor AccessController ZK-mediated permissions cache into a generic mechanism Key: HBASE-7254 URL: https://issues.apache.org/jira/browse/HBASE-7254 Project: HBase Issue Type: Task Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell After HBASE-5487 or HBASE-7212 goes in, we could replace the AccessController's permissions cache update via ZK using one of these more general frameworks, thus reducing functional duplication and code. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809595#comment-13809595 ] Nicolas Liochon commented on HBASE-9864: We have the cluster status, sent w/ a multicast message. It scales, and the server does not need to know about the client. ZK is interesting as it now has cheap local session (ZOOKEEPER-1147). It depends as well on the reliability we need, and if we need to see all states or not. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809599#comment-13809599 ] Andrew Purtell commented on HBASE-9864: --- See HBASE-7254 Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9045) Support Dictionary based Tag compression in HFiles
[
https://issues.apache.org/jira/browse/HBASE-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809605#comment-13809605
]
Hadoop QA commented on HBASE-9045:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611146/HBASE-9045_V3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7678//console
This message is automatically generated.
Support Dictionary based Tag compression in HFiles
--
Key: HBASE-9045
URL: https://issues.apache.org/jira/browse/HBASE-9045
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0
Attachments: HBASE-9045.patch, HBASE-9045_V2.patch,
HBASE-9045_V3.patch
Along with the DataBlockEncoding algorithms, Dictionary based Tag compression
can be done
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HBASE-7254) Refactor AccessController ZK-mediated permissions cache into a generic mechanism
[ https://issues.apache.org/jira/browse/HBASE-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-7254: - Assignee: (was: Andrew Purtell) Refactor AccessController ZK-mediated permissions cache into a generic mechanism Key: HBASE-7254 URL: https://issues.apache.org/jira/browse/HBASE-7254 Project: HBase Issue Type: Task Affects Versions: 0.98.0 Reporter: Andrew Purtell After HBASE-5487 or HBASE-7212 goes in, we could replace the AccessController's permissions cache update via ZK using one of these more general frameworks, thus reducing functional duplication and code. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9862) manage error per server and per region in the protobuffed client
[
https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809602#comment-13809602
]
Hadoop QA commented on HBASE-9862:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611147/9862.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7677//console
This message is automatically generated.
manage error per server and per region in the protobuffed client
Key: HBASE-9862
URL: https://issues.apache.org/jira/browse/HBASE-9862
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9862.v2.patch
The patch does not change anything else than the description says. The
changes are about extracting the common paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8369: - Attachment: hbase-8369_v6.patch Rebased patch MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v5.patch, hbase-8369_v6.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[
https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809624#comment-13809624
]
Hadoop QA commented on HBASE-8369:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611171/hbase-8369_v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:red}-1 hadoop1.0{color}. The patch failed to compile against the
hadoop 1.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7680//console
This message is automatically generated.
MapReduce over snapshot files
-
Key: HBASE-8369
URL: https://issues.apache.org/jira/browse/HBASE-8369
Project: HBase
Issue Type: New Feature
Components: mapreduce, snapshots
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0
Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch,
HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch,
HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch,
HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v5.patch,
hbase-8369_v6.patch
The idea is to add an InputFormat, which can run the mapreduce job over
snapshot files directly bypassing hbase server layer. The IF is similar in
usage to TableInputFormat, taking a Scan object from the user, but instead of
running from an online table, it runs from a table snapshot. We do one split
per region in the snapshot, and open an HRegion inside the RecordReader. A
RegionScanner is used internally for doing the scan without any HRegionServer
bits.
Users have been asking and searching for ways to run MR jobs by reading
directly from hfiles, so this allows new use cases if reading from stale data
is ok:
- Take snapshots periodically, and run MR jobs only on snapshots.
- Export snapshots to remote hdfs cluster, run the MR jobs at that cluster
without HBase cluster.
- (Future use case) Combine snapshot data with online hbase data: Scan from
yesterday's snapshot, but read today's data from online hbase cluster.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-7662) [Per-KV security] Store and apply per cell ACLs into/from KeyValue tags
[ https://issues.apache.org/jira/browse/HBASE-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7662: -- Attachment: 7662.patch Attached updated patch addressing review comments. [Per-KV security] Store and apply per cell ACLs into/from KeyValue tags --- Key: HBASE-7662 URL: https://issues.apache.org/jira/browse/HBASE-7662 Project: HBase Issue Type: Sub-task Components: Coprocessors, security Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: 7662.patch, 7662.patch We can improve the performance of per-cell authorization if the read of the cell ACL, if any, is combined with the sequential read of the cell data already in progress. When tags are inlined with KVs in block encoding (see HBASE-7448, and more generally HBASE-7233), we can use them to carry cell ACLs instead of using out-of-line storage (HBASE-7661) for that purpose. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9862) manage error per server and per region in the protobuffed client
[ https://issues.apache.org/jira/browse/HBASE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809663#comment-13809663 ] stack commented on HBASE-9862: -- +1 Throwing a few exceptions it keeps chugging along so basic operation seems fine. manage error per server and per region in the protobuffed client Key: HBASE-9862 URL: https://issues.apache.org/jira/browse/HBASE-9862 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9862.v2.patch The patch does not change anything else than the description says. The changes are about extracting the common paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9563) Autorestart doesn't work if zkcleaner fails
[ https://issues.apache.org/jira/browse/HBASE-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809673#comment-13809673 ] Enis Soztutar commented on HBASE-9563: -- bq. It's still an issue that the zkCleaner which should only be an optimization is getting in the way of starting but it doesn't hamper testing. That is already fixed with the committed patch. no? bq. That we'll just fail the disable? I was hoping to understand the sequencing that had us lose the .tableinfo file. Sorry, I did not get the full context. This issue is about master not starting due to failing to clean znode right? Is there a related issue for .tableinfo loss? Autorestart doesn't work if zkcleaner fails --- Key: HBASE-9563 URL: https://issues.apache.org/jira/browse/HBASE-9563 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: stack Priority: Critical Fix For: 0.98.0, 0.96.1 Attachments: 9563.txt, 9563v2.txt, categorization.txt, categorization.txt I've seen this several times where a master didn't autorestart because zk cleaner failed. We should still restart the daemon even if it's not possible to clean the zk nodes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9047) Tool to handle finishing replication when the cluster is offline
[
https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809694#comment-13809694
]
Hadoop QA commented on HBASE-9047:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611148/HBASE-9047-trunk-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7679//console
This message is automatically generated.
Tool to handle finishing replication when the cluster is offline
Key: HBASE-9047
URL: https://issues.apache.org/jira/browse/HBASE-9047
Project: HBase
Issue Type: New Feature
Affects Versions: 0.96.0
Reporter: Jean-Daniel Cryans
Assignee: Demai Ni
Fix For: 0.98.0
Attachments: HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch,
HBASE-9047-trunk-v1.patch, HBASE-9047-trunk-v2.patch,
HBASE-9047-trunk-v3.patch
We're having a discussion on the mailing list about replicating the data on a
cluster that was shut down in an offline fashion. The motivation could be
that you don't want to bring HBase back up but still need that data on the
slave.
So I have this idea of a tool that would be running on the master cluster
while it is down, although it could also run at any time. Basically it would
be able to read the replication state of each master region server, finish
replicating what's missing to all the slave, and then clear that state in
zookeeper.
The code that handles replication does most of that already, see
ReplicationSourceManager and ReplicationSource. Basically when
ReplicationSourceManager.init() is called, it will check all the queues in ZK
and try to grab those that aren't attached to a region server. If the whole
cluster is down, it will grab all of them.
The beautiful thing here is that you could start that tool on all your
machines and the load will be spread out, but that might not be a big concern
if replication wasn't lagging since it would take a few seconds to finish
replicating the missing data for each region server.
I'm guessing when starting ReplicationSourceManager you'd give it a fake
region server ID, and you'd tell it not to start its own source.
FWIW
[jira] [Commented] (HBASE-8552) fix coverage org.apache.hadoop.hbase.rest.filter
[ https://issues.apache.org/jira/browse/HBASE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809707#comment-13809707 ] Hudson commented on HBASE-8552: --- SUCCESS: Integrated in HBase-TRUNK #4656 (See [https://builds.apache.org/job/HBase-TRUNK/4656/]) HBASE-8552: fix coverage org.apache.hadoop.hbase.rest.filter(by Andrey Klochkov) (jeffreyz: rev 1537197) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java fix coverage org.apache.hadoop.hbase.rest.filter - Key: HBASE-8552 URL: https://issues.apache.org/jira/browse/HBASE-8552 Project: HBase Issue Type: Test Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8552-0.94.patch, HBASE-8552-trunk--n2.patch, HBASE-8552-trunk--n3.patch, HBASE-8552-trunk.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9822) IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
[
https://issues.apache.org/jira/browse/HBASE-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809708#comment-13809708
]
Hudson commented on HBASE-9822:
---
SUCCESS: Integrated in HBase-TRUNK #4656 (See
[https://builds.apache.org/job/HBase-TRUNK/4656/])
HBASE-9822: IntegrationTestLazyCfLoading failed occasionally in a secure
enviroment (jeffreyz: rev 1537233)
*
/hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestLazyCfLoading.java
IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
---
Key: HBASE-9822
URL: https://issues.apache.org/jira/browse/HBASE-9822
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Trivial
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9822.patch
This test case failed in a secure deployment once with the following error.
It's due to a race condition between writers starts writes and table ACLs
propagation to region servers.
{code}
2013-10-14 13:03:32,185 ERROR [HBaseWriterThread_8]
util.MultiThreadedWriterBase: Failed to insert: 10 after 167ms; region
information: cached:
region=IntegrationTestLazyCfLoading,bffd,1381755808862.456a11d22693f7dc27763c32e55521a8.,
hostname=gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694,
seqNum=1; cache is up to date; errors: exception from
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal:60020 for
d3d9446802a44259755d38e6d163e820-10
E org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (table=IntegrationTestLazyCfLoading, family: essential:filter,
action=WRITE)
{code}
Writes were sent at 13:03:32,032
{code}
2013-10-14 13:03:32,032 WARN [htable-pool11-t1] client.AsyncProcess: Attempt
#1/35 failed for 1 ops on
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694 NOT
resubmitting.
{code}
While the permission propagation happened at 13:03:32,109 on region server
{code}
2013-10-14 13:03:32,109 DEBUG [regionserver60020-EventThread]
access.ZKPermissionWatcher: Updating permissions cache from node
IntegrationTestLazyCfLoading with data:
PBUF\x0AA\x0A\x06hrt_qa\x127\x08\x033\x0A'\x0A\x07default\x12\x1CIntegrationTestLazyCfLoading
\x00 \x01 \x02 \x03 \x04
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8552) fix coverage org.apache.hadoop.hbase.rest.filter
[ https://issues.apache.org/jira/browse/HBASE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809732#comment-13809732 ] Hudson commented on HBASE-8552: --- FAILURE: Integrated in HBase-0.94 #1190 (See [https://builds.apache.org/job/HBase-0.94/1190/]) HBASE-8552: fix coverage org.apache.hadoop.hbase.rest.filter(by Andrey Klochkov) (jeffreyz: rev 1537199) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java fix coverage org.apache.hadoop.hbase.rest.filter - Key: HBASE-8552 URL: https://issues.apache.org/jira/browse/HBASE-8552 Project: HBase Issue Type: Test Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8552-0.94.patch, HBASE-8552-trunk--n2.patch, HBASE-8552-trunk--n3.patch, HBASE-8552-trunk.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8557) fix coverage org.apache.hadoop.hbase.rest.metrics
[ https://issues.apache.org/jira/browse/HBASE-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809733#comment-13809733 ] Hudson commented on HBASE-8557: --- FAILURE: Integrated in HBase-0.94 #1190 (See [https://builds.apache.org/job/HBase-0.94/1190/]) HBASE-8557: fix coverage org.apache.hadoop.hbase.rest.metrics(by Aleksey Gorshkov) (jeffreyz: rev 1537213) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/rest/TestRESTMetrics.java fix coverage org.apache.hadoop.hbase.rest.metrics - Key: HBASE-8557 URL: https://issues.apache.org/jira/browse/HBASE-8557 Project: HBase Issue Type: Test Affects Versions: 0.94.9 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 0.94.14 Attachments: HBASE-8557-0.94.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8543) fix coverage org.apache.hadoop.hbase.rest.client
[ https://issues.apache.org/jira/browse/HBASE-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809758#comment-13809758 ] Enis Soztutar commented on HBASE-8543: -- Hey [~aklochkov] this looks ready to go in. Can you update the 0.94 patch as well, so that we can commit them together. Thanks. fix coverage org.apache.hadoop.hbase.rest.client Key: HBASE-8543 URL: https://issues.apache.org/jira/browse/HBASE-8543 Project: HBase Issue Type: Test Affects Versions: 0.94.8, 0.95.2 Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8543-0.94--N2.patch, HBASE-8543-0.94.patch, HBASE-8543-trunk--N2.patch, HBASE-8543-trunk--n3.patch, HBASE-8543-trunk--n4.patch, HBASE-8543-trunk--n5.patch, HBASE-8543-trunk--n6.patch, HBASE-8543-trunk.patch fix coverage org.apache.hadoop.hbase.rest.client -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9563) Autorestart doesn't work if zkcleaner fails
[ https://issues.apache.org/jira/browse/HBASE-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809757#comment-13809757 ] stack commented on HBASE-9563: -- [~enis] Wrong issue. Ignore my comment. [~eclark] What you think of Enis remark above? Autorestart doesn't work if zkcleaner fails --- Key: HBASE-9563 URL: https://issues.apache.org/jira/browse/HBASE-9563 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: stack Priority: Critical Fix For: 0.98.0, 0.96.1 Attachments: 9563.txt, 9563v2.txt, categorization.txt, categorization.txt I've seen this several times where a master didn't autorestart because zk cleaner failed. We should still restart the daemon even if it's not possible to clean the zk nodes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9860:
--
Status: Patch Available (was: Open)
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9860:
--
Attachment: 9860-v1.txt
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-9860:
-
Assignee: Ted Yu
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9865) WALEdit.heapSize() is incorrect in certain replication scenarios which may cause RegionServers to go OOM
churro morales created HBASE-9865:
-
Summary: WALEdit.heapSize() is incorrect in certain replication
scenarios which may cause RegionServers to go OOM
Key: HBASE-9865
URL: https://issues.apache.org/jira/browse/HBASE-9865
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.5
Reporter: churro morales
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM.
A little background on this issue. We noticed that our source replication
regionservers would get into gc storms and sometimes even OOM.
We noticed a case where it showed that there were around 25k WALEdits to
replicate, each one with an ArrayList of KeyValues. The array list had a
capacity of around 90k (using 350KB of heap memory) but had around 6 non null
entries.
When the ReplicationSource.readAllEntriesToReplicateOrNextFile() gets a WALEdit
it removes all kv's that are scoped other than local.
But in doing so we don't account for the capacity of the ArrayList when
determining heapSize for a WALEdit. The logic for shipping a batch is whether
you have hit a size capacity or number of entries capacity.
Therefore if have a WALEdit with 25k entries and suppose all are removed:
The size of the arrayList is 0 (we don't even count the collection's heap size
currently) but the capacity is ignored.
This will yield a heapSize() of 0 bytes while in the best case it would be at
least 10 bytes (provided you pass initialCapacity and you have 32 bit JVM)
I have some ideas on how to address this problem and want to know everyone's
thoughts:
1. We use a probabalistic counter such as HyperLogLog and create something like:
* class CapacityEstimateArrayList implements ArrayList
** this class overrides all additive methods to update the
probabalistic counts
** it includes one additional method called estimateCapacity
(we would take estimateCapacity - size() and fill in sizes for all references)
* Then we can do something like this in WALEdit.heapSize:
{code}
public long heapSize() {
long ret = ClassSize.ARRAYLIST;
for (KeyValue kv : kvs) {
ret += kv.heapSize();
}
long nullEntriesEstimate = kvs.getCapacityEstimate() - kvs.size();
ret += ClassSize.align(nullEntriesEstimate * ClassSize.REFERENCE);
if (scopes != null) {
ret += ClassSize.TREEMAP;
ret += ClassSize.align(scopes.size() * ClassSize.MAP_ENTRY);
// TODO this isn't quite right, need help here
}
return ret;
}
{code}
2. In ReplicationSource.removeNonReplicableEdits() we know the size of the
array originally, and we provide some percentage threshold. When that
threshold is met (50% of the entries have been removed) we can call
kvs.trimToSize()
3. in the heapSize() method for WALEdit we could use reflection (Please don't
shoot me for this) to grab the actual capacity of the list. Doing something
like this:
{code}
public int getArrayListCapacity() {
try {
Field f = ArrayList.class.getDeclaredField(elementData);
f.setAccessible(true);
return ((Object[]) f.get(kvs)).length;
} catch (Exception e) {
log.warn(Exception in trying to get capacity on ArrayList, e);
return kvs.size();
}
{code}
I am partial to (1) using HyperLogLog and creating a CapacityEstimateArrayList,
this is reusable throughout the code for other classes that implement HeapSize
which contains ArrayLists. The memory footprint is very small and it is very
fast. The issue is that this is an estimate, although we can configure the
precision we most likely always be conservative. The estimateCapacity will
always be less than the actualCapacity, but it will be close. I think that
putting the logic in removeNonReplicableEdits will work, but this only solves
the heapSize problem in this particular scenario. Solution 3 is slow and
horrible but that gives us the exact answer.
I would love to hear if anyone else has any other ideas on how to remedy this
problem? I have code for trunk and 0.94 for all 3 ideas and can provide a
patch if the community thinks any of these approaches is a viable one.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8543) fix coverage org.apache.hadoop.hbase.rest.client
[ https://issues.apache.org/jira/browse/HBASE-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809774#comment-13809774 ] Andrey Klochkov commented on HBASE-8543: Yes, I'll update the 0.94 patch shortly. fix coverage org.apache.hadoop.hbase.rest.client Key: HBASE-8543 URL: https://issues.apache.org/jira/browse/HBASE-8543 Project: HBase Issue Type: Test Affects Versions: 0.94.8, 0.95.2 Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8543-0.94--N2.patch, HBASE-8543-0.94.patch, HBASE-8543-trunk--N2.patch, HBASE-8543-trunk--n3.patch, HBASE-8543-trunk--n4.patch, HBASE-8543-trunk--n5.patch, HBASE-8543-trunk--n6.patch, HBASE-8543-trunk.patch fix coverage org.apache.hadoop.hbase.rest.client -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-9659) some integration tests can no longer be run using maven
[ https://issues.apache.org/jira/browse/HBASE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HBASE-9659. - Resolution: Fixed some integration tests can no longer be run using maven --- Key: HBASE-9659 URL: https://issues.apache.org/jira/browse/HBASE-9659 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9659-v0-0.96.patch, HBASE-9659-v0.patch When I run mvn test (or verify) - Dtest=IntegrationTestIngest, the test fails instantly, seemingly because initialization doesn't run. -I am assuming junit is not picking before-after methods from the superclass-, could be some other issue. Also, if it does run, it won't be very useful because it runs with calm monkey by default. We need to detect being run locally rather than as AbstractHBaseTool (probably any time JUnit-annotated methods like Before are called), and set up a different chaos monkey, such as an old default one. May also apply to other tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9866) Support the mode where REST server authorizes proxy users
Devaraj Das created HBASE-9866:
--
Summary: Support the mode where REST server authorizes proxy users
Key: HBASE-9866
URL: https://issues.apache.org/jira/browse/HBASE-9866
Project: HBase
Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.96.1
In one use case, someone was trying to authorize with the REST server as a
proxy user. That mode is not supported today.
The curl request would be something like (assuming SPNEGO auth) -
{noformat}
curl -i --negotiate -u : http://HOST:PORT/version/cluster?doas=USER
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9295) Allow test-patch.sh to detect TreeMap keyed by byte[] which doesn't use proper comparator
[
https://issues.apache.org/jira/browse/HBASE-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809790#comment-13809790
]
Ted Yu commented on HBASE-9295:
---
Can I interpret the above as +1 ?
Allow test-patch.sh to detect TreeMap keyed by byte[] which doesn't use
proper comparator
-
Key: HBASE-9295
URL: https://issues.apache.org/jira/browse/HBASE-9295
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9295-v1.txt, 9295-v2.txt
There were two recent bug fixes (HBASE-9285 and HBASE-9238) for the case
where the TreeMap keyed by byte[] doesn't use proper comparator:
{code}
new TreeMapbyte[], ...()
{code}
test-patch.sh should be able to detect this situation and report accordingly.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9865) WALEdit.heapSize() is incorrect in certain replication scenarios which may cause RegionServers to go OOM
[
https://issues.apache.org/jira/browse/HBASE-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809789#comment-13809789
]
Ted Yu commented on HBASE-9865:
---
You may have seen this:
http://stackoverflow.com/questions/2497063/how-to-get-the-capacity-of-the-arraylist-in-java
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM
Key: HBASE-9865
URL: https://issues.apache.org/jira/browse/HBASE-9865
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5, 0.95.0
Reporter: churro morales
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM.
A little background on this issue. We noticed that our source replication
regionservers would get into gc storms and sometimes even OOM.
We noticed a case where it showed that there were around 25k WALEdits to
replicate, each one with an ArrayList of KeyValues. The array list had a
capacity of around 90k (using 350KB of heap memory) but had around 6 non null
entries.
When the ReplicationSource.readAllEntriesToReplicateOrNextFile() gets a
WALEdit it removes all kv's that are scoped other than local.
But in doing so we don't account for the capacity of the ArrayList when
determining heapSize for a WALEdit. The logic for shipping a batch is
whether you have hit a size capacity or number of entries capacity.
Therefore if have a WALEdit with 25k entries and suppose all are removed:
The size of the arrayList is 0 (we don't even count the collection's heap
size currently) but the capacity is ignored.
This will yield a heapSize() of 0 bytes while in the best case it would be at
least 10 bytes (provided you pass initialCapacity and you have 32 bit
JVM)
I have some ideas on how to address this problem and want to know everyone's
thoughts:
1. We use a probabalistic counter such as HyperLogLog and create something
like:
* class CapacityEstimateArrayList implements ArrayList
** this class overrides all additive methods to update the
probabalistic counts
** it includes one additional method called estimateCapacity
(we would take estimateCapacity - size() and fill in sizes for all references)
* Then we can do something like this in WALEdit.heapSize:
{code}
public long heapSize() {
long ret = ClassSize.ARRAYLIST;
for (KeyValue kv : kvs) {
ret += kv.heapSize();
}
long nullEntriesEstimate = kvs.getCapacityEstimate() - kvs.size();
ret += ClassSize.align(nullEntriesEstimate * ClassSize.REFERENCE);
if (scopes != null) {
ret += ClassSize.TREEMAP;
ret += ClassSize.align(scopes.size() * ClassSize.MAP_ENTRY);
// TODO this isn't quite right, need help here
}
return ret;
}
{code}
2. In ReplicationSource.removeNonReplicableEdits() we know the size of the
array originally, and we provide some percentage threshold. When that
threshold is met (50% of the entries have been removed) we can call
kvs.trimToSize()
3. in the heapSize() method for WALEdit we could use reflection (Please don't
shoot me for this) to grab the actual capacity of the list. Doing something
like this:
{code}
public int getArrayListCapacity() {
try {
Field f = ArrayList.class.getDeclaredField(elementData);
f.setAccessible(true);
return ((Object[]) f.get(kvs)).length;
} catch (Exception e) {
log.warn(Exception in trying to get capacity on ArrayList, e);
return kvs.size();
}
{code}
I am partial to (1) using HyperLogLog and creating a
CapacityEstimateArrayList, this is reusable throughout the code for other
classes that implement HeapSize which contains ArrayLists. The memory
footprint is very small and it is very fast. The issue is that this is an
estimate, although we can configure the precision we most likely always be
conservative. The estimateCapacity will always be less than the
actualCapacity, but it will be close. I think that putting the logic in
removeNonReplicableEdits will work, but this only solves the heapSize problem
in this particular scenario. Solution 3 is slow and horrible but that gives
us the exact answer.
I would love to hear if anyone else has any other ideas on how to remedy this
problem? I have code for trunk and 0.94 for all 3 ideas and can provide a
patch if the community thinks any of these approaches is a viable one.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8541) implement flush-into-stripes in stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809791#comment-13809791 ] Enis Soztutar commented on HBASE-8541: -- forwarding my +1 from RB. implement flush-into-stripes in stripe compactions -- Key: HBASE-8541 URL: https://issues.apache.org/jira/browse/HBASE-8541 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-v0.patch, HBASE-8541-v1.patch, HBASE-8541-v2.patch, HBASE-8541-v3.patch, HBASE-8541-v4.patch, HBASE-8541-v5.patch Flush will be able to flush into multiple files under this design, avoiding L0 I/O amplification. I have the patch which is missing just one feature - support for concurrent flushes and stripe changes. This can be done via extensive try-locking of stripe changes and flushes, or advisory flags without blocking flushes, dumping conflicting flushes into L0 in case of (very rare) collisions. For file loading for the latter, a set-cover-like problem needs to be solved to determine optimal stripes. That will also address Jimmy's concern of getting rid of metadata, btw. However currently I don't have time for that. I plan to attach the try-locking patch first, but this won't happen for a couple weeks probably and should not block main reviews. Hopefully this will be added on top of main reviews. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809804#comment-13809804
]
Jimmy Xiang commented on HBASE-9860:
If hri is not null, hri.getTable() should not be null, right? Is it ok to
remove hri.getTable() != null?
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8543) fix coverage org.apache.hadoop.hbase.rest.client
[ https://issues.apache.org/jira/browse/HBASE-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated HBASE-8543: --- Attachment: HBASE-8543-0.94--n6.patch updating the 0.94 patch fix coverage org.apache.hadoop.hbase.rest.client Key: HBASE-8543 URL: https://issues.apache.org/jira/browse/HBASE-8543 Project: HBase Issue Type: Test Affects Versions: 0.94.8, 0.95.2 Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8543-0.94--N2.patch, HBASE-8543-0.94--n6.patch, HBASE-8543-0.94.patch, HBASE-8543-trunk--N2.patch, HBASE-8543-trunk--n3.patch, HBASE-8543-trunk--n4.patch, HBASE-8543-trunk--n5.patch, HBASE-8543-trunk--n6.patch, HBASE-8543-trunk.patch fix coverage org.apache.hadoop.hbase.rest.client -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8543) fix coverage org.apache.hadoop.hbase.rest.client
[
https://issues.apache.org/jira/browse/HBASE-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809807#comment-13809807
]
Hadoop QA commented on HBASE-8543:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611220/HBASE-8543-0.94--n6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7682//console
This message is automatically generated.
fix coverage org.apache.hadoop.hbase.rest.client
Key: HBASE-8543
URL: https://issues.apache.org/jira/browse/HBASE-8543
Project: HBase
Issue Type: Test
Affects Versions: 0.94.8, 0.95.2
Reporter: Aleksey Gorshkov
Assignee: Andrey Klochkov
Fix For: 0.98.0, 0.96.1, 0.94.14
Attachments: HBASE-8543-0.94--N2.patch, HBASE-8543-0.94--n6.patch,
HBASE-8543-0.94.patch, HBASE-8543-trunk--N2.patch,
HBASE-8543-trunk--n3.patch, HBASE-8543-trunk--n4.patch,
HBASE-8543-trunk--n5.patch, HBASE-8543-trunk--n6.patch, HBASE-8543-trunk.patch
fix coverage org.apache.hadoop.hbase.rest.client
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9865) WALEdit.heapSize() is incorrect in certain replication scenarios which may cause RegionServers to go OOM
[
https://issues.apache.org/jira/browse/HBASE-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809826#comment-13809826
]
Lars Hofhansl commented on HBASE-9865:
--
So if I understand this correctly the gist of the problem is that we're reusing
the WALEdits (see ReplicationHLogReaderManager.readNextAndSetPosition reusing
entriesArray), and thus their internal kvs ArrayList can only grow and never
shrink. Some of the WALEdits can have a large list of KVs if they were created
by a batch operation.
Calculating the correct heapsize would be pasting over the problem (I think).
We should ensure that at some point we can reduce the capacity of the internal
kvs array of the reused WALEdit.
The right point seems to be where the WALEdits are reused for reading. We could
look at WALEdit.readFields. There we clear the kvs list (which does not reduce
its capacity of course).
It's not immediately clear to me what the correct solution is. We do not always
want to reset the capacity since that is expensive too and the next time we'll
need to recreate the internal array.
Upon reuse in WALEdit.readFields, we could check the current size before we
call clear, then if the new size is (say) twice the required size call
trimToSize() (which will set capacity to 0). I also think a WALEdit should
start with the kvs ArrayList of capacity 1 (rather than the default of 16).
Or we could a probabilistic approach and reset the ArrayList with a probability
proportional to the previous size.
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM
Key: HBASE-9865
URL: https://issues.apache.org/jira/browse/HBASE-9865
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5, 0.95.0
Reporter: churro morales
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM.
A little background on this issue. We noticed that our source replication
regionservers would get into gc storms and sometimes even OOM.
We noticed a case where it showed that there were around 25k WALEdits to
replicate, each one with an ArrayList of KeyValues. The array list had a
capacity of around 90k (using 350KB of heap memory) but had around 6 non null
entries.
When the ReplicationSource.readAllEntriesToReplicateOrNextFile() gets a
WALEdit it removes all kv's that are scoped other than local.
But in doing so we don't account for the capacity of the ArrayList when
determining heapSize for a WALEdit. The logic for shipping a batch is
whether you have hit a size capacity or number of entries capacity.
Therefore if have a WALEdit with 25k entries and suppose all are removed:
The size of the arrayList is 0 (we don't even count the collection's heap
size currently) but the capacity is ignored.
This will yield a heapSize() of 0 bytes while in the best case it would be at
least 10 bytes (provided you pass initialCapacity and you have 32 bit
JVM)
I have some ideas on how to address this problem and want to know everyone's
thoughts:
1. We use a probabalistic counter such as HyperLogLog and create something
like:
* class CapacityEstimateArrayList implements ArrayList
** this class overrides all additive methods to update the
probabalistic counts
** it includes one additional method called estimateCapacity
(we would take estimateCapacity - size() and fill in sizes for all references)
* Then we can do something like this in WALEdit.heapSize:
{code}
public long heapSize() {
long ret = ClassSize.ARRAYLIST;
for (KeyValue kv : kvs) {
ret += kv.heapSize();
}
long nullEntriesEstimate = kvs.getCapacityEstimate() - kvs.size();
ret += ClassSize.align(nullEntriesEstimate * ClassSize.REFERENCE);
if (scopes != null) {
ret += ClassSize.TREEMAP;
ret += ClassSize.align(scopes.size() * ClassSize.MAP_ENTRY);
// TODO this isn't quite right, need help here
}
return ret;
}
{code}
2. In ReplicationSource.removeNonReplicableEdits() we know the size of the
array originally, and we provide some percentage threshold. When that
threshold is met (50% of the entries have been removed) we can call
kvs.trimToSize()
3. in the heapSize() method for WALEdit we could use reflection (Please don't
shoot me for this) to grab the actual capacity of the list. Doing something
like this:
{code}
public int getArrayListCapacity() {
try {
Field f = ArrayList.class.getDeclaredField(elementData);
f.setAccessible(true);
return ((Object[]) f.get(kvs)).length;
} catch (Exception e) {
log.warn(Exception in trying to get
[jira] [Updated] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9860:
--
Attachment: 9860-v2.txt
Patch v2 addresses comment above.
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt, 9860-v2.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9865) WALEdit.heapSize() is incorrect in certain replication scenarios which may cause RegionServers to go OOM
[
https://issues.apache.org/jira/browse/HBASE-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9865:
-
Attachment: 9865-sample.txt
Something like this, maybe? Resets a WALEdit's size when we detect 99%
wastage. This does not have to be perfect, as long as we eventually clear out
the extreme outliers.
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM
Key: HBASE-9865
URL: https://issues.apache.org/jira/browse/HBASE-9865
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5, 0.95.0
Reporter: churro morales
Attachments: 9865-sample.txt
WALEdit.heapSize() is incorrect in certain replication scenarios which may
cause RegionServers to go OOM.
A little background on this issue. We noticed that our source replication
regionservers would get into gc storms and sometimes even OOM.
We noticed a case where it showed that there were around 25k WALEdits to
replicate, each one with an ArrayList of KeyValues. The array list had a
capacity of around 90k (using 350KB of heap memory) but had around 6 non null
entries.
When the ReplicationSource.readAllEntriesToReplicateOrNextFile() gets a
WALEdit it removes all kv's that are scoped other than local.
But in doing so we don't account for the capacity of the ArrayList when
determining heapSize for a WALEdit. The logic for shipping a batch is
whether you have hit a size capacity or number of entries capacity.
Therefore if have a WALEdit with 25k entries and suppose all are removed:
The size of the arrayList is 0 (we don't even count the collection's heap
size currently) but the capacity is ignored.
This will yield a heapSize() of 0 bytes while in the best case it would be at
least 10 bytes (provided you pass initialCapacity and you have 32 bit
JVM)
I have some ideas on how to address this problem and want to know everyone's
thoughts:
1. We use a probabalistic counter such as HyperLogLog and create something
like:
* class CapacityEstimateArrayList implements ArrayList
** this class overrides all additive methods to update the
probabalistic counts
** it includes one additional method called estimateCapacity
(we would take estimateCapacity - size() and fill in sizes for all references)
* Then we can do something like this in WALEdit.heapSize:
{code}
public long heapSize() {
long ret = ClassSize.ARRAYLIST;
for (KeyValue kv : kvs) {
ret += kv.heapSize();
}
long nullEntriesEstimate = kvs.getCapacityEstimate() - kvs.size();
ret += ClassSize.align(nullEntriesEstimate * ClassSize.REFERENCE);
if (scopes != null) {
ret += ClassSize.TREEMAP;
ret += ClassSize.align(scopes.size() * ClassSize.MAP_ENTRY);
// TODO this isn't quite right, need help here
}
return ret;
}
{code}
2. In ReplicationSource.removeNonReplicableEdits() we know the size of the
array originally, and we provide some percentage threshold. When that
threshold is met (50% of the entries have been removed) we can call
kvs.trimToSize()
3. in the heapSize() method for WALEdit we could use reflection (Please don't
shoot me for this) to grab the actual capacity of the list. Doing something
like this:
{code}
public int getArrayListCapacity() {
try {
Field f = ArrayList.class.getDeclaredField(elementData);
f.setAccessible(true);
return ((Object[]) f.get(kvs)).length;
} catch (Exception e) {
log.warn(Exception in trying to get capacity on ArrayList, e);
return kvs.size();
}
{code}
I am partial to (1) using HyperLogLog and creating a
CapacityEstimateArrayList, this is reusable throughout the code for other
classes that implement HeapSize which contains ArrayLists. The memory
footprint is very small and it is very fast. The issue is that this is an
estimate, although we can configure the precision we most likely always be
conservative. The estimateCapacity will always be less than the
actualCapacity, but it will be close. I think that putting the logic in
removeNonReplicableEdits will work, but this only solves the heapSize problem
in this particular scenario. Solution 3 is slow and horrible but that gives
us the exact answer.
I would love to hear if anyone else has any other ideas on how to remedy this
problem? I have code for trunk and 0.94 for all 3 ideas and can provide a
patch if the community thinks any of these approaches is a viable one.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809829#comment-13809829 ] Lars Hofhansl commented on HBASE-8369: -- If you make a 0.94 patch I'll offer to try on a real cluster :) MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v5.patch, hbase-8369_v6.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809831#comment-13809831
]
Hadoop QA commented on HBASE-9860:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611211/9860-v1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7681//console
This message is automatically generated.
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt, 9860-v2.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809837#comment-13809837
]
Jimmy Xiang commented on HBASE-9860:
+1
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt, 9860-v2.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9822) IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
[
https://issues.apache.org/jira/browse/HBASE-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809838#comment-13809838
]
Hudson commented on HBASE-9822:
---
FAILURE: Integrated in hbase-0.96 #171 (See
[https://builds.apache.org/job/hbase-0.96/171/])
HBASE-9822: IntegrationTestLazyCfLoading failed occasionally in a secure
enviroment (jeffreyz: rev 1537239)
*
/hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestLazyCfLoading.java
IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
---
Key: HBASE-9822
URL: https://issues.apache.org/jira/browse/HBASE-9822
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Trivial
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9822.patch
This test case failed in a secure deployment once with the following error.
It's due to a race condition between writers starts writes and table ACLs
propagation to region servers.
{code}
2013-10-14 13:03:32,185 ERROR [HBaseWriterThread_8]
util.MultiThreadedWriterBase: Failed to insert: 10 after 167ms; region
information: cached:
region=IntegrationTestLazyCfLoading,bffd,1381755808862.456a11d22693f7dc27763c32e55521a8.,
hostname=gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694,
seqNum=1; cache is up to date; errors: exception from
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal:60020 for
d3d9446802a44259755d38e6d163e820-10
E org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (table=IntegrationTestLazyCfLoading, family: essential:filter,
action=WRITE)
{code}
Writes were sent at 13:03:32,032
{code}
2013-10-14 13:03:32,032 WARN [htable-pool11-t1] client.AsyncProcess: Attempt
#1/35 failed for 1 ops on
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694 NOT
resubmitting.
{code}
While the permission propagation happened at 13:03:32,109 on region server
{code}
2013-10-14 13:03:32,109 DEBUG [regionserver60020-EventThread]
access.ZKPermissionWatcher: Updating permissions cache from node
IntegrationTestLazyCfLoading with data:
PBUF\x0AA\x0A\x06hrt_qa\x127\x08\x033\x0A'\x0A\x07default\x12\x1CIntegrationTestLazyCfLoading
\x00 \x01 \x02 \x03 \x04
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9866) Support the mode where REST server authorizes proxy users
[
https://issues.apache.org/jira/browse/HBASE-9866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-9866:
---
Attachment: 9866-1.txt
First version of the patch.
Support the mode where REST server authorizes proxy users
-
Key: HBASE-9866
URL: https://issues.apache.org/jira/browse/HBASE-9866
Project: HBase
Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.96.1
Attachments: 9866-1.txt
In one use case, someone was trying to authorize with the REST server as a
proxy user. That mode is not supported today.
The curl request would be something like (assuming SPNEGO auth) -
{noformat}
curl -i --negotiate -u : http://HOST:PORT/version/cluster?doas=USER
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-3787) Increment is non-idempotent but client retries RPC
[ https://issues.apache.org/jira/browse/HBASE-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809853#comment-13809853 ] Sergey Shelukhin commented on HBASE-3787: - trying to grok rpc changes, sent some email to Nicolas. Will maybe update by EOW Increment is non-idempotent but client retries RPC -- Key: HBASE-3787 URL: https://issues.apache.org/jira/browse/HBASE-3787 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.4, 0.95.2 Reporter: dhruba borthakur Assignee: Sergey Shelukhin Priority: Blocker Attachments: HBASE-3787-partial.patch, HBASE-3787-v0.patch, HBASE-3787-v1.patch, HBASE-3787-v2.patch, HBASE-3787-v3.patch, HBASE-3787-v4.patch, HBASE-3787-v5.patch, HBASE-3787-v5.patch The HTable.increment() operation is non-idempotent. The client retries the increment RPC a few times (as specified by configuration) before throwing an error to the application. This makes it possible that the same increment call be applied twice at the server. For increment operations, is it better to use HConnectionManager.getRegionServerWithoutRetries()? Another option would be to enhance the IPC module to make the RPC server correctly identify if the RPC is a retry attempt and handle accordingly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9822) IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
[
https://issues.apache.org/jira/browse/HBASE-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809857#comment-13809857
]
Sergey Shelukhin commented on HBASE-9822:
-
Hmm... doesn't this indicate a real (albeit small) problem?
Insufficient permission is not retriable so app code against HBase could run
into the same problem. So table is created but not really ready when create
ends.
IntegrationTestLazyCfLoading failed occasionally in a secure enviroment
---
Key: HBASE-9822
URL: https://issues.apache.org/jira/browse/HBASE-9822
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Trivial
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9822.patch
This test case failed in a secure deployment once with the following error.
It's due to a race condition between writers starts writes and table ACLs
propagation to region servers.
{code}
2013-10-14 13:03:32,185 ERROR [HBaseWriterThread_8]
util.MultiThreadedWriterBase: Failed to insert: 10 after 167ms; region
information: cached:
region=IntegrationTestLazyCfLoading,bffd,1381755808862.456a11d22693f7dc27763c32e55521a8.,
hostname=gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694,
seqNum=1; cache is up to date; errors: exception from
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal:60020 for
d3d9446802a44259755d38e6d163e820-10
E org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (table=IntegrationTestLazyCfLoading, family: essential:filter,
action=WRITE)
{code}
Writes were sent at 13:03:32,032
{code}
2013-10-14 13:03:32,032 WARN [htable-pool11-t1] client.AsyncProcess: Attempt
#1/35 failed for 1 ops on
gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694 NOT
resubmitting.
{code}
While the permission propagation happened at 13:03:32,109 on region server
{code}
2013-10-14 13:03:32,109 DEBUG [regionserver60020-EventThread]
access.ZKPermissionWatcher: Updating permissions cache from node
IntegrationTestLazyCfLoading with data:
PBUF\x0AA\x0A\x06hrt_qa\x127\x08\x033\x0A'\x0A\x07default\x12\x1CIntegrationTestLazyCfLoading
\x00 \x01 \x02 \x03 \x04
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9360) Enable 0.94 - 0.96 replication to minimize upgrade down time
[ https://issues.apache.org/jira/browse/HBASE-9360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809858#comment-13809858 ] Jeffrey Zhong commented on HBASE-9360: -- I did a prototype @https://github.com/hortonworks/HBaseReplicationBridgeServer and tested replication from a 0.94 cluster to a 0.96 cluster. The remaining work is to bring the slave cluster to a good base(which we're currently using CopyTable) when setting up replication. h6. Support import from a 0.94 export sequence file. Currently we PBed org.apache.hadoop.hbase.client.Result so a 0.96 cluster cannot import 0.94 exported files while we can easily add that support. (personal preferred option) h6. Use snapshot without any code changes: 1) Setup replication without starting replication bridge server so source cluster is queuing WALs 2) Use Snapshot to bring destination cluster to a good base 3) Starts replication bridge servers to drain WALs queued up from step 1 h6. Enable CopyTable against Replication Bridge Server This option is least desired one because it will involves significant code changes to have a faked root znode, support root table scan, support meta table scan, delete and multi command in replication bridge server. Enable 0.94 - 0.96 replication to minimize upgrade down time - Key: HBASE-9360 URL: https://issues.apache.org/jira/browse/HBASE-9360 Project: HBase Issue Type: Brainstorming Components: migration Affects Versions: 0.98.0, 0.96.0 Reporter: Jeffrey Zhong As we know 0.96 is a singularity release, as of today a 0.94 hbase user has to do in-place upgrade: make corresponding client changes, recompile client application code, fully shut down existing 0.94 hbase cluster, deploy 0.96 binary, run upgrade script and then start the upgraded cluster. You can image the down time will be extended if something is wrong in between. To minimize the down time, another possible way is to setup a secondary 0.96 cluster and then setup replication between the existing 0.94 cluster and the new 0.96 slave cluster. Once the 0.96 cluster is synced, a user can switch the traffic to the 0.96 cluster and decommission the old one. The ideal steps will be: 1) Setup a 0.96 cluster 2) Setup replication between a running 0.94 cluster to the newly created 0.96 cluster 3) Wait till they're in sync in replication 4) Starts duplicated writes to both 0.94 and 0.96 clusters(could stop relocation now) 5) Forward read traffic to the slave 0.96 cluster 6) After a certain period, stop writes to the original 0.94 cluster if everything is good and completes upgrade To get us there, there are two tasks: 1) Enable replication from 0.94 - 0.96 I've run the idea with [~jdcryans], [~devaraj] and [~ndimiduk]. Currently it seems the best approach is to build a very similar service or on top of https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep with support three commands replicateLogEntries, multi and delete. Inside the three commands, we just pass down the corresponding requests to the destination 0.96 cluster as a bridge. The reason to support the multi and delete is for CopyTable to copy data from a 0.94 cluster to a 0.96 one. The other approach is to provide limited support of 0.94 RPC protocol in 0.96. While an issue on this is that a 0.94 client needs to talk to zookeeper firstly before it can connect to a 0.96 region server. Therefore, we need a faked Zookeeper setup in front of a 0.96 cluster for a 0.94 client to connect. It may also pollute 0.96 code base with 0.94 RPC code. 2) To support writes to a 0.96 cluster and a 0.94 at the same time, we need to load both hbase clients into one single JVM using different class loader. Let me know if you think this is worth to do and any better approach we could take. Thanks! -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9860) Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809868#comment-13809868
]
Hadoop QA commented on HBASE-9860:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611224/9860-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7683//console
This message is automatically generated.
Intermittent TestHBaseFsck#testMissingRegionInfoQualifier failure due to
NullPointerException
-
Key: HBASE-9860
URL: https://issues.apache.org/jira/browse/HBASE-9860
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 9860-v1.txt, 9860-v2.txt
From
https://builds.apache.org/job/hbase-0.96-hadoop2/105/testReport/junit/org.apache.hadoop.hbase.util/TestHBaseFsck/testMissingRegionInfoQualifier/
:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.TestHBaseFsck$4.processRow(TestHBaseFsck.java:1891)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:179)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingRegionInfoQualifier(TestHBaseFsck.java:1887)
{code}
Looks like MetaScanner.getHRegionInfo() returned null below:
{code}
public boolean processRow(Result rowResult) throws IOException {
if(!MetaScanner.getHRegionInfo(rowResult).getTable().isSystemTable()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9000) Linear reseek in Memstore
[ https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809870#comment-13809870 ] Chao Shi commented on HBASE-9000: - bq. 'reseek to next row' gets slower with patch. If linear.search.limit is lowered, would the slow down be less ? Yes, the optimal value should vary case by case. I think a default value of 5 should not add much observable overhead. I think a long-term solution would be implementing our own version of lock-free skip-list, where we can access its higher-level of next pointers (i.e. to skip) from the current position. This patch could be a temporary solution for now, as it is very simple. Linear reseek in Memstore - Key: HBASE-9000 URL: https://issues.apache.org/jira/browse/HBASE-9000 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Shane Hogan Priority: Minor Fix For: 0.89-fb Attachments: hbase-9000-benchmark-program.patch, hbase-9000-port-fb.patch This is to address the linear reseek in MemStoreScanner. Currently reseek iterates over the kvset and the snapshot linearly by just calling next repeatedly. The new solution is to do this linear seek up to a configurable maximum amount of times then if the seek is not yet complete fall back to logarithmic seek. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9659) some integration tests can no longer be run using maven
[ https://issues.apache.org/jira/browse/HBASE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809878#comment-13809878 ] Hudson commented on HBASE-9659: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #107 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/107/]) HBASE-9659 some integration tests can no longer be run using maven (sershe: rev 1537352) * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestBase.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngest.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/MonkeyFactory.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java some integration tests can no longer be run using maven --- Key: HBASE-9659 URL: https://issues.apache.org/jira/browse/HBASE-9659 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9659-v0-0.96.patch, HBASE-9659-v0.patch When I run mvn test (or verify) - Dtest=IntegrationTestIngest, the test fails instantly, seemingly because initialization doesn't run. -I am assuming junit is not picking before-after methods from the superclass-, could be some other issue. Also, if it does run, it won't be very useful because it runs with calm monkey by default. We need to detect being run locally rather than as AbstractHBaseTool (probably any time JUnit-annotated methods like Before are called), and set up a different chaos monkey, such as an old default one. May also apply to other tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8552) fix coverage org.apache.hadoop.hbase.rest.filter
[ https://issues.apache.org/jira/browse/HBASE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809877#comment-13809877 ] Hudson commented on HBASE-8552: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #107 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/107/]) HBASE-8552: fix coverage org.apache.hadoop.hbase.rest.filter(by Andrey Klochkov) (jeffreyz: rev 1537198) * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestGZIPResponseWrapper.java fix coverage org.apache.hadoop.hbase.rest.filter - Key: HBASE-8552 URL: https://issues.apache.org/jira/browse/HBASE-8552 Project: HBase Issue Type: Test Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: HBASE-8552-0.94.patch, HBASE-8552-trunk--n2.patch, HBASE-8552-trunk--n3.patch, HBASE-8552-trunk.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
