[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770468#comment-13770468 ] Lars Hofhansl commented on HBASE-9534: -- Let's keep those members final. There's a slight bit more code duplication, but it's cleaner. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9534: - Status: Open (was: Patch Available) Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9534: - Attachment: 9534-0.94.txt Let me know what you think about these. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9534: - Attachment: 9534-trunk.txt Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9369) Add support for 1- and 2-byte integers in OrderedBytes and provide types
[ https://issues.apache.org/jira/browse/HBASE-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9369: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.96 and trunk. Thanks Liangliang! Add support for 1- and 2-byte integers in OrderedBytes and provide types Key: HBASE-9369 URL: https://issues.apache.org/jira/browse/HBASE-9369 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: He Liangliang Fix For: 0.98.0, 0.96.0 Attachments: 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, HBASE-9369.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9369) Add support for 1- and 2-byte integers in OrderedBytes and provide types
[ https://issues.apache.org/jira/browse/HBASE-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9369: Release Note: This change adds support for 8- and 16-bit integral types. It also alters the header byte written by OrderedBytes for NAN, FIXED_INT32, FIXED_INT64, TEXT, BLOB_VAR, and BLOB_COPY values. Data of those types written using a 0.95 release will be incompatible with this change. Add support for 1- and 2-byte integers in OrderedBytes and provide types Key: HBASE-9369 URL: https://issues.apache.org/jira/browse/HBASE-9369 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: He Liangliang Fix For: 0.98.0, 0.96.0 Attachments: 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, HBASE-9369.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9343) Implement stateless scanner for Stargate
[ https://issues.apache.org/jira/browse/HBASE-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770484#comment-13770484 ] Devaraj Das commented on HBASE-9343: Not sure whether this would work or not but would this work - we host the scan at /table/scan?queryParams. [~ndimiduk], not meaning to rush it, but maybe we can have the discussion based on your writeup in a follow up jira? Implement stateless scanner for Stargate Key: HBASE-9343 URL: https://issues.apache.org/jira/browse/HBASE-9343 Project: HBase Issue Type: Improvement Components: REST Affects Versions: 0.94.11 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9343_94.00.patch, HBASE-9343_94.01.patch, HBASE-9343_trunk.00.patch, HBASE-9343_trunk.01.patch, HBASE-9343_trunk.01.patch, HBASE-9343_trunk.02.patch The current scanner implementation for scanner stores state and hence not very suitable for REST server failure scenarios. The current JIRA proposes to implement a stateless scanner. In the first version of the patch, a new resource class ScanResource has been added and all the scan parameters will be specified as query params. The following are the scan parameters startrow - The start row for the scan. endrow - The end row for the scan. columns - The columns to scan. starttime, endtime - To only retrieve columns within a specific range of version timestamps,both start and end time must be specified. maxversions - To limit the number of versions of each column to be returned. batchsize - To limit the maximum number of values returned for each call to next(). limit - The number of rows to return in the scan operation. More on start row, end row and limit parameters. 1. If start row, end row and limit not specified, then the whole table will be scanned. 2. If start row and limit (say N) is specified, then the scan operation will return N rows from the start row specified. 3. If only limit parameter is specified, then the scan operation will return N rows from the start of the table. 4. If limit and end row are specified, then the scan operation will return N rows from start of table till the end row. If the end row is reached before N rows ( say M and M lt; N ), then M rows will be returned to the user. 5. If start row, end row and limit (say N ) are specified and N lt; number of rows between start row and end row, then N rows from start row will be returned to the user. If N gt; (number of rows between start row and end row (say M), then M number of rows will be returned to the user. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9566) Add back WALEdit#get/setScopes method
[
https://issues.apache.org/jira/browse/HBASE-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770486#comment-13770486
]
Lars Hofhansl commented on HBASE-9566:
--
{code}
for (Entrybyte[], Integer scope : scopes.entrySet()) {
this.putIntoScope(scope.getKey(), scope.getValue());
}
{code}
Can we written as: {{this.scopes.putAll(scopes);}}
Will do that and commit.
Add back WALEdit#get/setScopes method
-
Key: HBASE-9566
URL: https://issues.apache.org/jira/browse/HBASE-9566
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.12
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.94.12
Attachments: hbase-9566-v0.patch
We removed WALEdit's getScopes and setScopes methods in favor of getFromScope
and putIntoScope methods. However, this breaks compatibility between 0.94
versions. Fix is just to add back the get/setScopes methods to support
existing clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9566) Add back WALEdit#get/setScopes method
[ https://issues.apache.org/jira/browse/HBASE-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9566: - Attachment: 9566-v2.txt Like this. Add back WALEdit#get/setScopes method - Key: HBASE-9566 URL: https://issues.apache.org/jira/browse/HBASE-9566 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.12 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.12 Attachments: 9566-v2.txt, hbase-9566-v0.patch We removed WALEdit's getScopes and setScopes methods in favor of getFromScope and putIntoScope methods. However, this breaks compatibility between 0.94 versions. Fix is just to add back the get/setScopes methods to support existing clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-9566) Add back WALEdit#get/setScopes method
[ https://issues.apache.org/jira/browse/HBASE-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-9566. -- Resolution: Fixed Hadoop Flags: Reviewed Done. Add back WALEdit#get/setScopes method - Key: HBASE-9566 URL: https://issues.apache.org/jira/browse/HBASE-9566 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.12 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.12 Attachments: 9566-v2.txt, hbase-9566-v0.patch We removed WALEdit's getScopes and setScopes methods in favor of getFromScope and putIntoScope methods. However, this breaks compatibility between 0.94 versions. Fix is just to add back the get/setScopes methods to support existing clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9534: - Status: Patch Available (was: Open) Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8751) Enable peer cluster to choose/change the ColumnFamilies/Tables it really want to replicate from a source cluster
[ https://issues.apache.org/jira/browse/HBASE-8751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770501#comment-13770501 ] Feng Honghua commented on HBASE-8751: - Thanks [~techbuddy] :) Enable peer cluster to choose/change the ColumnFamilies/Tables it really want to replicate from a source cluster Key: HBASE-8751 URL: https://issues.apache.org/jira/browse/HBASE-8751 Project: HBase Issue Type: Improvement Components: Replication Reporter: Feng Honghua Attachments: HBASE-8751-0.94-V0.patch Consider scenarios (all cf are with replication-scope=1): 1) cluster S has 3 tables, table A has cfA,cfB, table B has cfX,cfY, table C has cf1,cf2. 2) cluster X wants to replicate table A : cfA, table B : cfX and table C from cluster S. 3) cluster Y wants to replicate table B : cfY, table C : cf2 from cluster S. Current replication implementation can't achieve this since it'll push the data of all the replicatable column-families from cluster S to all its peers, X/Y in this scenario. This improvement provides a fine-grained replication theme which enable peer cluster to choose the column-families/tables they really want from the source cluster: A). Set the table:cf-list for a peer when addPeer: hbase-shell add_peer '3', zk:1100:/hbase, table1; table2:cf1,cf2; table3:cf2 B). View the table:cf-list config for a peer using show_peer_tableCFs: hbase-shell show_peer_tableCFs 1 C). Change/set the table:cf-list for a peer using set_peer_tableCFs: hbase-shell set_peer_tableCFs '2', table1:cfX; table2:cf1; table3:cf1,cf2 In this theme, replication-scope=1 only means a column-family CAN be replicated to other clusters, but only the 'table:cf-list list' determines WHICH cf/table will actually be replicated to a specific peer. To provide back-compatibility, empty 'table:cf-list list' will replicate all replicatable cf/table. (this means we don't allow a peer which replicates nothing from a source cluster, we think it's reasonable: if replicating nothing why bother adding a peer?) This improvement addresses the exact problem raised by the first FAQ in http://hbase.apache.org/replication.html: GLOBAL means replicate? Any provision to replicate only to cluster X and not to cluster Y? or is that for later? Yes, this is for much later. I also noticed somebody mentioned replication-scope as integer rather than a boolean is for such fine-grained replication purpose, but I think extending replication-scope can't achieve the same replication granularity flexibility as providing above per-peer replication configurations. This improvement has been running smoothly in our production clusters (Xiaomi) for several months. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9488) Improve performance for small scan
[ https://issues.apache.org/jira/browse/HBASE-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770510#comment-13770510 ] Hudson commented on HBASE-9488: --- FAILURE: Integrated in hbase-0.96 #66 (See [https://builds.apache.org/job/hbase-0.96/66/]) HBASE-9488 Improve performance for small scan (zjushch: rev 1524273) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/branches/0.96/hbase-protocol/src/main/protobuf/Client.proto * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Improve performance for small scan -- Key: HBASE-9488 URL: https://issues.apache.org/jira/browse/HBASE-9488 Project: HBase Issue Type: Improvement Components: Client, Performance, Scanners Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.96.0 Attachments: hbase-9488-94-v3.patch, HBASE-9488-trunk.patch, HBASE-9488-trunkV2.patch, HBASE-9488-trunkV3.patch, HBASE-9488-trunkV4.patch, HBASE-9488-trunkV4.patch, HBASE-9488-trunkV5.patch, mergeRpcCallForScan.patch, test results.jpg review board: https://reviews.apache.org/r/14059/ *Performance Improvement* Test shows about 1.5~3X improvement for small scan where limit=50 under cache hit ratio=100%. See more performance test result from the picture attachment *Usage:* Scan scan = new Scan(startRow,stopRow); scan.setSmall(true); ResultScanner scanner = table.getScanner(scan); Set the new 'small' attribute as true for scan object, others are the same Now, one scan operation would call 3 RPC at least: openScanner(); next(); closeScanner(); I think we could reduce the RPC call to one for small scan to get better performance Also using pread is better than seek+read for small scan (For this point, see more on HBASE-7266) Implements such a small scan as the patch, and take the performance test as following: a.Environment： patched on 0.94 version one regionserver; one client with 50 concurrent threads; KV size:50/100; 100% LRU cache hit ratio; Random start row of scan b.Results: See the picture attachment -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[
https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770526#comment-13770526
]
Hadoop QA commented on HBASE-9534:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603757/hbase-9534-trunk-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7287//console
This message is automatically generated.
Short-Circuit Coprocessor HTable access when on the same server
---
Key: HBASE-9534
URL: https://issues.apache.org/jira/browse/HBASE-9534
Project: HBase
Issue Type: Bug
Reporter: Jesse Yates
Assignee: Jesse Yates
Labels: coprocessors, performance, regionserver
Fix For: 0.98.0, 0.94.12, 0.96.1
Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch,
hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch,
hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch
Coprocessors currently create a full HTable when they want to write. However,
we know that coprocessors must run from within an HBase server (either master
or RS). For the master, its rare that we are going to be doing performance
sensitive operations, but RS calls could be very time-intensive.
Therefore, we should be able to tell when a call from a CP attempts to talk
to the RS on which it lives and just short-circuit to calling that RS, rather
than going the long way around (which does the full marshalling/unmarshalling
of data, as well as going over the loopback interface).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9558) PerformanceEvaluation is in hbase-server, and create a dependency to MiniDFSCluster
[ https://issues.apache.org/jira/browse/HBASE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770539#comment-13770539 ] Nicolas Liochon commented on HBASE-9558: Or should we just remove the capability to use it with an internal mini cluster (it would still be possible to use on a local instance? [~jmspaggi], what do you think? Do you use the mini cluster feature for the perf eval? PerformanceEvaluation is in hbase-server, and create a dependency to MiniDFSCluster --- Key: HBASE-9558 URL: https://issues.apache.org/jira/browse/HBASE-9558 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Nicolas Liochon Priority: Minor It's the only dependency that is not in the tests package. I'm not clear on how to fix it. Any idea? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
Yang Wang created HBASE-9570:
Summary: With AccessDeniedException, HBase shell would be better
to just display the error message to be user friendly
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed as
follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated HBASE-9570:
-
Attachment: HBASE-9570
Attach a patch to improve HBase shell.
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Attachments: HBASE-9570
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Status: Open (was: Patch Available) Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Attachment: 4955.v7.patch Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated HBASE-9570:
-
Status: Patch Available (was: Open)
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Attachments: HBASE-9570
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Status: Patch Available (was: Open) Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9571) hbase-client does not depend on map reduce, but the pom.xml includes it
Nicolas Liochon created HBASE-9571: -- Summary: hbase-client does not depend on map reduce, but the pom.xml includes it Key: HBASE-9571 URL: https://issues.apache.org/jira/browse/HBASE-9571 Project: HBase Issue Type: Bug Components: build, Client Affects Versions: 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 mvn dependency:analyze is strange on this one. You can remove or add the dependency, it won't complain about used but undeclared or undeclared but used dependencies. But it does a difference in dependency:tree or for the client application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9571) hbase-client does not depend on map reduce, but the pom.xml includes it
[ https://issues.apache.org/jira/browse/HBASE-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9571: --- Attachment: 9571.v1.patch hbase-client does not depend on map reduce, but the pom.xml includes it --- Key: HBASE-9571 URL: https://issues.apache.org/jira/browse/HBASE-9571 Project: HBase Issue Type: Bug Components: build, Client Affects Versions: 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Attachments: 9571.v1.patch mvn dependency:analyze is strange on this one. You can remove or add the dependency, it won't complain about used but undeclared or undeclared but used dependencies. But it does a difference in dependency:tree or for the client application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9571) hbase-client does not depend on map reduce, but the pom.xml includes it
[ https://issues.apache.org/jira/browse/HBASE-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9571: --- Status: Patch Available (was: Open) hbase-client does not depend on map reduce, but the pom.xml includes it --- Key: HBASE-9571 URL: https://issues.apache.org/jira/browse/HBASE-9571 Project: HBase Issue Type: Bug Components: build, Client Affects Versions: 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Attachments: 9571.v1.patch mvn dependency:analyze is strange on this one. You can remove or add the dependency, it won't complain about used but undeclared or undeclared but used dependencies. But it does a difference in dependency:tree or for the client application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9566) Add back WALEdit#get/setScopes method
[ https://issues.apache.org/jira/browse/HBASE-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770558#comment-13770558 ] Hudson commented on HBASE-9566: --- SUCCESS: Integrated in HBase-0.94-security #294 (See [https://builds.apache.org/job/HBase-0.94-security/294/]) HBASE-9566 Add back WALEdit#get/setScopes method (Jesse) (larsh: rev 1524304) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java Add back WALEdit#get/setScopes method - Key: HBASE-9566 URL: https://issues.apache.org/jira/browse/HBASE-9566 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.12 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.12 Attachments: 9566-v2.txt, hbase-9566-v0.patch We removed WALEdit's getScopes and setScopes methods in favor of getFromScope and putIntoScope methods. However, this breaks compatibility between 0.94 versions. Fix is just to add back the get/setScopes methods to support existing clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9369) Add support for 1- and 2-byte integers in OrderedBytes and provide types
[ https://issues.apache.org/jira/browse/HBASE-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770571#comment-13770571 ] Hudson commented on HBASE-9369: --- SUCCESS: Integrated in HBase-TRUNK #4527 (See [https://builds.apache.org/job/HBase-TRUNK/4527/]) HBASE-9369 Add support for 1- and 2-byte integers in OrderedBytes and provide types (He Liangliang) (ndimiduk: rev 1524297) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt16.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt8.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawByte.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawShort.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java Add support for 1- and 2-byte integers in OrderedBytes and provide types Key: HBASE-9369 URL: https://issues.apache.org/jira/browse/HBASE-9369 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: He Liangliang Fix For: 0.98.0, 0.96.0 Attachments: 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, HBASE-9369.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9569) TestHLog is broken
[
https://issues.apache.org/jira/browse/HBASE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770570#comment-13770570
]
Hudson commented on HBASE-9569:
---
SUCCESS: Integrated in HBase-TRUNK #4527 (See
[https://builds.apache.org/job/HBase-TRUNK/4527/])
HBASE-9569 TestHLog is broken (stack: rev 1524294)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
TestHLog is broken
--
Key: HBASE-9569
URL: https://issues.apache.org/jira/browse/HBASE-9569
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.96.0
Attachments: 9569.txt
TestHLog is failing since last night.
Git bisect shows that the problem commit is this one:
{code}
f5b8ee6c7d678ee7da0095504d9870457a34d7b8 is the first bad commit
commit f5b8ee6c7d678ee7da0095504d9870457a34d7b8
Author: Michael Stack st...@apache.org
Date: Tue Sep 17 21:23:17 2013 +
HBASE-9562 Make HLogPE run against configured FS
git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.96@1524229
13f79535-47bb-0310-9956-ffa450edef68
{code}
It broke the test because it made it work on hdfs (which looks to have been
what was intended).
Looking...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770573#comment-13770573 ] Lars Hofhansl commented on HBASE-9534: -- Should also add a test to TestHCM.testClusterConnection, to make sure the internal pool is closed correctly. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[
https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770577#comment-13770577
]
Hadoop QA commented on HBASE-9534:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603762/9534-trunk.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7288//console
This message is automatically generated.
Short-Circuit Coprocessor HTable access when on the same server
---
Key: HBASE-9534
URL: https://issues.apache.org/jira/browse/HBASE-9534
Project: HBase
Issue Type: Bug
Reporter: Jesse Yates
Assignee: Jesse Yates
Labels: coprocessors, performance, regionserver
Fix For: 0.98.0, 0.94.12, 0.96.1
Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch,
hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch,
hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch
Coprocessors currently create a full HTable when they want to write. However,
we know that coprocessors must run from within an HBase server (either master
or RS). For the master, its rare that we are going to be doing performance
sensitive operations, but RS calls could be very time-intensive.
Therefore, we should be able to tell when a call from a CP attempts to talk
to the RS on which it lives and just short-circuit to calling that RS, rather
than going the long way around (which does the full marshalling/unmarshalling
of data, as well as going over the loopback interface).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770586#comment-13770586
]
Yang Wang commented on HBASE-9570:
--
Manually tested the attached patch, with it the display is as follows:
{noformat}
[u1@bdpe-wy conf]$ hbase shell
HBase Shell; enter 'helpRETURN' for list of supported commands.
Type exitRETURN to leave the HBase Shell
Version 0.97.0-SNAPSHOT, rUnknown, Wed Sep 18 16:47:53 CST 2013
hbase(main):001:0 scan 't1'
ROW COLUMN+CELL
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
Here is some help for this command:
...
{noformat}
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Attachments: HBASE-9570
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9566) Add back WALEdit#get/setScopes method
[ https://issues.apache.org/jira/browse/HBASE-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770605#comment-13770605 ] Hudson commented on HBASE-9566: --- FAILURE: Integrated in HBase-0.94 #1150 (See [https://builds.apache.org/job/HBase-0.94/1150/]) HBASE-9566 Add back WALEdit#get/setScopes method (Jesse) (larsh: rev 1524304) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java Add back WALEdit#get/setScopes method - Key: HBASE-9566 URL: https://issues.apache.org/jira/browse/HBASE-9566 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.12 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.12 Attachments: 9566-v2.txt, hbase-9566-v0.patch We removed WALEdit's getScopes and setScopes methods in favor of getFromScope and putIntoScope methods. However, this breaks compatibility between 0.94 versions. Fix is just to add back the get/setScopes methods to support existing clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9369) Add support for 1- and 2-byte integers in OrderedBytes and provide types
[ https://issues.apache.org/jira/browse/HBASE-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770614#comment-13770614 ] Hudson commented on HBASE-9369: --- FAILURE: Integrated in hbase-0.96 #67 (See [https://builds.apache.org/job/hbase-0.96/67/]) HBASE-9369 Add support for 1- and 2-byte integers in OrderedBytes and provide types (He Liangliang) (ndimiduk: rev 1524299) * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt16.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt8.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawByte.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawShort.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java * /hbase/branches/0.96/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java Add support for 1- and 2-byte integers in OrderedBytes and provide types Key: HBASE-9369 URL: https://issues.apache.org/jira/browse/HBASE-9369 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: He Liangliang Fix For: 0.98.0, 0.96.0 Attachments: 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, HBASE-9369.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9569) TestHLog is broken
[
https://issues.apache.org/jira/browse/HBASE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770613#comment-13770613
]
Hudson commented on HBASE-9569:
---
FAILURE: Integrated in hbase-0.96 #67 (See
[https://builds.apache.org/job/hbase-0.96/67/])
HBASE-9569 TestHLog is broken (stack: rev 1524295)
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
TestHLog is broken
--
Key: HBASE-9569
URL: https://issues.apache.org/jira/browse/HBASE-9569
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.96.0
Attachments: 9569.txt
TestHLog is failing since last night.
Git bisect shows that the problem commit is this one:
{code}
f5b8ee6c7d678ee7da0095504d9870457a34d7b8 is the first bad commit
commit f5b8ee6c7d678ee7da0095504d9870457a34d7b8
Author: Michael Stack st...@apache.org
Date: Tue Sep 17 21:23:17 2013 +
HBASE-9562 Make HLogPE run against configured FS
git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.96@1524229
13f79535-47bb-0310-9956-ffa450edef68
{code}
It broke the test because it made it work on hdfs (which looks to have been
what was intended).
Looking...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[
https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770622#comment-13770622
]
Liang Xie commented on HBASE-9535:
--
to me, seems the RPC's ByteBuffer.allocate is not in the top contributor list
for gc.
I added logs below
{code}
data = ByteBuffer.allocate(dataLength);
{code}
in SecureServer.java (my test env is security enabled)
for a 100%read ycsb scenario, yes, this log shows the data lengths are similar
with each other(that means it's very easier to reuse/pool the byte buffers),
but after a raw estimation:
my config: Xmn512m Xmx4096m
my result: SecureServer.readAndProcess, dataLength:162;read throughput 7000ops;
jstat -gcutil shows about 5ygc per second
7000 * 162 is just 1MB more or less, is just a little portion of Xmn size, so
per my view, seems it doesn't very helpful to improve gc...
Try a pool of direct byte buffers handling incoming ipc requests
Key: HBASE-9535
URL: https://issues.apache.org/jira/browse/HBASE-9535
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Assignee: stack
ipc takes in a query by allocating a ByteBuffer of the size of the request
and then reading off the socket into this on-heap BB.
Experiment with keeping a pool of BBs so we have some buffer reuse to cut on
garbage generated. Could checkout from pool in RpcServer#Reader. Could
check back into the pool when Handler is done just before it queues the
response on the Responder's queue. We should be good since, at least for
now, kvs get copied up into MSLAB (not references) when data gets stuffed
into MemStore; this should make it so no references left over when we check
the BB back into the pool for use next time around.
If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes
time so if already allocated, should be good. GC of DBBs is a pain but if in
a pool, we shouldn't be wanting this to happen). The copy from socket to the
DBB will be off-heap (should be fast).
Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by
size (we might want to bypass the pool if an object is size N).
DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when
short-circuit reading. It'd be a bummer if we had to allocate big objects
on-heap. Would still be an improvement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9571) hbase-client does not depend on map reduce, but the pom.xml includes it
[
https://issues.apache.org/jira/browse/HBASE-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770630#comment-13770630
]
Hadoop QA commented on HBASE-9571:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603780/9571.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:red}-1 hadoop2.0{color}. The patch failed to compile against the
hadoop 2.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7292//console
This message is automatically generated.
hbase-client does not depend on map reduce, but the pom.xml includes it
---
Key: HBASE-9571
URL: https://issues.apache.org/jira/browse/HBASE-9571
Project: HBase
Issue Type: Bug
Components: build, Client
Affects Versions: 0.98.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.96.1
Attachments: 9571.v1.patch
mvn dependency:analyze is strange on this one. You can remove or add the
dependency, it won't complain about used but undeclared or undeclared but
used dependencies. But it does a difference in dependency:tree or for the
client application.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Affects Version/s: 0.96.0 0.98.0 Status: Patch Available (was: Open) Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Status: Open (was: Patch Available) Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Attachment: 4955.v7.patch Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9571) hbase-client does not depend on map reduce, but the pom.xml includes it
[ https://issues.apache.org/jira/browse/HBASE-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9571: --- Resolution: Invalid Status: Resolved (was: Patch Available) my error. hbase-client does not depend on map reduce, but the pom.xml includes it --- Key: HBASE-9571 URL: https://issues.apache.org/jira/browse/HBASE-9571 Project: HBase Issue Type: Bug Components: build, Client Affects Versions: 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.96.1 Attachments: 9571.v1.patch mvn dependency:analyze is strange on this one. You can remove or add the dependency, it won't complain about used but undeclared or undeclared but used dependencies. But it does a difference in dependency:tree or for the client application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770654#comment-13770654
]
Hadoop QA commented on HBASE-9570:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603776/HBASE-9570
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7291//console
This message is automatically generated.
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Attachments: HBASE-9570
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
[jira] [Commented] (HBASE-9569) TestHLog is broken
[
https://issues.apache.org/jira/browse/HBASE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770714#comment-13770714
]
Hudson commented on HBASE-9569:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #738 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/738/])
HBASE-9569 TestHLog is broken (stack: rev 1524294)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
TestHLog is broken
--
Key: HBASE-9569
URL: https://issues.apache.org/jira/browse/HBASE-9569
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.96.0
Attachments: 9569.txt
TestHLog is failing since last night.
Git bisect shows that the problem commit is this one:
{code}
f5b8ee6c7d678ee7da0095504d9870457a34d7b8 is the first bad commit
commit f5b8ee6c7d678ee7da0095504d9870457a34d7b8
Author: Michael Stack st...@apache.org
Date: Tue Sep 17 21:23:17 2013 +
HBASE-9562 Make HLogPE run against configured FS
git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.96@1524229
13f79535-47bb-0310-9956-ffa450edef68
{code}
It broke the test because it made it work on hdfs (which looks to have been
what was intended).
Looking...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9488) Improve performance for small scan
[ https://issues.apache.org/jira/browse/HBASE-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770715#comment-13770715 ] Hudson commented on HBASE-9488: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #738 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/738/]) HBASE-9488 Improve performance for small scan (zjushch: rev 1524272) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Improve performance for small scan -- Key: HBASE-9488 URL: https://issues.apache.org/jira/browse/HBASE-9488 Project: HBase Issue Type: Improvement Components: Client, Performance, Scanners Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.96.0 Attachments: hbase-9488-94-v3.patch, HBASE-9488-trunk.patch, HBASE-9488-trunkV2.patch, HBASE-9488-trunkV3.patch, HBASE-9488-trunkV4.patch, HBASE-9488-trunkV4.patch, HBASE-9488-trunkV5.patch, mergeRpcCallForScan.patch, test results.jpg review board: https://reviews.apache.org/r/14059/ *Performance Improvement* Test shows about 1.5~3X improvement for small scan where limit=50 under cache hit ratio=100%. See more performance test result from the picture attachment *Usage:* Scan scan = new Scan(startRow,stopRow); scan.setSmall(true); ResultScanner scanner = table.getScanner(scan); Set the new 'small' attribute as true for scan object, others are the same Now, one scan operation would call 3 RPC at least: openScanner(); next(); closeScanner(); I think we could reduce the RPC call to one for small scan to get better performance Also using pread is better than seek+read for small scan (For this point, see more on HBASE-7266) Implements such a small scan as the patch, and take the performance test as following: a.Environment： patched on 0.94 version one regionserver; one client with 50 concurrent threads; KV size:50/100; 100% LRU cache hit ratio; Random start row of scan b.Results: See the picture attachment -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9400) [UI] Catalog tables section isn't aligned
[ https://issues.apache.org/jira/browse/HBASE-9400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770717#comment-13770717 ] Hudson commented on HBASE-9400: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #738 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/738/]) HBASE-9400 [UI] Catalog tables section isn't aligned (eclark: rev 1524262) * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/static/css/hbase.css [UI] Catalog tables section isn't aligned - Key: HBASE-9400 URL: https://issues.apache.org/jira/browse/HBASE-9400 Project: HBase Issue Type: Bug Components: UI Affects Versions: 0.95.2, 0.96.0 Reporter: Jimmy Xiang Assignee: Elliott Clark Priority: Minor Fix For: 0.98.0, 0.96.0 Attachments: catalog-tables.png, HBASE-9400-0.patch, HBASE-9400-1.patch I attached a picture of what I got. You can see it doesn't look right. One more thing is that the page doesn't auto-refresh when you switch tabs. For example, click Catalog tables, create a new table, click User tables, you don't see the new table. You need to refresh the page to see it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9369) Add support for 1- and 2-byte integers in OrderedBytes and provide types
[ https://issues.apache.org/jira/browse/HBASE-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770716#comment-13770716 ] Hudson commented on HBASE-9369: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #738 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/738/]) HBASE-9369 Add support for 1- and 2-byte integers in OrderedBytes and provide types (He Liangliang) (ndimiduk: rev 1524297) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt16.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt8.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawByte.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawShort.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java Add support for 1- and 2-byte integers in OrderedBytes and provide types Key: HBASE-9369 URL: https://issues.apache.org/jira/browse/HBASE-9369 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: He Liangliang Fix For: 0.98.0, 0.96.0 Attachments: 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, HBASE-9369.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9569) TestHLog is broken
[
https://issues.apache.org/jira/browse/HBASE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770735#comment-13770735
]
Hudson commented on HBASE-9569:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #36 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/36/])
HBASE-9569 TestHLog is broken (stack: rev 1524295)
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
TestHLog is broken
--
Key: HBASE-9569
URL: https://issues.apache.org/jira/browse/HBASE-9569
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.96.0
Attachments: 9569.txt
TestHLog is failing since last night.
Git bisect shows that the problem commit is this one:
{code}
f5b8ee6c7d678ee7da0095504d9870457a34d7b8 is the first bad commit
commit f5b8ee6c7d678ee7da0095504d9870457a34d7b8
Author: Michael Stack st...@apache.org
Date: Tue Sep 17 21:23:17 2013 +
HBASE-9562 Make HLogPE run against configured FS
git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.96@1524229
13f79535-47bb-0310-9956-ffa450edef68
{code}
It broke the test because it made it work on hdfs (which looks to have been
what was intended).
Looking...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9400) [UI] Catalog tables section isn't aligned
[ https://issues.apache.org/jira/browse/HBASE-9400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770738#comment-13770738 ] Hudson commented on HBASE-9400: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #36 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/36/]) HBASE-9400 [UI] Catalog tables section isn't aligned (eclark: rev 1524261) * /hbase/branches/0.96/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/branches/0.96/hbase-server/src/main/resources/hbase-webapps/static/css/hbase.css [UI] Catalog tables section isn't aligned - Key: HBASE-9400 URL: https://issues.apache.org/jira/browse/HBASE-9400 Project: HBase Issue Type: Bug Components: UI Affects Versions: 0.95.2, 0.96.0 Reporter: Jimmy Xiang Assignee: Elliott Clark Priority: Minor Fix For: 0.98.0, 0.96.0 Attachments: catalog-tables.png, HBASE-9400-0.patch, HBASE-9400-1.patch I attached a picture of what I got. You can see it doesn't look right. One more thing is that the page doesn't auto-refresh when you switch tabs. For example, click Catalog tables, create a new table, click User tables, you don't see the new table. You need to refresh the page to see it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9488) Improve performance for small scan
[ https://issues.apache.org/jira/browse/HBASE-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770736#comment-13770736 ] Hudson commented on HBASE-9488: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #36 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/36/]) HBASE-9488 Improve performance for small scan (zjushch: rev 1524273) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/branches/0.96/hbase-protocol/src/main/protobuf/Client.proto * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Improve performance for small scan -- Key: HBASE-9488 URL: https://issues.apache.org/jira/browse/HBASE-9488 Project: HBase Issue Type: Improvement Components: Client, Performance, Scanners Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.96.0 Attachments: hbase-9488-94-v3.patch, HBASE-9488-trunk.patch, HBASE-9488-trunkV2.patch, HBASE-9488-trunkV3.patch, HBASE-9488-trunkV4.patch, HBASE-9488-trunkV4.patch, HBASE-9488-trunkV5.patch, mergeRpcCallForScan.patch, test results.jpg review board: https://reviews.apache.org/r/14059/ *Performance Improvement* Test shows about 1.5~3X improvement for small scan where limit=50 under cache hit ratio=100%. See more performance test result from the picture attachment *Usage:* Scan scan = new Scan(startRow,stopRow); scan.setSmall(true); ResultScanner scanner = table.getScanner(scan); Set the new 'small' attribute as true for scan object, others are the same Now, one scan operation would call 3 RPC at least: openScanner(); next(); closeScanner(); I think we could reduce the RPC call to one for small scan to get better performance Also using pread is better than seek+read for small scan (For this point, see more on HBASE-7266) Implements such a small scan as the patch, and take the performance test as following: a.Environment： patched on 0.94 version one regionserver; one client with 50 concurrent threads; KV size:50/100; 100% LRU cache hit ratio; Random start row of scan b.Results: See the picture attachment -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9369) Add support for 1- and 2-byte integers in OrderedBytes and provide types
[ https://issues.apache.org/jira/browse/HBASE-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770737#comment-13770737 ] Hudson commented on HBASE-9369: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #36 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/36/]) HBASE-9369 Add support for 1- and 2-byte integers in OrderedBytes and provide types (He Liangliang) (ndimiduk: rev 1524299) * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt16.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/OrderedInt8.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawByte.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/types/RawShort.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java * /hbase/branches/0.96/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java Add support for 1- and 2-byte integers in OrderedBytes and provide types Key: HBASE-9369 URL: https://issues.apache.org/jira/browse/HBASE-9369 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: He Liangliang Fix For: 0.98.0, 0.96.0 Attachments: 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, 0001-HBASE-9369-Add-support-for-1-and-2-byte-integers-in-.patch, HBASE-9369.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9564) Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
[
https://issues.apache.org/jira/browse/HBASE-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770779#comment-13770779
]
Ted Yu commented on HBASE-9564:
---
Integrated Chao's patch to trunk.
Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
Key: HBASE-9564
URL: https://issues.apache.org/jira/browse/HBASE-9564
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Chao Shi
Priority: Minor
Attachments: 9564-v1.txt, hbase-9564-chaoshi.patch
From
https://builds.apache.org/job/PreCommit-HBASE-Build/7274/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testHandlerIsolation/
:
{code}
java.lang.AssertionError: expected:3 but was:2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testHandlerIsolation(TestSimpleRpcScheduler.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
This was likely due to verify(task, timeout(1000)) call timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-9564) Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
[
https://issues.apache.org/jira/browse/HBASE-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-9564:
-
Assignee: Chao Shi (was: Ted Yu)
Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
Key: HBASE-9564
URL: https://issues.apache.org/jira/browse/HBASE-9564
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Chao Shi
Priority: Minor
Attachments: 9564-v1.txt, hbase-9564-chaoshi.patch
From
https://builds.apache.org/job/PreCommit-HBASE-Build/7274/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testHandlerIsolation/
:
{code}
java.lang.AssertionError: expected:3 but was:2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testHandlerIsolation(TestSimpleRpcScheduler.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
This was likely due to verify(task, timeout(1000)) call timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770819#comment-13770819 ] Nicolas Liochon commented on HBASE-4955: v7 is about trying surefire 2.16 with the latest trunk. 3 tries: 1) local: tests seems to run well, but get stuck at a moment in the middle of the tests. 2) apache attempt 1: stuck. Some tests had errors. 3) apache attempt 2: as attempt 1. So it means that I will have to go after each surefire commit to find the guilty one. Will do, but later. Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0 Environment: all Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 8204.v4.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9558) PerformanceEvaluation is in hbase-server, and create a dependency to MiniDFSCluster
[ https://issues.apache.org/jira/browse/HBASE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770879#comment-13770879 ] stack commented on HBASE-9558: -- [~liochon] Just remove --minicluster. When would that option be useful? (running all inside the one jvm? That'd be ugly!) PerformanceEvaluation is in hbase-server, and create a dependency to MiniDFSCluster --- Key: HBASE-9558 URL: https://issues.apache.org/jira/browse/HBASE-9558 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Nicolas Liochon Priority: Minor It's the only dependency that is not in the tests package. I'm not clear on how to fix it. Any idea? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[
https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770888#comment-13770888
]
stack commented on HBASE-8755:
--
Just parking the numbers here. Takes same time to write 5x as much (5M
entries by 5 threads) as 1M entries by 1 thread. 50x takes 6x longer (50
threads writing 50M entries)
{code}
/tmp/log-patch1.1.txt:2013-09-17 21:19:31,495 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=1, iterations=100 took
991.258s 1008.819ops/s
/tmp/log-patch1.2.txt:2013-09-17 21:35:04,715 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=1, iterations=100 took
924.881s 1081.220ops/s
/tmp/log-patch1.3.txt:2013-09-17 21:51:32,416 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=1, iterations=100 took
979.312s 1021.125ops/s
/tmp/log-patch50.1.txt:2013-09-17 23:29:45,188 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=50, iterations=100 took
2960.095s 16891.350ops/s
/tmp/log-patch50.2.txt:2013-09-18 00:22:48,849 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=50, iterations=100 took
2924.844s 17094.930ops/s
/tmp/log-patch50.3.txt:2013-09-18 01:15:58,646 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=50, iterations=100 took
2927.617s 17078.736ops/s
/tmp/log-patch5.1.txt:2013-09-17 22:07:31,712 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=5, iterations=100 took
950.968s 5257.800ops/s
/tmp/log-patch5.2.txt:2013-09-17 22:23:39,680 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=5, iterations=100 took
939.312s 5323.045ops/s
/tmp/log-patch5.3.txt:2013-09-17 22:39:56,011 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=5, iterations=100 took
947.183s 5278.811ops/s
{code}
A new write thread model for HLog to improve the overall HBase write
throughput
---
Key: HBASE-8755
URL: https://issues.apache.org/jira/browse/HBASE-8755
Project: HBase
Issue Type: Improvement
Components: Performance, wal
Reporter: Feng Honghua
Assignee: stack
Priority: Critical
Fix For: 0.96.1
Attachments: 8755trunkV2.txt, HBASE-8755-0.94-V0.patch,
HBASE-8755-0.94-V1.patch, HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch
In current write model, each write handler thread (executing put()) will
individually go through a full 'append (hlog local buffer) = HLog writer
append (write to hdfs) = HLog writer sync (sync hdfs)' cycle for each write,
which incurs heavy race condition on updateLock and flushLock.
The only optimization where checking if current syncTillHere txid in
expectation for other thread help write/sync its own txid to hdfs and
omitting the write/sync actually help much less than expectation.
Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi
proposed a new write thread model for writing hdfs sequence file and the
prototype implementation shows a 4X improvement for throughput (from 17000 to
7+).
I apply this new write thread model in HLog and the performance test in our
test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1
RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size)
even beats the one of BigTable (Precolator published in 2011 says Bigtable's
write throughput then is 31002). I can provide the detailed performance test
results if anyone is interested.
The change for new write thread model is as below:
1 All put handler threads append the edits to HLog's local pending buffer;
(it notifies AsyncWriter thread that there is new edits in local buffer)
2 All put handler threads wait in HLog.syncer() function for underlying
threads to finish the sync that contains its txid;
3 An single AsyncWriter thread is responsible for retrieve all the buffered
edits in HLog's local pending buffer and write to the hdfs
(hlog.writer.append); (it notifies AsyncFlusher thread that there is new
writes to hdfs that needs a sync)
4 An single AsyncFlusher thread is responsible for issuing a sync to hdfs
to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread
that sync watermark increases)
5 An single AsyncNotifier thread is responsible for notifying all pending
put handler threads which are waiting in the HLog.syncer() function
6 No LogSyncer thread any more (since there is always
AsyncWriter/AsyncFlusher threads do the same job it does)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770925#comment-13770925 ] Jesse Yates commented on HBASE-9534: Smaller patch, but HTable is a little bit less clean with another round of duplicated code. Meh, I'm fine with either. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9567) Make room for additional encodings in OrderedBytes
[ https://issues.apache.org/jira/browse/HBASE-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9567: Fix Version/s: (was: 0.96.0) Make room for additional encodings in OrderedBytes -- Key: HBASE-9567 URL: https://issues.apache.org/jira/browse/HBASE-9567 Project: HBase Issue Type: Improvement Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-9567.00.patch I'd like to make a little more room in the OrderedBytes header byte. Also, reserve two values for HBASE-9369. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9566) Add back WALEdit#get/setScopes method
[ https://issues.apache.org/jira/browse/HBASE-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770924#comment-13770924 ] Jesse Yates commented on HBASE-9566: Did it the way I did so if putIntoScope changes we keep the same logic for setScopes. Right now its equivalent, but later...maybe not? I think the runtime is the same in either case. NBD, just wanted to explain my logic there. Thanks for committing Lars. Add back WALEdit#get/setScopes method - Key: HBASE-9566 URL: https://issues.apache.org/jira/browse/HBASE-9566 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.12 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.12 Attachments: 9566-v2.txt, hbase-9566-v0.patch We removed WALEdit's getScopes and setScopes methods in favor of getFromScope and putIntoScope methods. However, this breaks compatibility between 0.94 versions. Fix is just to add back the get/setScopes methods to support existing clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9572) [types] extend the breadth and depth of test coverage
Nick Dimiduk created HBASE-9572:
---
Summary: [types] extend the breadth and depth of test coverage
Key: HBASE-9572
URL: https://issues.apache.org/jira/browse/HBASE-9572
Project: HBase
Issue Type: Test
Affects Versions: 0.96.0
Reporter: Nick Dimiduk
Direct test coverage over the data types API is minimal. Existing tests cover
the underlying encoding mechanisms and the {{Struct}} components, but that's
about it. We should expand on the test coverage, something like what Orderly
accomplishes with its api-driven harness for generating random values and
verifying them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9564) Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
[
https://issues.apache.org/jira/browse/HBASE-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770933#comment-13770933
]
Hudson commented on HBASE-9564:
---
SUCCESS: Integrated in HBase-TRUNK #4528 (See
[https://builds.apache.org/job/HBase-TRUNK/4528/])
HBASE-9564 Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
(tedyu: rev 1524415)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java
Intermittent TestSimpleRpcScheduler#testHandlerIsolation failure
Key: HBASE-9564
URL: https://issues.apache.org/jira/browse/HBASE-9564
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Chao Shi
Priority: Minor
Attachments: 9564-v1.txt, hbase-9564-chaoshi.patch
From
https://builds.apache.org/job/PreCommit-HBASE-Build/7274/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testHandlerIsolation/
:
{code}
java.lang.AssertionError: expected:3 but was:2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testHandlerIsolation(TestSimpleRpcScheduler.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
This was likely due to verify(task, timeout(1000)) call timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9556) Provide key range support to bulkload to avoid too many reducers even the data belongs to few regions
[ https://issues.apache.org/jira/browse/HBASE-9556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770943#comment-13770943 ] Nick Dimiduk commented on HBASE-9556: - I like it. Provide key range support to bulkload to avoid too many reducers even the data belongs to few regions - Key: HBASE-9556 URL: https://issues.apache.org/jira/browse/HBASE-9556 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: rajeshbabu Assignee: rajeshbabu Priority: Minor Presently the number of reducers in bulk load are equal to number of regions. Lets suppose a table has 500 regions and import data only belongs 10 regions, still we are starting 500(equal to no. of regions) reducers instead of 10. Which will consume more time and resources. If user knows the row key range of import data, then we can pass startkey and/or endkey as input and based on the key range we can define the partitions and number of reducers(regions to which the data belongs). This helps to avoid too many reducers to start and do nothing and also avoids contention in shuffling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9343) Implement stateless scanner for Stargate
[
https://issues.apache.org/jira/browse/HBASE-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770948#comment-13770948
]
Nick Dimiduk commented on HBASE-9343:
-
Yes, you're probably right [~devaraj]. Is there a way we can push this into the
existing {{/table/scanner}} API? Currently that endpoint expects a PUT or
POST to request a scanner creation. Can we put the GET onto the same endpoint
to initiate the streaming connection? At least then all the scanner stuff is in
the same place.
[~avandana], [~toffer], [~apurtell] What do you think?
Implement stateless scanner for Stargate
Key: HBASE-9343
URL: https://issues.apache.org/jira/browse/HBASE-9343
Project: HBase
Issue Type: Improvement
Components: REST
Affects Versions: 0.94.11
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-9343_94.00.patch, HBASE-9343_94.01.patch,
HBASE-9343_trunk.00.patch, HBASE-9343_trunk.01.patch,
HBASE-9343_trunk.01.patch, HBASE-9343_trunk.02.patch
The current scanner implementation for scanner stores state and hence not
very suitable for REST server failure scenarios. The current JIRA proposes to
implement a stateless scanner. In the first version of the patch, a new
resource class ScanResource has been added and all the scan parameters will
be specified as query params.
The following are the scan parameters
startrow - The start row for the scan.
endrow - The end row for the scan.
columns - The columns to scan.
starttime, endtime - To only retrieve columns within a specific range of
version timestamps,both start and end time must be specified.
maxversions - To limit the number of versions of each column to be returned.
batchsize - To limit the maximum number of values returned for each call to
next().
limit - The number of rows to return in the scan operation.
More on start row, end row and limit parameters.
1. If start row, end row and limit not specified, then the whole table will
be scanned.
2. If start row and limit (say N) is specified, then the scan operation will
return N rows from the start row specified.
3. If only limit parameter is specified, then the scan operation will return
N rows from the start of the table.
4. If limit and end row are specified, then the scan operation will return N
rows from start of table till the end row. If the end row is
reached before N rows ( say M and M lt; N ), then M rows will be returned to
the user.
5. If start row, end row and limit (say N ) are specified and N lt; number
of rows between start row and end row, then N rows from start row
will be returned to the user. If N gt; (number of rows between start row and
end row (say M), then M number of rows will be returned to the
user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[
https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770956#comment-13770956
]
Ted Yu commented on HBASE-4364:
---
{code}
+// Select clause has its own column qualifiers
{code}
Better replace 'Select clause' with other term.
Approach looks feasible.
Have you benchmarked the performance of the patch ?
Please put next patch on review board.
Filters applied to columns not in the selected column list are ignored
--
Key: HBASE-4364
URL: https://issues.apache.org/jira/browse/HBASE-4364
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Todd Lipcon
Priority: Critical
Attachments:
HBASE-4364-failing-test-with-simplest-custom-filter.patch, HBASE-4364.patch,
hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch
For a scan, if you select some set of columns using addColumns(), and then
apply a SingleColumnValueFilter that restricts the results based on some
other columns which aren't selected, then those filter conditions are ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770962#comment-13770962 ] Lars Hofhansl commented on HBASE-9534: -- If it's the same to you, let's do the slightly larger one. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9510) Namespace operations should throw clean exceptions
[
https://issues.apache.org/jira/browse/HBASE-9510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770973#comment-13770973
]
Francis Liu commented on HBASE-9510:
Created HBASE-9573
Namespace operations should throw clean exceptions
--
Key: HBASE-9510
URL: https://issues.apache.org/jira/browse/HBASE-9510
Project: HBase
Issue Type: Bug
Components: master
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.0
Attachments: 9510v4.txt, 9510v4.txt, hbase-9510_v1.patch,
hbase-9510_v2.patch, hbase-9510_v3.patch
Some of the namespace operations does not throw clean exceptions mimicking
table exceptions (TableNotFoundException, etc).
For example:
{code}
hbase(main):007:0 describe_namespace 'non_existing_namespace'
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1816)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$0(SimpleRpcScheduler.java:161)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toProtoNamespaceDescriptor(ProtobufUtil.java:2138)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3029)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32904)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2079)
... 5 more
{code}
We can clean up the exceptions thrown from ns commands.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9573) provide namespaceExists api
Francis Liu created HBASE-9573: -- Summary: provide namespaceExists api Key: HBASE-9573 URL: https://issues.apache.org/jira/browse/HBASE-9573 Project: HBase Issue Type: Improvement Reporter: Francis Liu Assignee: Francis Liu Presently in order to test existence of a namespace one has to use the get api and catch the NamespaceNotFound exception. This is undersirable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771037#comment-13771037 ] Vasu Mariyala commented on HBASE-4364: -- Thanks [~yuzhih...@gmail.com] for your comments. Before doing the performance benchmarking, I wanted to get some comments on the approach. I would work on it and post the observations. Sure, I would post the next patch on the review board. Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, HBASE-4364.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9558) PerformanceEvaluation is in hbase-server, and create a dependency to MiniDFSCluster
[ https://issues.apache.org/jira/browse/HBASE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771053#comment-13771053 ] Nicolas Liochon commented on HBASE-9558: Ok, if nobody objects I will do that. PerformanceEvaluation is in hbase-server, and create a dependency to MiniDFSCluster --- Key: HBASE-9558 URL: https://issues.apache.org/jira/browse/HBASE-9558 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Nicolas Liochon Priority: Minor It's the only dependency that is not in the tests package. I'm not clear on how to fix it. Any idea? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771061#comment-13771061 ] Jesse Yates commented on HBASE-9534: Hmmm, looks like you have a little cruft in your 0.94 patch there Lars. I'll post a new version (that I'm committing) minus the pom changes. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[
https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771078#comment-13771078
]
Hadoop QA commented on HBASE-9534:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603881/hbase-9534-0.94-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7294//console
This message is automatically generated.
Short-Circuit Coprocessor HTable access when on the same server
---
Key: HBASE-9534
URL: https://issues.apache.org/jira/browse/HBASE-9534
Project: HBase
Issue Type: Bug
Reporter: Jesse Yates
Assignee: Jesse Yates
Labels: coprocessors, performance, regionserver
Fix For: 0.98.0, 0.94.12, 0.96.1
Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch,
hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch,
hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch,
hbase-9534-trunk-v2.patch
Coprocessors currently create a full HTable when they want to write. However,
we know that coprocessors must run from within an HBase server (either master
or RS). For the master, its rare that we are going to be doing performance
sensitive operations, but RS calls could be very time-intensive.
Therefore, we should be able to tell when a call from a CP attempts to talk
to the RS on which it lives and just short-circuit to calling that RS, rather
than going the long way around (which does the full marshalling/unmarshalling
of data, as well as going over the loopback interface).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9574) [types] provide a mini-language for describing types
Nick Dimiduk created HBASE-9574:
---
Summary: [types] provide a mini-language for describing types
Key: HBASE-9574
URL: https://issues.apache.org/jira/browse/HBASE-9574
Project: HBase
Issue Type: Improvement
Reporter: Nick Dimiduk
We can make types more widely accessible if we can provide a simple way for
users to describe their types. This would be as java-agnostic as possible (ie,
can be used at the shell or in a Hive DDL statement). This would be similar in
functionality to {{ParseFilter}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9343) Implement stateless scanner for Stargate
[ https://issues.apache.org/jira/browse/HBASE-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771095#comment-13771095 ] Francis Liu commented on HBASE-9343: [~ndimiduk] IMHO adding the new streaming scanner api to /table/scanner would convolute that resource. I think your original proposal of 'table/*' (AKA suffix globing in the doc) is inline with existing apis and I'd be more amenable to that. It seems that the suffix globing api only has one query parameter so there shouldn't be any conflicts. Are we trying to avoid adding a new resource, is that the concern? Implement stateless scanner for Stargate Key: HBASE-9343 URL: https://issues.apache.org/jira/browse/HBASE-9343 Project: HBase Issue Type: Improvement Components: REST Affects Versions: 0.94.11 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9343_94.00.patch, HBASE-9343_94.01.patch, HBASE-9343_trunk.00.patch, HBASE-9343_trunk.01.patch, HBASE-9343_trunk.01.patch, HBASE-9343_trunk.02.patch The current scanner implementation for scanner stores state and hence not very suitable for REST server failure scenarios. The current JIRA proposes to implement a stateless scanner. In the first version of the patch, a new resource class ScanResource has been added and all the scan parameters will be specified as query params. The following are the scan parameters startrow - The start row for the scan. endrow - The end row for the scan. columns - The columns to scan. starttime, endtime - To only retrieve columns within a specific range of version timestamps,both start and end time must be specified. maxversions - To limit the number of versions of each column to be returned. batchsize - To limit the maximum number of values returned for each call to next(). limit - The number of rows to return in the scan operation. More on start row, end row and limit parameters. 1. If start row, end row and limit not specified, then the whole table will be scanned. 2. If start row and limit (say N) is specified, then the scan operation will return N rows from the start row specified. 3. If only limit parameter is specified, then the scan operation will return N rows from the start of the table. 4. If limit and end row are specified, then the scan operation will return N rows from start of table till the end row. If the end row is reached before N rows ( say M and M lt; N ), then M rows will be returned to the user. 5. If start row, end row and limit (say N ) are specified and N lt; number of rows between start row and end row, then N rows from start row will be returned to the user. If N gt; (number of rows between start row and end row (say M), then M number of rows will be returned to the user. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9534: --- Resolution: Fixed Status: Resolved (was: Patch Available) Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9534: --- Attachment: hbase-9534-0.94-v2.patch Attaching updated 0.94-Lars patch which was got committed. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-0.94.txt, 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9534: - Attachment: (was: 9534-0.94.txt) Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771125#comment-13771125 ] Lars Hofhansl commented on HBASE-9534: -- Whoops. Removed in order not to confuse anybody. Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9575) Add script to automate much of the rc-making process
[ https://issues.apache.org/jira/browse/HBASE-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9575: - Attachment: 9575.txt 9569.txt Simple script. Add script to automate much of the rc-making process Key: HBASE-9575 URL: https://issues.apache.org/jira/browse/HBASE-9575 Project: HBase Issue Type: Task Components: build Reporter: stack Assignee: stack Attachments: 9569.txt, 9575.txt We script a good part of the RC-making process. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-9575) Add script to automate much of the rc-making process
[ https://issues.apache.org/jira/browse/HBASE-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9575. -- Resolution: Fixed Fix Version/s: 0.96.0 0.98.0 Applied to 0.96 and trunk. Have verified basically works. Add script to automate much of the rc-making process Key: HBASE-9575 URL: https://issues.apache.org/jira/browse/HBASE-9575 Project: HBase Issue Type: Task Components: build Reporter: stack Assignee: stack Fix For: 0.98.0, 0.96.0 Attachments: 9569.txt, 9575.txt We script a good part of the RC-making process. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9575) Add script to automate much of the rc-making process
stack created HBASE-9575: Summary: Add script to automate much of the rc-making process Key: HBASE-9575 URL: https://issues.apache.org/jira/browse/HBASE-9575 Project: HBase Issue Type: Task Components: build Reporter: stack Assignee: stack We script a good part of the RC-making process. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5603) rolling-restart.sh script hangs when attempting to detect expiration of /hbase/master znode.
[
https://issues.apache.org/jira/browse/HBASE-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5603:
-
Fix Version/s: 0.98.0
rolling-restart.sh script hangs when attempting to detect expiration of
/hbase/master znode.
Key: HBASE-5603
URL: https://issues.apache.org/jira/browse/HBASE-5603
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.92.0, 0.94.0, 0.95.2
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Priority: Blocker
Fix For: 0.94.0, 0.98.0, 0.95.0
Attachments: HBASE-5603.patch
Due to bugfix ZOOKEEPER-1059 (ZK 3.4.0+), the rolling-restart.sh script will
hang when attempting to make sure the /hbase/master znode is deleted.
Here's the code
{code}
# make sure the master znode has been deleted before continuing
zparent=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool
zookeeper.znode.parent`
if [ $zparent == null ]; then zparent=/hbase; fi
zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool
zookeeper.znode.master`
if [ $zmaster == null ]; then zmaster=master; fi
zmaster=$zparent/$zmaster
echo -n Waiting for Master ZNode ${zmaster} to expire
while bin/hbase zkcli stat $zmaster /dev/null 21; do
echo -n .
sleep 1
done
echo #force a newline
{code}
Prior to ZOOKEEPER-1059, stat on a null znode would NPE and cause zkcli to
exit with retcode 1. Afterwards, the null is caught, zkcli will exit with 0
in the case where the znode is present and in the case where it does not
exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[
https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-6311:
-
Fix Version/s: 0.98.0
Data error after majorCompaction caused by keeping MVCC for opened scanners
---
Key: HBASE-6311
URL: https://issues.apache.org/jira/browse/HBASE-6311
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Blocker
Fix For: 0.94.1, 0.98.0, 0.95.0
Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch,
HBASE-6311v2.patch
It is a big problem we found in 0.94, and you could reproduce the problem in
Trunk using the test case I uploaded.
When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC
for opened scanners;
However,It will make data mistake after majorCompaction because we will skip
delete type KV but keep the put type kv in the compacted storefile.
The following is the reason from code:
In StoreFileScanner, enforceMVCC is false when compaction, so we could read
the delete type KV,
However, we will skip this delete type KV in ScanQueryMatcher because
following code
{code}
if (kv.isDelete())
{
...
if (includeDeleteMarker
kv.getMemstoreTS() = maxReadPointToTrackVersions) {
System.out.println(add deletes,maxReadPointToTrackVersions=
+ maxReadPointToTrackVersions);
this.deletes.add(bytes, offset, qualLength, timestamp, type);
}
...
}
{code}
Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
and kv.getMemstoreTS() maxReadPointToTrackVersions
So we won't add this to DeleteTracker.
Why test case passed if remove the line
MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Because in the StoreFileScanner#skipKVsNewerThanReadpoint
{code}
if (cur.getMemstoreTS() = readPoint) {
cur.setMemstoreTS(0);
}
{code}
So if we remove the line
MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will
add it to DeleteTracker in ScanQueryMatcher
Solution:
We use smallestReadPoint of region when compaction to keep MVCC for OPENED
scanner, So we should retain delete type kv in output in the case(Already
deleted KV is retained in output to make old opened scanner could read this
KV) even if it is a majorcompaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5864:
-
Fix Version/s: (was: 0.94.0)
Error while reading from hfile in 0.94
--
Key: HBASE-5864
URL: https://issues.apache.org/jira/browse/HBASE-5864
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.95.0
Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
HBASE-5864_3.patch, HBASE-5864_test.patch
Got the following stacktrace during region split.
{noformat}
2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
Failed getting store size for value
java.io.IOException: Requested block is out of range: 2906737606134037404,
lastDataBlockOffset: 84764558
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
at
org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5861) Hadoop 23 compilation broken due to tests introduced in HBASE-5604
[
https://issues.apache.org/jira/browse/HBASE-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5861:
-
Fix Version/s: (was: 0.95.0)
Hadoop 23 compilation broken due to tests introduced in HBASE-5604
--
Key: HBASE-5861
URL: https://issues.apache.org/jira/browse/HBASE-5861
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.0, 0.95.2
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Priority: Blocker
Fix For: 0.94.0
Attachments: 5861.txt, 5861-v4.patch, hbase-5861-jon.patch,
hbase-5861-v2.patch, hbase-5861-v3.patch
When attempting to compile HBase 0.94rc1 against hadoop 23, I got this set of
compilation error messages:
{code}
jon@swoop:~/proj/hbase-0.94$ mvn clean test -Dhadoop.profile=23 -DskipTests
...
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 18.926s
[INFO] Finished at: Mon Apr 23 10:38:47 PDT 2012
[INFO] Final Memory: 55M/555M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure: Compilation
failure:
[ERROR]
/home/jon/proj/hbase-0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[147,46]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jon/proj/hbase-0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[153,29]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jon/proj/hbase-0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[194,46]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jon/proj/hbase-0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[206,29]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jon/proj/hbase-0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[213,29]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jon/proj/hbase-0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[226,29]
org.apache.hadoop.mapreduce.TaskAttemptContext is abstract; cannot be
instantiated
[ERROR] - [Help 1]
{code}
Upon further investigation this issue is due to code introduced in HBASE-5064
and is also present in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5641) decayingSampleTick1 prevents HBase from shutting down.
[
https://issues.apache.org/jira/browse/HBASE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5641:
-
Fix Version/s: (was: 0.95.0)
decayingSampleTick1 prevents HBase from shutting down.
--
Key: HBASE-5641
URL: https://issues.apache.org/jira/browse/HBASE-5641
Project: HBase
Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Blocker
Fix For: 0.94.0
Attachments: 5641.txt
I think this is the problem. It creates a non-daemon thread.
{code}
private static final ScheduledExecutorService TICK_SERVICE =
Executors.newScheduledThreadPool(1,
Threads.getNamedThreadFactory(decayingSampleTick));
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System
[ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5885: - Fix Version/s: (was: 0.95.0) Invalid HFile block magic on Local file System -- Key: HBASE-5885 URL: https://issues.apache.org/jira/browse/HBASE-5885 Project: HBase Issue Type: Bug Affects Versions: 0.94.0, 0.95.2 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Blocker Fix For: 0.94.0 Attachments: 5885-trunk-v2.txt, HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch ERROR: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=7, exceptions: Thu Apr 26 11:19:18 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for reader reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=01/info:data/1335463981520/Put, lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, avgValueLen=1000, entries=1215085, length=1264354417, cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0] at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) Caused by: java.io.IOException: Invalid HFile block magic: \xEC\xD5\x9D\xB4\xC2bfo at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153) at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164) at org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130) ... 12 more Thu Apr 26 11:19:19 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:216) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630) at
[jira] [Updated] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4470:
-
Fix Version/s: 0.92.2
ServerNotRunningException coming out of assignRootAndMeta kills the Master
--
Key: HBASE-4470
URL: https://issues.apache.org/jira/browse/HBASE-4470
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Gregory Chanan
Priority: Critical
Fix For: 0.92.2, 0.94.1, 0.95.0
Attachments: HBASE-4470-90.patch, HBASE-4470-v2-90.patch,
HBASE-4470-v2-92_94.patch, HBASE-4470-v2-trunk.patch
I'm surprised we still have issues like that and I didn't get a hit while
googling so forgive me if there's already a jira about it.
When the master starts it verifies the locations of root and meta before
assigning them, if the server is started but not running you'll get this:
{quote}
2011-09-23 04:47:44,859 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
RemoteException connecting to RS
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
{quote}
I hit that 3-4 times this week while debugging something else. The worst is
that when you restart the master it sees that as a failover, but none of the
regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[
https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3996:
-
Release Note:
Adds MultiTableInputFormat.
Usage example:
{code}
Scan scan1 = new Scan();
scan1.setStartRow(start1);
scan1.setStopRow(end1);
Scan scan2 = new Scan();
scan2.setStartRow(start2);
scan2.setStopRow(end2);
MultiTableInputCollection mtic = new MultiTableInputCollection();
mtic.Add(tableName1, scan1);
mtic.Add(tableName2, scan2);
TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class,
IntWritable.class, job1);
{code}
Support multiple tables and scanners as input to the mapper in map/reduce
jobs
---
Key: HBASE-3996
URL: https://issues.apache.org/jira/browse/HBASE-3996
Project: HBase
Issue Type: Improvement
Components: mapreduce
Reporter: Eran Kutner
Assignee: Bryan Baugher
Priority: Critical
Fix For: 0.94.5, 0.95.0
Attachments: 3996-0.94.txt, 3996-v10.txt, 3996-v11.txt, 3996-v12.txt,
3996-v13.txt, 3996-v14.txt, 3996-v15.txt, 3996-v2.txt, 3996-v3.txt,
3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, 3996-v8.txt, 3996-v9.txt,
HBase-3996.patch
It seems that in many cases feeding data from multiple tables or multiple
scanners on a single table can save a lot of time when running map/reduce
jobs.
I propose a new MultiTableInputFormat class that would allow doing this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5864) Error while reading from hfile in 0.94
[
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5864:
-
Fix Version/s: 0.94.0
Error while reading from hfile in 0.94
--
Key: HBASE-5864
URL: https://issues.apache.org/jira/browse/HBASE-5864
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0, 0.95.0
Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch,
HBASE-5864_3.patch, HBASE-5864_test.patch
Got the following stacktrace during region split.
{noformat}
2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store:
Failed getting store size for value
java.io.IOException: Requested block is out of range: 2906737606134037404,
lastDataBlockOffset: 84764558
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
at
org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix
[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-6200:
-
Fix Version/s: 0.92.2
KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---
Key: HBASE-6200
URL: https://issues.apache.org/jira/browse/HBASE-6200
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
Fix For: 0.92.2, 0.94.1, 0.95.0
Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch,
6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt
As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird
behavior when some families share the same prefix. He posted a link to his
code to show how it fails, http://pastebin.com/7TBA1XGh
Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families
and qualifiers so f:a is said to be bigger than f1:, which is false. Then
what happens is that the KVs are returned in the right order from the RS but
then doing {{Result.binarySearch}} it uses
{{KeyComparator.compareWithoutRow}} which has a different sorting so the end
result is undetermined.
I added some debug and I can see that the data is returned in the right order
but {{Arrays.binarySearch}} returned the wrong KV, which is then verified
agains the passed family and qualifier which fails so null is returned.
I don't know how frequent it is for users to have families with the same
prefix, but those that do have that and that use those families at the same
time will have big correctness issues. This is why I mark this as a blocker.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5611:
-
Fix Version/s: (was: 0.95.0)
0.92.2
Replayed edits from regions that failed to open during recovery aren't
removed from the global MemStore size
Key: HBASE-5611
URL: https://issues.apache.org/jira/browse/HBASE-5611
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
Fix For: 0.92.2, 0.94.0
Attachments: 5611-94.addendum, 5611-94-v2.txt, HBASE-5611-92.patch,
HBASE-5611-94-minorchange.patch, HBASE-5611-trunk-v2-minorchange.patch
This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
it's still possible to hit it if a region fails to open for more obscure
reasons like HDFS errors.
Consider a region that just went through distributed splitting and that's now
being opened by a new RS. The first thing it does is to read the recovery
files and put the edits in the {{MemStores}}. If this process takes a long
time, the master will move that region away. At that point the edits are
still accounted for in the global {{MemStore}} size but they are dropped when
the {{HRegion}} gets cleaned up. It's completely invisible until the
{{MemStoreFlusher}} needs to force flush a region and that none of them have
edits:
{noformat}
2012-03-21 00:33:39,303 DEBUG
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
because memory above low water=5.9g
2012-03-21 00:33:39,303 ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
for entry null
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The {{null}} here is a region. In my case I had so many edits in the
{{MemStore}} during recovery that I'm over the low barrier although in fact
I'm at 0. It happened yesterday and it still printing this out.
To fix this we need to be able to decrease the global {{MemStore}} size when
the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
[
https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3443:
-
Fix Version/s: (was: 0.95.0)
ICV optimization to look in memstore first and then store files (HBASE-3082)
does not work when deletes are in the mix
--
Key: HBASE-3443
URL: https://issues.apache.org/jira/browse/HBASE-3443
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6,
0.92.0, 0.92.1
Reporter: Kannan Muthukkaruppan
Assignee: Lars Hofhansl
Priority: Critical
Labels: corruption
Fix For: 0.94.0
Attachments: 3443.txt
For incrementColumnValue() HBASE-3082 adds an optimization to check memstores
first, and only if not present in the memstore then check the store files. In
the presence of deletes, the above optimization is not reliable.
If the column is marked as deleted in the memstore, one should not look
further into the store files. But currently, the code does so.
Sample test code outline:
{code}
admin.createTable(desc)
table = HTable.new(conf, tableName)
table.incrementColumnValue(Bytes.toBytes(row), cf1name,
Bytes.toBytes(column), 5);
admin.flush(tableName)
sleep(2)
del = Delete.new(Bytes.toBytes(row))
table.delete(del)
table.incrementColumnValue(Bytes.toBytes(row), cf1name,
Bytes.toBytes(column), 5);
get = Get.new(Bytes.toBytes(row))
keyValues = table.get(get).raw()
keyValues.each do |keyValue|
puts Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())};
end
{code}
The above prints:
{code}
Expect 5; Got Value=10
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771206#comment-13771206 ] Hudson commented on HBASE-9534: --- SUCCESS: Integrated in HBase-0.94-security #295 (See [https://builds.apache.org/job/HBase-0.94-security/295/]) HBASE-9534: Short-Circuit Coprocessor HTable access when on the same server (jyates: rev 1524522) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/CoprocessorHConnection.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5916) RS restart just before master intialization we make the cluster non operative
[
https://issues.apache.org/jira/browse/HBASE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5916:
-
Fix Version/s: 0.92.2
RS restart just before master intialization we make the cluster non operative
-
Key: HBASE-5916
URL: https://issues.apache.org/jira/browse/HBASE-5916
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.92.2, 0.94.1, 0.95.0
Attachments: HBASE-5916_92.patch, HBASE-5916_94.patch,
HBASE-5916_trunk_1.patch, HBASE-5916_trunk_1.patch, HBASE-5916_trunk_2.patch,
HBASE-5916_trunk_3.patch, HBASE-5916_trunk_4.patch, HBASE-5916_trunk.patch,
HBASE-5916_trunk_v5.patch, HBASE-5916_trunk_v6.patch,
HBASE-5916_trunk_v7.patch, HBASE-5916_trunk_v8.patch,
HBASE-5916_trunk_v9.patch, HBASE-5916v8.patch
Consider a case where my master is getting restarted. RS that was alive when
the master restart started, gets restarted before the master initializes the
ServerShutDownHandler.
{code}
serverShutdownHandlerEnabled = true;
{code}
In this case when the RS tries to register with the master, the master will
try to expire the server but the server cannot be expired as still the
serverShutdownHandler is not enabled.
This case may happen when i have only one RS gets restarted or all the RS
gets restarted at the same time.(before assignRootandMeta).
{code}
LOG.info(message);
if (existingServer.getStartcode() serverName.getStartcode()) {
LOG.info(Triggering server recovery; existingServer +
existingServer + looks stale, new server: + serverName);
expireServer(existingServer);
}
{code}
If another RS is brought up then the cluster comes back to normalcy.
May be a very corner case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9518) getFakedKey() improvement
[ https://issues.apache.org/jira/browse/HBASE-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771223#comment-13771223 ] Jean-Daniel Cryans commented on HBASE-9518: --- +1 When you have some time [~xieliang007], it'd be nice if you added the faked keys to this documentation http://hbase.apache.org/book.html#hfilev2 getFakedKey() improvement - Key: HBASE-9518 URL: https://issues.apache.org/jira/browse/HBASE-9518 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.98.0, 0.96.1 Reporter: Liang Xie Assignee: Liang Xie Attachments: HBASE-9518.txt, HBASE-9518-v2.txt make generating faked key algo more aggressive -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9534) Short-Circuit Coprocessor HTable access when on the same server
[ https://issues.apache.org/jira/browse/HBASE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771222#comment-13771222 ] Hudson commented on HBASE-9534: --- SUCCESS: Integrated in hbase-0.95 #513 (See [https://builds.apache.org/job/hbase-0.95/513/]) HBASE-9534: Short-Circuit Coprocessor HTable access when on the same server (jyates: rev 1524523) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/client/CoprocessorHConnection.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java Short-Circuit Coprocessor HTable access when on the same server --- Key: HBASE-9534 URL: https://issues.apache.org/jira/browse/HBASE-9534 Project: HBase Issue Type: Bug Reporter: Jesse Yates Assignee: Jesse Yates Labels: coprocessors, performance, regionserver Fix For: 0.98.0, 0.94.12, 0.96.1 Attachments: 9534-trunk.txt, hbase-9534-0.94-v0.patch, hbase-9534-0.94-v1.patch, hbase-9534-0.94-v2.patch, hbase-9534-trunk-v0.patch, hbase-9534-trunk-v1.patch, hbase-9534-trunk-v2.patch Coprocessors currently create a full HTable when they want to write. However, we know that coprocessors must run from within an HBase server (either master or RS). For the master, its rare that we are going to be doing performance sensitive operations, but RS calls could be very time-intensive. Therefore, we should be able to tell when a call from a CP attempts to talk to the RS on which it lives and just short-circuit to calling that RS, rather than going the long way around (which does the full marshalling/unmarshalling of data, as well as going over the loopback interface). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9502) HStore.seekToScanner should handle magic value
[ https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771231#comment-13771231 ] Jean-Daniel Cryans commented on HBASE-9502: --- [~xieliang007] please fix the indentation in TestFromClientSide. HStore.seekToScanner should handle magic value -- Key: HBASE-9502 URL: https://issues.apache.org/jira/browse/HBASE-9502 Project: HBase Issue Type: Bug Components: regionserver, Scanners Affects Versions: 0.98.0, 0.95.2, 0.96.1 Reporter: Liang Xie Assignee: Liang Xie Attachments: HBASE-9502.txt due to faked key, the seekTo probably reture -2, and HStore.seekToScanner should handle this corner case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7336) HFileBlock.readAtOffset does not work well with multiple threads
[ https://issues.apache.org/jira/browse/HBASE-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7336: - Release Note: Improves read concurrency by changing HFileBlock#readAtOffset from synchronized to reentrant lock; will have contending scanners fall back to preading rather than synchronized seek+read. Keeps all cores busy rather than just the one. HFileBlock.readAtOffset does not work well with multiple threads Key: HBASE-7336 URL: https://issues.apache.org/jira/browse/HBASE-7336 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Critical Fix For: 0.94.4, 0.95.0 Attachments: 7336-0.94.txt, 7336-0.96.txt HBase grinds to a halt when many threads scan along the same set of blocks and neither read short circuit is nor block caching is enabled for the dfs client ... disabling the block cache makes sense on very large scans. It turns out that synchronizing in istream in HFileBlock.readAtOffset is the culprit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7336) HFileBlock.readAtOffset does not work well with multiple threads
[ https://issues.apache.org/jira/browse/HBASE-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7336: - Component/s: Performance HFileBlock.readAtOffset does not work well with multiple threads Key: HBASE-7336 URL: https://issues.apache.org/jira/browse/HBASE-7336 Project: HBase Issue Type: Sub-task Components: Performance Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Critical Fix For: 0.94.4, 0.95.0 Attachments: 7336-0.94.txt, 7336-0.96.txt HBase grinds to a halt when many threads scan along the same set of blocks and neither read short circuit is nor block caching is enabled for the dfs client ... disabling the block cache makes sense on very large scans. It turns out that synchronizing in istream in HFileBlock.readAtOffset is the culprit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9364) Get request with multiple columns returns partial results
[ https://issues.apache.org/jira/browse/HBASE-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771238#comment-13771238 ] Francis Liu commented on HBASE-9364: [~ndimiduk] LGTM. You've also added a rest unit test to verify. Thanks for rebasing the patch. Vandana is on leave so I'll be acting in her stead. Get request with multiple columns returns partial results - Key: HBASE-9364 URL: https://issues.apache.org/jira/browse/HBASE-9364 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.94.11 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9364.00.patch, HBASE-9364.01.patch, hbase-9364_trunk.00.patch, HBASE-9364_trunk.01.patch, HBASE-9364_trunk.02.patch, HBASE-9364_trunk.02.patch, HBASE-9364_trunk.03.patch, HBASE-9364_trunk.03.patch, HBASE-9364_trunk.04.patch When a GET request is issue for a table row with multiple columns and columns have empty qualifier like f1: , results for empty qualifiers is being ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9472) If the memstore size is under .1 or greater than .9 the memstore size defaults to the default memstore size
[
https://issues.apache.org/jira/browse/HBASE-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
churro morales updated HBASE-9472:
--
Attachment: HBASE-9742-0.94.patch
For regionservers with large heaps ~ 20GB, it might not make sense to have a
restriction where the memstore size has to be a certain percentage of the heap
size. Instead the logic used is the memstore has to be either 100MB or 10% of
the heap.
If the memstore size is under .1 or greater than .9 the memstore size
defaults to the default memstore size
---
Key: HBASE-9472
URL: https://issues.apache.org/jira/browse/HBASE-9472
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: churro morales
Attachments: HBASE-9742-0.94.patch
In HbaseConfiguration.checkForClusterFreeMemoryLimit it does a check to see
if the blockCache + memstore .8 this threshold ensures we do not run out of
memory.
But MemStoreFlusher.getMemStoreLimit does this check:
{code}
if (limit = 0.9f || limit 0.1f) {
LOG.warn(Setting global memstore limit to default of + defaultLimit +
because supplied value outside allowed range of 0.1 - 0.9);
effectiveLimit = defaultLimit;
}
{code}
In our cluster we had the block cache set to an upper limit of 0.76 and the
memstore upper limit was set to 0.04. We noticed the memstore size was
exceeding the limit we had set and after looking at the getMemStoreLimit code
it seems that the memstore upper limit is sized to the default value if the
configuration value is less than .1 or greater than .9. This now makes the
block cache and memstore greater than our available heap.
We can remove the check for the greater than 90% of the heap as this can
never happen due to the check in
HbaseConfiguration.checkForClusterFreeMemoryLimit()
This check doesn't seem necessary anymore as we have the HbaseConfiguration
class checking for the cluster free limit. Am I correct in this assumption?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9472) If the memstore size is under .1 or greater than .9 the memstore size defaults to the default memstore size
[
https://issues.apache.org/jira/browse/HBASE-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771242#comment-13771242
]
churro morales commented on HBASE-9472:
---
If you guys feel this logic is appropriate I can provide a patch for trunk.
If the memstore size is under .1 or greater than .9 the memstore size
defaults to the default memstore size
---
Key: HBASE-9472
URL: https://issues.apache.org/jira/browse/HBASE-9472
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: churro morales
Attachments: HBASE-9742-0.94.patch
In HbaseConfiguration.checkForClusterFreeMemoryLimit it does a check to see
if the blockCache + memstore .8 this threshold ensures we do not run out of
memory.
But MemStoreFlusher.getMemStoreLimit does this check:
{code}
if (limit = 0.9f || limit 0.1f) {
LOG.warn(Setting global memstore limit to default of + defaultLimit +
because supplied value outside allowed range of 0.1 - 0.9);
effectiveLimit = defaultLimit;
}
{code}
In our cluster we had the block cache set to an upper limit of 0.76 and the
memstore upper limit was set to 0.04. We noticed the memstore size was
exceeding the limit we had set and after looking at the getMemStoreLimit code
it seems that the memstore upper limit is sized to the default value if the
configuration value is less than .1 or greater than .9. This now makes the
block cache and memstore greater than our available heap.
We can remove the check for the greater than 90% of the heap as this can
never happen due to the check in
HbaseConfiguration.checkForClusterFreeMemoryLimit()
This check doesn't seem necessary anymore as we have the HbaseConfiguration
class checking for the cluster free limit. Am I correct in this assumption?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6925: - Component/s: IPC/RPC Release Note: Up RPC base buffer size from 8k to 64k. Improves scanner throughput. Change socket write size from 8K to 64K for HBaseServer --- Key: HBASE-6925 URL: https://issues.apache.org/jira/browse/HBASE-6925 Project: HBase Issue Type: Sub-task Components: IPC/RPC, Performance Reporter: Karthik Ranganathan Assignee: Karthik Ranganathan Priority: Critical Fix For: 0.94.3, 0.95.0 Attachments: HBASE-6925.patch Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
