[jira] [Commented] (HBASE-11315) Keeping MVCC for configurable longer time
[ https://issues.apache.org/jira/browse/HBASE-11315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053376#comment-14053376 ] Hudson commented on HBASE-11315: FAILURE: Integrated in HBase-1.0 #16 (See [https://builds.apache.org/job/HBase-1.0/16/]) HBase-11315: Keeping MVCC for configurable longer time (jeffreyz: rev cc4f54770bc9b517f561a6fefcca087ba15c79c2) * hbase-common/src/test/java/org/apache/hadoop/hbase/codec/TestCellCodec.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeCell.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/row/data/TestRowDataDifferentTimestamps.java * hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteRangeWithKVSerialization.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestPayloadCarryingRpcController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * hbase-protocol/src/main/protobuf/WAL.proto * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java Keeping MVCC for configurable longer time -- Key: HBASE-11315 URL: https://issues.apache.org/jira/browse/HBASE-11315 Project: HBase Issue Type: Improvement Affects Versions: 0.99.0 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Attachments: hbase-11315-v1.patch, hbase-11315.patch After hbase-8763, we need keep mvcc number longer in hfile so that it can be used to order changes during writes. For example, the known put,delete,put,... scenario, cross region server scan, out of order puts(in recovery case). Current thinking is that we make the retention period configurable(below we're using 1 day to explain). During major compaction, we check hfile's creation time if a hfile creation time is older than 1 day then all mvcc of KVs in that hfile will be removed. If a hfile is created within 1 day, then all mvccs of KVs in that hfile will be kept. In case there are time clock skew, we can firstly sort hfiles based on its seqId in ascending order and find the first hfile's creation time stamp less than 1 day. Then mvcc of all hfiles before the found file will be removed during compaction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11317) Expand unit testing to cover Mockito and MRUnit and give more examples
[
https://issues.apache.org/jira/browse/HBASE-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053388#comment-14053388
]
Hadoop QA commented on HBASE-11317:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654267/HBASE-11317.patch
against trunk revision .
ATTACHMENT ID: 12654267
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+
xlink:href=http://blog.cloudera.com/blog/2013/09/how-to-test-hbase-applications-using-popular-tools/;http://blog.cloudera.com/blog/2013/09/how-to-test-hbase-applications-using-popular-tools//link.
+The following sections discuss JUnit, Mockito, MRUnit, and
HBaseTestingUtility. /para
+ assertEquals(obj.getData1(), Bytes.toString(put.get(Bytes.toBytes(CF),
Bytes.toBytes(CQ-1)).get(0).getValue()));
+ assertEquals(obj.getData2(), Bytes.toString(put.get(Bytes.toBytes(CF),
Bytes.toBytes(CQ-2)).get(0).getValue()));/userinput
+
xlink:href=https://github.com/junit-team/junit/wiki/Getting-started;https://github.com/junit-team/junit/wiki/Getting-started/link.
+
xlink:href=https://code.google.com/p/mockito/;https://code.google.com/p/mockito//link./para
+
assertEquals(Bytes.toString(put.get(Bytes.toBytes(CF),Bytes.toBytes(CQ-1)).get(0).getValue()),
DATA-1);
+
assertEquals(Bytes.toString(put.get(Bytes.toBytes(CF),Bytes.toBytes(CQ-2)).get(0).getValue()),
DATA-2);
+paraThis code populates codeHBaseTestObj/code with
âROWKEY-1â, âDATA-1â,
+âDATA-2â as values. It then inserts the record into
the mocked table. The Put
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.testMROnTable(TestImportTSVWithVisibilityLabels.java:162)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9977//console
This message is automatically generated.
Expand unit testing to cover Mockito and MRUnit and give more examples
--
Key: HBASE-11317
URL: https://issues.apache.org/jira/browse/HBASE-11317
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Mike Drob
Assignee: Misty Stanley-Jones
Priority: Trivial
Attachments: HBASE-11317.patch
The section at
[jira] [Commented] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[
https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053404#comment-14053404
]
Hadoop QA commented on HBASE-9005:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654273/HBASE-11317.patch
against trunk revision .
ATTACHMENT ID: 12654273
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+
xlink:href=http://blog.cloudera.com/blog/2013/09/how-to-test-hbase-applications-using-popular-tools/;http://blog.cloudera.com/blog/2013/09/how-to-test-hbase-applications-using-popular-tools//link.
+The following sections discuss JUnit, Mockito, MRUnit, and
HBaseTestingUtility. /para
+ assertEquals(obj.getData1(), Bytes.toString(put.get(Bytes.toBytes(CF),
Bytes.toBytes(CQ-1)).get(0).getValue()));
+ assertEquals(obj.getData2(), Bytes.toString(put.get(Bytes.toBytes(CF),
Bytes.toBytes(CQ-2)).get(0).getValue()));/userinput
+
xlink:href=https://github.com/junit-team/junit/wiki/Getting-started;https://github.com/junit-team/junit/wiki/Getting-started/link.
+
xlink:href=https://code.google.com/p/mockito/;https://code.google.com/p/mockito//link./para
+
assertEquals(Bytes.toString(put.get(Bytes.toBytes(CF),Bytes.toBytes(CQ-1)).get(0).getValue()),
DATA-1);
+
assertEquals(Bytes.toString(put.get(Bytes.toBytes(CF),Bytes.toBytes(CQ-2)).get(0).getValue()),
DATA-2);
+paraThis code populates codeHBaseTestObj/code with
âROWKEY-1â, âDATA-1â,
+âDATA-2â as values. It then inserts the record into
the mocked table. The Put
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9978//console
This message is automatically generated.
Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete
markers
-
Key: HBASE-9005
URL: https://issues.apache.org/jira/browse/HBASE-9005
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Lars Hofhansl
Assignee: Misty Stanley-Jones
Priority: Minor
Fix For: 0.99.0
Attachments: 9005.txt, HBASE-11317.patch
Without KEEP_DELETED_CELLS all timerange queries are broken if their range
covers a delete marker.
As some internal discussions with colleagues showed, this
[jira] [Updated] (HBASE-11447) Proposal for a generic transaction API for HBase
[ https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John de Roo updated HBASE-11447: Attachment: Proposal for a common transactional API for HBase v0.3_1.pdf Update of the original proposal. This includes all feedback so far, but does not yet include a factory class. Proposal for a generic transaction API for HBase Key: HBASE-11447 URL: https://issues.apache.org/jira/browse/HBASE-11447 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 1.0.0 Environment: Any. Reporter: John de Roo Priority: Minor Labels: features, newbie Fix For: 1.0.0 Attachments: Open Transaction Interface.pdf, Proposal for a common transactional API for HBase v0.3_1.pdf, Re Proposal for a generic transaction API for HBase.htm HBase transaction management today is provided by a number of products, each implementing a different API, each having different strengths. The lack of a common API for transactional interfaces means that applications need to be coded to work with a specific Transaction Manager. This proposal outlines an API which, if implemented by the different Transaction Manager vendors would provide stability and choice to HBase application developers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11447) Proposal for a generic transaction API for HBase
[ https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John de Roo updated HBASE-11447: Attachment: (was: Open Transaction Interface.pdf) Proposal for a generic transaction API for HBase Key: HBASE-11447 URL: https://issues.apache.org/jira/browse/HBASE-11447 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 1.0.0 Environment: Any. Reporter: John de Roo Priority: Minor Labels: features, newbie Fix For: 1.0.0 Attachments: Proposal for a common transactional API for HBase v0.3_1.pdf, Re Proposal for a generic transaction API for HBase.htm HBase transaction management today is provided by a number of products, each implementing a different API, each having different strengths. The lack of a common API for transactional interfaces means that applications need to be coded to work with a specific Transaction Manager. This proposal outlines an API which, if implemented by the different Transaction Manager vendors would provide stability and choice to HBase application developers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11447) Proposal for a generic transaction API for HBase
[ https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053440#comment-14053440 ] John de Roo commented on HBASE-11447: - I've deleted the original proposal to avoid confusion. If anyone want's a copy for comparison, let me know. Proposal for a generic transaction API for HBase Key: HBASE-11447 URL: https://issues.apache.org/jira/browse/HBASE-11447 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 1.0.0 Environment: Any. Reporter: John de Roo Priority: Minor Labels: features, newbie Fix For: 1.0.0 Attachments: Proposal for a common transactional API for HBase v0.3_1.pdf, Re Proposal for a generic transaction API for HBase.htm HBase transaction management today is provided by a number of products, each implementing a different API, each having different strengths. The lack of a common API for transactional interfaces means that applications need to be coded to work with a specific Transaction Manager. This proposal outlines an API which, if implemented by the different Transaction Manager vendors would provide stability and choice to HBase application developers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11447) Proposal for a generic transaction API for HBase
[ https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053475#comment-14053475 ] John de Roo commented on HBASE-11447: - Version 0.3 attempts to address the following issues identified in the original. If you think I've missed something or not addressed it in the best way, please let me know. Support for isolation level, read-only transactions and transaction timeout. Isolation levels are based on ANSI SQL, but does not address Snapshot isolation, MVCC and the distinction with Lock Management. It is assumed that MVCC support (without read lists) equates to Read Committed and full Snapshot isolation (with read lists) equates to Repeatable Reads. However, it may be better to provide the ability to distinguish between MVCC/Snapshot isolation and Lock Management - a concurrency control mode. This would be provided in a similar syntax to isolation level. TransactionTable should match the signature of HTable and include all methods HTable supports. Added versions of all the HTable constructors with Transaction as a parameter. Also provided a setTransaction method so that TransactionTable methods match HTable. This has some implications for threading which are also described. Removed resume and suspend. They are not needed with streamTo and constructFrom. Provided more information usage including an example. Added an iterator function. See getAll. Factory class. Several reviews identified the need for a factory class. This is the one outstanding issue not covered by the updated proposal. I'm still working on the details. Pseudo code/examples. Added a number of examples. Added ability to override TM default values in hbase-site.xml file or equivalent. Currently only specifies the isolation level and transaction timeout, but I'm sure we need more here to allow configuration of the Transaction Manager. Rationalised exceptions including removing heuristics which really only apply to heterogeneous 2PC which is not supported by the proposal. Changed the name of TransactionManager interface to TransactionServiceClient. Kept the TransactionManager interface (now TransactionServiceClient) and the Transaction class rather than combining the functions. For example it would be possible to include both begin and constructFrom methods as Transaction constructors, but we would still need a way to provide the global/static methods. I'm open to changing this if you have a better approach. 2PC interface. No 2PC interface (prepare/commit) is provided because this is an application API supporting only a single TM. Provided the option for read-only transactions. Modified transaction timeout interface to be consistent with isolation level and transaction type. Anything I've missed?? Proposal for a generic transaction API for HBase Key: HBASE-11447 URL: https://issues.apache.org/jira/browse/HBASE-11447 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 1.0.0 Environment: Any. Reporter: John de Roo Priority: Minor Labels: features, newbie Fix For: 1.0.0 Attachments: Proposal for a common transactional API for HBase v0.3_1.pdf, Re Proposal for a generic transaction API for HBase.htm HBase transaction management today is provided by a number of products, each implementing a different API, each having different strengths. The lack of a common API for transactional interfaces means that applications need to be coded to work with a specific Transaction Manager. This proposal outlines an API which, if implemented by the different Transaction Manager vendors would provide stability and choice to HBase application developers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053522#comment-14053522 ] ramkrishna.s.vasudevan commented on HBASE-10885: Just to add on The deletes happen based on pattern matching. If suppose a Cell has SECRETTOPSECRET as the visibility expression then any delete coming with the same expression is matched for, but TOPSECRETSECRET is also considered to be same AB=BA A|B=B|A. This is the difference between what Accumulo does in terms of delete where only the exact match is looked out for. Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_0.98_1.patch, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_branch_1.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v15.patch, HBASE-10885_v17.patch, HBASE-10885_v17.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053546#comment-14053546 ] Anoop Sam John commented on HBASE-10885: Ram, can you add release notes pls Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_0.98_1.patch, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_branch_1.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v15.patch, HBASE-10885_v17.patch, HBASE-10885_v17.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[ https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11384: --- Attachment: HBASE-11384_1.patch Updated patch. Address some of the review comments except the checkAuths() comment because that is the place where the labels are extracted out from the label expression. Correct all the testcases to pass with the new behaviour. [Visibility Controller]Check for users covering authorizations for every mutation - Key: HBASE-11384 URL: https://issues.apache.org/jira/browse/HBASE-11384 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.5 Attachments: HBASE-11384.patch, HBASE-11384_1.patch As part of discussions, it is better that every mutation either Put/Delete with Visibility expressions should validate if the expression has labels for which the user has authorization. If not fail the mutation. Suppose User A is assoicated with A,B and C. The put has a visibility expression AD. Then fail the mutation as D is not associated with User A. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[ https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11384: --- Status: Patch Available (was: Open) [Visibility Controller]Check for users covering authorizations for every mutation - Key: HBASE-11384 URL: https://issues.apache.org/jira/browse/HBASE-11384 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.5 Attachments: HBASE-11384.patch, HBASE-11384_1.patch As part of discussions, it is better that every mutation either Put/Delete with Visibility expressions should validate if the expression has labels for which the user has authorization. If not fail the mutation. Suppose User A is assoicated with A,B and C. The put has a visibility expression AD. Then fail the mutation as D is not associated with User A. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11463) (findbugs) HE: Class defines equals() and uses Object.hashCode()
[
https://issues.apache.org/jira/browse/HBASE-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053611#comment-14053611
]
Mike Drob commented on HBASE-11463:
---
Ah, missed that because everything ran fine in my IDE. Thanks for the fix and
applying, [~saint@gmail.com]!
(findbugs) HE: Class defines equals() and uses Object.hashCode()
Key: HBASE-11463
URL: https://issues.apache.org/jira/browse/HBASE-11463
Project: HBase
Issue Type: Bug
Reporter: Mike Drob
Assignee: Mike Drob
Priority: Trivial
Labels: findbugs
Fix For: 0.99.0, 2.0.0
Attachments: 11463v3.txt, HBASE-11463.patch, HBASE-11463.patch,
HBASE-11463.patch.txt, HBASE-11463.v2.patch.txt
Findbugs warns that several classes define {{equals}} but not {{hashcode}}:
{noformat}
HE: Class defines equals() and uses Object.hashCode() (HE_EQUALS_USE_HASHCODE)
This class overrides equals(Object), but does not override hashCode(), and
inherits the implementation of hashCode() from java.lang.Object (which
returns the identity hash code, an arbitrary value assigned to the object by
the VM). Therefore, the class is very likely to violate the invariant that
equal objects must have equal hashcodes.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[
https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053618#comment-14053618
]
Hadoop QA commented on HBASE-11384:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654305/HBASE-11384_1.patch
against trunk revision .
ATTACHMENT ID: 12654305
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ public static final ImmutableBytesWritable CHECK_AUTHS_FOR_MUTATION_KEY
= new ImmutableBytesWritable(
+ VisibilityClient.setAuths(conf, new String[] { SECRET, CONFIDENTIAL,
}, SUPERUSER.getShortName());
+ private static HTable createTableAndWriteDataWithLabels(final TableName
tableName, final String... labelExps)
+PrivilegedExceptionActionVisibilityLabelsResponse action = new
PrivilegedExceptionActionVisibilityLabelsResponse() {
+ put.setCellVisibility(new CellVisibility(( + CONFIDENTIAL + +
PRIVATE + )|( + TOPSECRET
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManager
org.apache.hadoop.hbase.rest.TestScannersWithLabels
org.apache.hadoop.hbase.master.TestZKLessAMOnCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:197)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9979//console
This message is automatically generated.
[Visibility Controller]Check for users covering authorizations for every
mutation
-
Key: HBASE-11384
URL: https://issues.apache.org/jira/browse/HBASE-11384
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.3
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.5
Attachments: HBASE-11384.patch, HBASE-11384_1.patch
As part of discussions, it is better that every mutation either Put/Delete
with Visibility expressions should validate if the expression has labels for
which the user has authorization. If not fail the mutation.
Suppose User A is assoicated with A,B and C. The put has a visibility
expression AD. Then fail the mutation as D is not associated with User A.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11317) Expand unit testing to cover Mockito and MRUnit and give more examples
[
https://issues.apache.org/jira/browse/HBASE-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053654#comment-14053654
]
Mike Drob commented on HBASE-11317:
---
Very helpful, thanks!
Expand unit testing to cover Mockito and MRUnit and give more examples
--
Key: HBASE-11317
URL: https://issues.apache.org/jira/browse/HBASE-11317
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Mike Drob
Assignee: Misty Stanley-Jones
Priority: Trivial
Attachments: HBASE-11317.patch
The section at http://hbase.apache.org/book.html#mockito only has a {{TODO}}
where examples should go.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[ https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053682#comment-14053682 ] Lars Hofhansl commented on HBASE-9005: -- I do not see my changes in the latest attached patch. (And if I may comment on the test related changes, they seem to be a bit too specific for the HBase book :) ). Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers - Key: HBASE-9005 URL: https://issues.apache.org/jira/browse/HBASE-9005 Project: HBase Issue Type: Bug Components: documentation Reporter: Lars Hofhansl Assignee: Misty Stanley-Jones Priority: Minor Fix For: 0.99.0 Attachments: 9005.txt, HBASE-11317.patch Without KEEP_DELETED_CELLS all timerange queries are broken if their range covers a delete marker. As some internal discussions with colleagues showed, this feature is not well understand and documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10851) Wait for regionservers to join the cluster
[ https://issues.apache.org/jira/browse/HBASE-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053801#comment-14053801 ] Jimmy Xiang commented on HBASE-10851: - If you set it to distributed mode, usually it is not a single node cluster, right? In this case, can you also set the ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART to 1? Wait for regionservers to join the cluster -- Key: HBASE-10851 URL: https://issues.apache.org/jira/browse/HBASE-10851 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Critical Fix For: 0.99.0 Attachments: hbase-10851.patch, hbase-10851_v2.patch With HBASE-10569, if regionservers are started a while after the master, all regions will be assigned to the master. That may not be what users expect. A work-around is to always start regionservers before masters. I was wondering if the master can wait a little for other regionservers to join. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Release Note: Deletes can be specified with Cell Visibility as done for puts. Cells covered by the delete is found by doing pattern matching. A deleteFamily issued for row1, f1 with Cell Visibility (A B) would delete only those cells of row1 under family f1 which has cell visibility AB or BA. A delete without any cell visibility would only delete those cells that does not have any cell visibility. Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_0.98_1.patch, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_branch_1.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v15.patch, HBASE-10885_v17.patch, HBASE-10885_v17.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11437: --- Attachment: HBASE-11437.patch Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[ https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11384: --- Status: Open (was: Patch Available) [Visibility Controller]Check for users covering authorizations for every mutation - Key: HBASE-11384 URL: https://issues.apache.org/jira/browse/HBASE-11384 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.5 Attachments: HBASE-11384.patch, HBASE-11384_1.patch As part of discussions, it is better that every mutation either Put/Delete with Visibility expressions should validate if the expression has labels for which the user has authorization. If not fail the mutation. Suppose User A is assoicated with A,B and C. The put has a visibility expression AD. Then fail the mutation as D is not associated with User A. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11437: --- Fix Version/s: (was: 0.94.5) 2.0.0 0.98.5 Affects Version/s: 0.98.0 Status: Patch Available (was: Open) Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[ https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11384: --- Status: Patch Available (was: Open) [Visibility Controller]Check for users covering authorizations for every mutation - Key: HBASE-11384 URL: https://issues.apache.org/jira/browse/HBASE-11384 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.5 Attachments: HBASE-11384.patch, HBASE-11384_1.patch, HBASE-11384_2.patch As part of discussions, it is better that every mutation either Put/Delete with Visibility expressions should validate if the expression has labels for which the user has authorization. If not fail the mutation. Suppose User A is assoicated with A,B and C. The put has a visibility expression AD. Then fail the mutation as D is not associated with User A. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[ https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11384: --- Attachment: HBASE-11384_2.patch Wraps long line. TestScannersWithLabels passes now. [Visibility Controller]Check for users covering authorizations for every mutation - Key: HBASE-11384 URL: https://issues.apache.org/jira/browse/HBASE-11384 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.5 Attachments: HBASE-11384.patch, HBASE-11384_1.patch, HBASE-11384_2.patch As part of discussions, it is better that every mutation either Put/Delete with Visibility expressions should validate if the expression has labels for which the user has authorization. If not fail the mutation. Suppose User A is assoicated with A,B and C. The put has a visibility expression AD. Then fail the mutation as D is not associated with User A. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053863#comment-14053863 ] Anoop Sam John commented on HBASE-11437: IMO, Cell#getTagsLength() can be kept as deprecated in 0.98.x and 0.99 and can be removed from master (2.0.0). What do you say? Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Release Note: Deletes can be specified with Cell Visibility as done for puts. Cells covered by the delete is found by doing pattern matching. A deleteFamily issued for row1, f1 with Cell Visibility (A B) would delete only those cells of row1 under family f1 which has cell visibility AB or BA. A delete without any cell visibility would only delete those cells that does not have any cell visibility. In case of delete specific column with latest version only the latest cell with the specified cell visibility will be covered by the delete marker. In case there is no such cell with the specified cell visibility then no cell gets deleted. was: Deletes can be specified with Cell Visibility as done for puts. Cells covered by the delete is found by doing pattern matching. A deleteFamily issued for row1, f1 with Cell Visibility (A B) would delete only those cells of row1 under family f1 which has cell visibility AB or BA. A delete without any cell visibility would only delete those cells that does not have any cell visibility. Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_0.98_1.patch, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_branch_1.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v15.patch, HBASE-10885_v17.patch, HBASE-10885_v17.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053883#comment-14053883 ] ramkrishna.s.vasudevan commented on HBASE-11437: bq.Cell#getTagsLength() can be kept as deprecated in 0.98.x and 0.99 and can be removed from master (2.0.0). What do you say? +1 Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov reassigned HBASE-7767: -- Assignee: Mikhail Antonov (was: Enis Soztutar) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 0.95.2 Reporter: Enis Soztutar Assignee: Mikhail Antonov As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[
https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053904#comment-14053904
]
Hadoop QA commented on HBASE-11437:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654336/HBASE-11437.patch
against trunk revision .
ATTACHMENT ID: 12654336
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.codec.TestCellCodecWithTags
org.apache.hadoop.hbase.codec.TestKeyValueCodecWithTags
org.apache.hadoop.hbase.TestKeyValue
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9981//console
This message is automatically generated.
Modify cell tag handling code to treat the length as unsigned
-
Key: HBASE-11437
URL: https://issues.apache.org/jira/browse/HBASE-11437
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0, 0.98.5, 2.0.0
Attachments: HBASE-11437.patch
We store each tag's length and total tags length with 2 bytes in KeyValue
buffer, HFiles etc and we treat these lengths as short through out the code.
So the max length can be Short.MAX_VALUE. We can treat these lengths as
unsigned and +ve always. So we can actually treat these lengths as int and
store with 2 bytes so that the max length can reach 65535.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11458) NPEs if RegionServer cannot initialize
[ https://issues.apache.org/jira/browse/HBASE-11458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053915#comment-14053915 ] Jeffrey Zhong commented on HBASE-11458: --- +1. NPEs if RegionServer cannot initialize -- Key: HBASE-11458 URL: https://issues.apache.org/jira/browse/HBASE-11458 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11458_v1.patch If master aborts, or RS aborts before initialization is complete, we run into a lot of NPE's in region server abort / stop code path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11447) Proposal for a generic transaction API for HBase
[
https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053951#comment-14053951
]
Ted Yu commented on HBASE-11447:
{code}
public Transaction begin(final TransactionIsolationLevel isolationLevel,
final int seconds);
{code}
Name the second parameter timeout or timeoutInSeconds ?
{code}
public Byte[] streamTo();
{code}
How about naming the above method:
{code}
public byte[] toByteArray() throws IOException;
{code}
Would it be simpler if TransactionTable has setTransaction() only ?
What would happen if setTransaction() is called on a TransactionTable which has
an active Transaction associated with it ?
Would it be better if TransactionTable is called TransactionTableInterface ?
bq. Also Transaction instances created by constructFrom can be used in parallel
with the Transaction instance from the beginner and with any other calls to
constructFrom in the same or different threads or processes.
By 'in parallel', do you mean 'interchangeably' ?
bq. If the Transaction Manager does not support the specified isolation level,
a NotSupportedException is thrown.
Would it make sense to add another method which returns the supported isolation
levels ?
Proposal for a generic transaction API for HBase
Key: HBASE-11447
URL: https://issues.apache.org/jira/browse/HBASE-11447
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 1.0.0
Environment: Any.
Reporter: John de Roo
Priority: Minor
Labels: features, newbie
Fix For: 1.0.0
Attachments: Proposal for a common transactional API for HBase
v0.3_1.pdf, Re Proposal for a generic transaction API for HBase.htm
HBase transaction management today is provided by a number of products, each
implementing a different API, each having different strengths. The lack of a
common API for transactional interfaces means that applications need to be
coded to work with a specific Transaction Manager. This proposal outlines an
API which, if implemented by the different Transaction Manager vendors would
provide stability and choice to HBase application developers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053963#comment-14053963 ] Nick Dimiduk commented on HBASE-8: -- [~stack] FYI, building my fat-jar project with -Dhbase.version=0.98.4-8-SNAPSHOT isn't pulling the snapshot jars from the apache repo. I don't know if it's something wrong with that project's pom or with the snapshot release itself. non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053984#comment-14053984 ] Nick Dimiduk commented on HBASE-8: -- You patch includes the necessary changes to pom.xml files, excellent! I installed this build and ran the fat-jar tests. This patch fixes the issue, at least as far as this test case is concerned. Nice one [~stack]! non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11465) [VisibilityController] Reserved tags check not happening for Append/Increment
[ https://issues.apache.org/jira/browse/HBASE-11465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053993#comment-14053993 ] Enis Soztutar commented on HBASE-11465: --- bq. This is a bug fix and so Enis Soztutar should be fine on commiting this to branch-1 I believe Yep. Late +1. Thanks Anoop. [VisibilityController] Reserved tags check not happening for Append/Increment - Key: HBASE-11465 URL: https://issues.apache.org/jira/browse/HBASE-11465 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11465.patch, HBASE-11465.patch, HBASE-11465_0.98.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11466) HConnectionImplementation should not use ZK
Mikhail Antonov created HBASE-11466: --- Summary: HConnectionImplementation should not use ZK Key: HBASE-11466 URL: https://issues.apache.org/jira/browse/HBASE-11466 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 2.0.0 Reporter: Mikhail Antonov Fix For: 2.0.0 Currently ConnectionManager.HConnectionImplementation uses ZK to get address of current master. Should instead use pluggable interface to get location of master to connect to (current active master in the cluster, until we have multiple active masters) elsewhere (e.g. round-robin over the list of masters set in the client's hbase-site.xml) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11384) [Visibility Controller]Check for users covering authorizations for every mutation
[
https://issues.apache.org/jira/browse/HBASE-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054000#comment-14054000
]
Hadoop QA commented on HBASE-11384:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654337/HBASE-11384_2.patch
against trunk revision .
ATTACHMENT ID: 12654337
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 25 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManager
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3301)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9980//console
This message is automatically generated.
[Visibility Controller]Check for users covering authorizations for every
mutation
-
Key: HBASE-11384
URL: https://issues.apache.org/jira/browse/HBASE-11384
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.3
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.5
Attachments: HBASE-11384.patch, HBASE-11384_1.patch,
HBASE-11384_2.patch
As part of discussions, it is better that every mutation either Put/Delete
with Visibility expressions should validate if the expression has labels for
which the user has authorization. If not fail the mutation.
Suppose User A is assoicated with A,B and C. The put has a visibility
expression AD. Then fail the mutation as D is not associated with User A.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11466) HConnectionImplementation should not use ZK
[ https://issues.apache.org/jira/browse/HBASE-11466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11466: Description: Currently ConnectionManager.HConnectionImplementation uses ZK to get address of current master. Should instead use pluggable interface to get location of master to connect to (current active master in the cluster, until we have multiple active masters) elsewhere (e.g. round-robin over the list of masters set in the client's hbase-site.xml). Currently it uses MasterAddressTracker, which reads from ZK, and this is the only place where MasterAddressTracker is used on the client side (except ZkUtil util method which dumps ZK namespace to log). So implementation of failover proxy which fails over multiple masters will probably used only here. was:Currently ConnectionManager.HConnectionImplementation uses ZK to get address of current master. Should instead use pluggable interface to get location of master to connect to (current active master in the cluster, until we have multiple active masters) elsewhere (e.g. round-robin over the list of masters set in the client's hbase-site.xml) HConnectionImplementation should not use ZK --- Key: HBASE-11466 URL: https://issues.apache.org/jira/browse/HBASE-11466 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 2.0.0 Reporter: Mikhail Antonov Fix For: 2.0.0 Currently ConnectionManager.HConnectionImplementation uses ZK to get address of current master. Should instead use pluggable interface to get location of master to connect to (current active master in the cluster, until we have multiple active masters) elsewhere (e.g. round-robin over the list of masters set in the client's hbase-site.xml). Currently it uses MasterAddressTracker, which reads from ZK, and this is the only place where MasterAddressTracker is used on the client side (except ZkUtil util method which dumps ZK namespace to log). So implementation of failover proxy which fails over multiple masters will probably used only here. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11467) New impl of Registry interface not using ZK + new RPCs on master protocol.
Mikhail Antonov created HBASE-11467: --- Summary: New impl of Registry interface not using ZK + new RPCs on master protocol. Key: HBASE-11467 URL: https://issues.apache.org/jira/browse/HBASE-11467 Project: HBase Issue Type: Sub-task Reporter: Mikhail Antonov Currently there' only one implementation of Registry interface, which is using ZK to get info about meta. Need to create implementation which will be using RPC calls to master the client is connected to. That includes adding several new methods to master RPC protocol: - GetMetaRegionLocation - GetClusterId - IsTableOnlineState - GetCurrentNrHRS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11467) New impl of Registry interface not using ZK + new RPCs on master protocol
[ https://issues.apache.org/jira/browse/HBASE-11467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11467: Summary: New impl of Registry interface not using ZK + new RPCs on master protocol (was: New impl of Registry interface not using ZK + new RPCs on master protocol.) New impl of Registry interface not using ZK + new RPCs on master protocol - Key: HBASE-11467 URL: https://issues.apache.org/jira/browse/HBASE-11467 Project: HBase Issue Type: Sub-task Components: Client, Consensus, Zookeeper Reporter: Mikhail Antonov Fix For: 2.0.0 Currently there' only one implementation of Registry interface, which is using ZK to get info about meta. Need to create implementation which will be using RPC calls to master the client is connected to. That includes adding several new methods to master RPC protocol: - GetMetaRegionLocation - GetClusterId - IsTableOnlineState - GetCurrentNrHRS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11447) Proposal for a generic transaction API for HBase
[
https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054045#comment-14054045
]
Ted Yu commented on HBASE-11447:
w.r.t. exception classes, it would be nice if they have common base class which
is not Exception.
{code}
public class IllegalStateException extends Exception
{code}
IllegalStateException is defined by Java.
{code}
public class RollbackException extends Exception
{code}
Maybe name the above RolledBackException ?
{code}
namehbase.transaction.isolationlevel/name
value2/value
{code}
Would it be better if symbolic names are used to specify isolation levels ?
That way users doesn't need to look up the numeric value for the isolation
level they want.
For 5.1:
{code}
txSC = new TransactionServiceClient(conf);
Configuration config = HBaseConfiguration.create();
{code}
'config' should be used for constructing TransactionServiceClient.
For 5.3,
bq. The concurrency control method (locking or MVCC) and isolation level will
have no effect on this because both puts are performed within the same
transaction.
What's the background for the use case described in 5.3 ? Such usage would
result in indeterministic outcome when commit is called, right ?
bq. Should the TransactionServiceClass contain the begin and constructFrom
methods or should these be constructors
for the Transaction class?
Looks like constructFrom method should belong to Transaction class.
Proposal for a generic transaction API for HBase
Key: HBASE-11447
URL: https://issues.apache.org/jira/browse/HBASE-11447
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 1.0.0
Environment: Any.
Reporter: John de Roo
Priority: Minor
Labels: features, newbie
Fix For: 1.0.0
Attachments: Proposal for a common transactional API for HBase
v0.3_1.pdf, Re Proposal for a generic transaction API for HBase.htm
HBase transaction management today is provided by a number of products, each
implementing a different API, each having different strengths. The lack of a
common API for transactional interfaces means that applications need to be
coded to work with a specific Transaction Manager. This proposal outlines an
API which, if implemented by the different Transaction Manager vendors would
provide stability and choice to HBase application developers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11349) [Thrift] support authentication/impersonation
[ https://issues.apache.org/jira/browse/HBASE-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-11349: Attachment: hbase-11349_v2.patch Attached v2 that creates a proxy user object only when needed. [Thrift] support authentication/impersonation - Key: HBASE-11349 URL: https://issues.apache.org/jira/browse/HBASE-11349 Project: HBase Issue Type: Improvement Components: security, Thrift Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-11349.patch, hbase-11349_v2.patch Thrift server can access HBase as a fixed authenticated user. However, we don't authenticate thrift clients. It will be great if the thrift server can authenticate clients, and support impersonation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11468) MasterAddressTracker needs to be moved to hbase-server
Mikhail Antonov created HBASE-11468: --- Summary: MasterAddressTracker needs to be moved to hbase-server Key: HBASE-11468 URL: https://issues.apache.org/jira/browse/HBASE-11468 Project: HBase Issue Type: Sub-task Reporter: Mikhail Antonov Later we should get rid of it. For now, for the purpose to abstract client from ZK, we don't use in in hbase-client (where we read master location elsewhere), but on the server side we can use it for now, and still keep current znode tracking location of current active master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11349) [Thrift] support authentication/impersonation
[ https://issues.apache.org/jira/browse/HBASE-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-11349: Status: Patch Available (was: Open) [Thrift] support authentication/impersonation - Key: HBASE-11349 URL: https://issues.apache.org/jira/browse/HBASE-11349 Project: HBase Issue Type: Improvement Components: security, Thrift Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-11349.patch, hbase-11349_v2.patch Thrift server can access HBase as a fixed authenticated user. However, we don't authenticate thrift clients. It will be great if the thrift server can authenticate clients, and support impersonation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11469) MetaTableLocator should be moved to hbase-server
Mikhail Antonov created HBASE-11469: --- Summary: MetaTableLocator should be moved to hbase-server Key: HBASE-11469 URL: https://issues.apache.org/jira/browse/HBASE-11469 Project: HBase Issue Type: Sub-task Reporter: Mikhail Antonov It shall not be used on client side, clients shall make rpc calls to master to find out about meta. Also it needs to either be removed from Server interface available on client side and moved into RegionServer class or something, or whole Server interface needs to be moved to hbase-server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11470) Move Server interface to hbase-server
Mikhail Antonov created HBASE-11470: --- Summary: Move Server interface to hbase-server Key: HBASE-11470 URL: https://issues.apache.org/jira/browse/HBASE-11470 Project: HBase Issue Type: Sub-task Reporter: Mikhail Antonov Looks like it's not being used on client side anyway. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11471) Move TableStateManager and ZkTableStateManager to hbase-server
Mikhail Antonov created HBASE-11471: --- Summary: Move TableStateManager and ZkTableStateManager to hbase-server Key: HBASE-11471 URL: https://issues.apache.org/jira/browse/HBASE-11471 Project: HBase Issue Type: Sub-task Reporter: Mikhail Antonov -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054113#comment-14054113 ] Andrew Purtell commented on HBASE-11437: +1 bq. IMO, Cell#getTagsLength() can be kept as deprecated in 0.98.x and 0.99 and can be removed from master (2.0.0). What do you say? Another option would be to deprecate both 'short Cell#getTagsLength' and 'int Cell#getTagsLengthUnsigned' in 0.98 and 0.99 and for 2.0.0 replace both with 'int Cell#getTagsLength'. Thus, in trunk we replace a short return type with an int and keep a reasonable method name, while respecting deprecation convention. What do you think? Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralBy
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8: - Attachment: HBASE-8.00.patch HBASE-8-0.98.00.patch Attaching patches for 0.98 and master. Please consider these candidates for commit. These are basically just [~stack]'s patches, rebased to appropriate HEAD and omitting the pom changes. non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.00.patch, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[
https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054165#comment-14054165
]
Hadoop QA commented on HBASE-8:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654375/HBASE-8.00.patch
against trunk revision .
ATTACHMENT ID: 12654375
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[20,33]
error: package org.apache.commons.logging does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[21,33]
error: package org.apache.commons.logging does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[31,23]
error: cannot find symbol
[ERROR] symbol: class Log
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project hbase-protocol: Compilation failure: Compilation failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[20,33]
error: package org.apache.commons.logging does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[21,33]
error: package org.apache.commons.logging does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[31,23]
error: cannot find symbol
[ERROR] symbol: class Log
[ERROR] location: class ByteStringer
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java:[31,33]
error: cannot find symbol
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-protocol
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9983//console
This message is automatically generated.
non environment variable solution for IllegalAccessError: class
com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass
com.google.protobuf.LiteralByteString
--
Key: HBASE-8
URL: https://issues.apache.org/jira/browse/HBASE-8
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: André Kelpe
Priority: Blocker
Fix For: 0.99.0
Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt,
1118.suggested.undoing.optimization.on.clientside.txt,
1118.suggested.undoing.optimization.on.clientside.txt,
HBASE-8-0.98.00.patch, HBASE-8-0.98.patch.gz,
HBASE-8-trunk.patch.gz, HBASE-8.00.patch, shade_attempt.patch
I am running into the problem described in
https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a
newer version within cascading.hbase
(https://github.com/cascading/cascading.hbase).
One of the features of cascading.hbase is that you can use it from lingual
(http://www.cascading.org/projects/lingual/), our SQL layer for hadoop.
lingual has a notion of providers, which are fat jars that we pull down
dynamically at runtime. Those jars give users the ability to talk to any
system or format from SQL. They are added to the classpath programmatically
before we submit jobs to a hadoop cluster.
Since lingual does not know upfront , which providers are going to be used in
a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really
clunky and breaks the ease of use we had before. No other provider requires
this right now.
It would be great to have a
[jira] [Created] (HBASE-11472) Remove ZKTableStateClientSideReader class
Mikhail Antonov created HBASE-11472: --- Summary: Remove ZKTableStateClientSideReader class Key: HBASE-11472 URL: https://issues.apache.org/jira/browse/HBASE-11472 Project: HBase Issue Type: Sub-task Reporter: Mikhail Antonov On client side this is now only used by ZooKeeperRegistry. Shouldn't be needed once we have registry implementation not using ZK. On server side we should be using TableStateManager instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11466) HConnectionImplementation should not use ZK
[ https://issues.apache.org/jira/browse/HBASE-11466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11466: Component/s: Consensus HConnectionImplementation should not use ZK --- Key: HBASE-11466 URL: https://issues.apache.org/jira/browse/HBASE-11466 Project: HBase Issue Type: Sub-task Components: Client, Consensus Affects Versions: 2.0.0 Reporter: Mikhail Antonov Fix For: 2.0.0 Currently ConnectionManager.HConnectionImplementation uses ZK to get address of current master. Should instead use pluggable interface to get location of master to connect to (current active master in the cluster, until we have multiple active masters) elsewhere (e.g. round-robin over the list of masters set in the client's hbase-site.xml). Currently it uses MasterAddressTracker, which reads from ZK, and this is the only place where MasterAddressTracker is used on the client side (except ZkUtil util method which dumps ZK namespace to log). So implementation of failover proxy which fails over multiple masters will probably used only here. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralBy
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8: - Attachment: HBASE-8.01.patch HBASE-8-0.98.01.patch non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch, HBASE-8.01.patch, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[ https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054187#comment-14054187 ] Misty Stanley-Jones commented on HBASE-9005: That's because I uploaded the wrong patch. Fixed. Please review HBASE-11317 about the changes related to unit testing. Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers - Key: HBASE-9005 URL: https://issues.apache.org/jira/browse/HBASE-9005 Project: HBase Issue Type: Bug Components: documentation Reporter: Lars Hofhansl Assignee: Misty Stanley-Jones Priority: Minor Fix For: 0.99.0 Attachments: 9005.txt, HBASE-11317.patch Without KEEP_DELETED_CELLS all timerange queries are broken if their range covers a delete marker. As some internal discussions with colleagues showed, this feature is not well understand and documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[ https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-9005: --- Attachment: HBASE-9005-1.patch Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers - Key: HBASE-9005 URL: https://issues.apache.org/jira/browse/HBASE-9005 Project: HBase Issue Type: Bug Components: documentation Reporter: Lars Hofhansl Assignee: Misty Stanley-Jones Priority: Minor Fix For: 0.99.0 Attachments: 9005.txt, HBASE-9005-1.patch Without KEEP_DELETED_CELLS all timerange queries are broken if their range covers a delete marker. As some internal discussions with colleagues showed, this feature is not well understand and documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[ https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-9005: --- Attachment: (was: HBASE-11317.patch) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers - Key: HBASE-9005 URL: https://issues.apache.org/jira/browse/HBASE-9005 Project: HBase Issue Type: Bug Components: documentation Reporter: Lars Hofhansl Assignee: Misty Stanley-Jones Priority: Minor Fix For: 0.99.0 Attachments: 9005.txt, HBASE-9005-1.patch Without KEEP_DELETED_CELLS all timerange queries are broken if their range covers a delete marker. As some internal discussions with colleagues showed, this feature is not well understand and documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11458) NPEs if RegionServer cannot initialize
[ https://issues.apache.org/jira/browse/HBASE-11458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11458: -- Attachment: hbase-11458_v1-0.98.patch Here is the 0.98 patch. Ping [~andrew.purt...@gmail.com] NPEs if RegionServer cannot initialize -- Key: HBASE-11458 URL: https://issues.apache.org/jira/browse/HBASE-11458 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11458_v1-0.98.patch, hbase-11458_v1.patch If master aborts, or RS aborts before initialization is complete, we run into a lot of NPE's in region server abort / stop code path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11458) NPEs if RegionServer cannot initialize
[
https://issues.apache.org/jira/browse/HBASE-11458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054199#comment-14054199
]
Hadoop QA commented on HBASE-11458:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654387/hbase-11458_v1-0.98.patch
against trunk revision .
ATTACHMENT ID: 12654387
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9984//console
This message is automatically generated.
NPEs if RegionServer cannot initialize
--
Key: HBASE-11458
URL: https://issues.apache.org/jira/browse/HBASE-11458
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, 0.98.4, 2.0.0
Attachments: hbase-11458_v1-0.98.patch, hbase-11458_v1.patch
If master aborts, or RS aborts before initialization is complete, we run into
a lot of NPE's in region server abort / stop code path.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11473) Add BaseWALObserver class
Enis Soztutar created HBASE-11473: - Summary: Add BaseWALObserver class Key: HBASE-11473 URL: https://issues.apache.org/jira/browse/HBASE-11473 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 We lack a BaseWALObserver class for WALObserver coprocessors. Other coprocessor interfaces come with a default base class which makes life a little bit easier for coprocessor implementors. I think we can add this to 0.98+. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11473) Add BaseWALObserver class
[ https://issues.apache.org/jira/browse/HBASE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11473: -- Attachment: hbase-11473_v1.patch Trivial patch. [~andrew.purt...@gmail.com] do you want this in 0.98 ? Add BaseWALObserver class - Key: HBASE-11473 URL: https://issues.apache.org/jira/browse/HBASE-11473 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11473_v1.patch We lack a BaseWALObserver class for WALObserver coprocessors. Other coprocessor interfaces come with a default base class which makes life a little bit easier for coprocessor implementors. I think we can add this to 0.98+. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11474) [Thrift2] support authentication/impersonation
Jimmy Xiang created HBASE-11474: --- Summary: [Thrift2] support authentication/impersonation Key: HBASE-11474 URL: https://issues.apache.org/jira/browse/HBASE-11474 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang This is to do the same as HBASE-11349 for Thrift2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11475) Distributed log replay should also replay compaction events
Enis Soztutar created HBASE-11475: - Summary: Distributed log replay should also replay compaction events Key: HBASE-11475 URL: https://issues.apache.org/jira/browse/HBASE-11475 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Compaction events are written to WAL as of HBASE-2231 got committed. However, it seems that distributed log replay skips those entries without sending it to the replaying region. In contrast log split replays those. It seems that we should fix log replay to also replay the compaction events. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11461) Compilation errors are not posted back to JIRA during QA run
[ https://issues.apache.org/jira/browse/HBASE-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11461: -- Fix Version/s: 2.0.0 Compilation errors are not posted back to JIRA during QA run Key: HBASE-11461 URL: https://issues.apache.org/jira/browse/HBASE-11461 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.0.0 Attachments: 11461-v1.txt Compile errors are not posted to JIRA because checkCompilationErrors misses call to submitJiraComment Here is an example: https://builds.apache.org/job/PreCommit-HBASE-Build/9957/console -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11461) Compilation errors are not posted back to JIRA during QA run
[ https://issues.apache.org/jira/browse/HBASE-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054285#comment-14054285 ] Enis Soztutar commented on HBASE-11461: --- Ted, can you please set the fixVersion when you commit and resolve an issue. Compilation errors are not posted back to JIRA during QA run Key: HBASE-11461 URL: https://issues.apache.org/jira/browse/HBASE-11461 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.0.0 Attachments: 11461-v1.txt Compile errors are not posted to JIRA because checkCompilationErrors misses call to submitJiraComment Here is an example: https://builds.apache.org/job/PreCommit-HBASE-Build/9957/console -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054288#comment-14054288
]
Enis Soztutar commented on HBASE-11450:
---
Please use the fixVersion=2.0.0 when committing to master.
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0, 0.98.4, 0.94.22, 2.0.0
Attachments: HBASE-11450-v0.patch, HBASE-11450-v1.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-11450:
--
Fix Version/s: 2.0.0
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0, 0.98.4, 0.94.22, 2.0.0
Attachments: HBASE-11450-v0.patch, HBASE-11450-v1.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11461) Compilation errors are not posted back to JIRA during QA run
[ https://issues.apache.org/jira/browse/HBASE-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054289#comment-14054289 ] Ted Yu commented on HBASE-11461: Will set fixVersion in the future. Compilation errors are not posted back to JIRA during QA run Key: HBASE-11461 URL: https://issues.apache.org/jira/browse/HBASE-11461 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.0.0 Attachments: 11461-v1.txt Compile errors are not posted to JIRA because checkCompilationErrors misses call to submitJiraComment Here is an example: https://builds.apache.org/job/PreCommit-HBASE-Build/9957/console -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11344) Hide row keys and such from the web UIs
[ https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054290#comment-14054290 ] Enis Soztutar commented on HBASE-11344: --- Please use 2.0.0 when committing to master. Hide row keys and such from the web UIs --- Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 2.0.0 Attachments: 11344-1.txt, 11344-2.txt, 11344-3.txt, 11344-committed.txt The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11344) Hide row keys and such from the web UIs
[ https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11344: -- Fix Version/s: 2.0.0 Hide row keys and such from the web UIs --- Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 2.0.0 Attachments: 11344-1.txt, 11344-2.txt, 11344-3.txt, 11344-committed.txt The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[
https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054291#comment-14054291
]
Hadoop QA commented on HBASE-9005:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654385/HBASE-9005-1.patch
against trunk revision .
ATTACHMENT ID: 12654385
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.snapshot.TestExportSnapshot
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.master.TestTableLockManager.testTableReadLock(TestTableLockManager.java:337)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9986//console
This message is automatically generated.
Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete
markers
-
Key: HBASE-9005
URL: https://issues.apache.org/jira/browse/HBASE-9005
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Lars Hofhansl
Assignee: Misty Stanley-Jones
Priority: Minor
Fix For: 0.99.0
Attachments: 9005.txt, HBASE-9005-1.patch
Without KEEP_DELETED_CELLS all timerange queries are broken if their range
covers a delete marker.
As some internal discussions with colleagues showed, this feature is not well
understand and documented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11458) NPEs if RegionServer cannot initialize
[ https://issues.apache.org/jira/browse/HBASE-11458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054294#comment-14054294 ] Andrew Purtell commented on HBASE-11458: +1 NPEs if RegionServer cannot initialize -- Key: HBASE-11458 URL: https://issues.apache.org/jira/browse/HBASE-11458 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11458_v1-0.98.patch, hbase-11458_v1.patch If master aborts, or RS aborts before initialization is complete, we run into a lot of NPE's in region server abort / stop code path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11473) Add BaseWALObserver class
[ https://issues.apache.org/jira/browse/HBASE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054295#comment-14054295 ] Andrew Purtell commented on HBASE-11473: +1 Add BaseWALObserver class - Key: HBASE-11473 URL: https://issues.apache.org/jira/browse/HBASE-11473 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11473_v1.patch We lack a BaseWALObserver class for WALObserver coprocessors. Other coprocessor interfaces come with a default base class which makes life a little bit easier for coprocessor implementors. I think we can add this to 0.98+. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11344) Hide row keys and such from the web UIs
[ https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054297#comment-14054297 ] Enis Soztutar commented on HBASE-11344: --- +1 for branch-1 as well. However, I am not a big fan of adding more methods to HRI which is a public class, but we are adding InterfaceAudience.Private methods to it. No need to amend it now, but something to keep in mind if we do more changes in that area later. Hide row keys and such from the web UIs --- Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 2.0.0 Attachments: 11344-1.txt, 11344-2.txt, 11344-3.txt, 11344-committed.txt The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054296#comment-14054296 ] Andrew Purtell commented on HBASE-8: +1 for 0.98. Please fix imports on commit non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch, HBASE-8.01.patch, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11240) Print hdfs pipeline when hlog's sync is slow
[ https://issues.apache.org/jira/browse/HBASE-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11240: -- Resolution: Fixed Status: Resolved (was: Patch Available) Print hdfs pipeline when hlog's sync is slow Key: HBASE-11240 URL: https://issues.apache.org/jira/browse/HBASE-11240 Project: HBase Issue Type: Improvement Components: Operability, wal Reporter: Liu Shaohui Assignee: Liu Shaohui Fix For: 2.0.0 Attachments: HBASE-11240-trunk-v1.diff, HBASE-11240-trunk-v2.diff Sometimes the slow sync of hlog writer is because there is an abnormal datanode in the pipeline. So it will be helpful to print the pipeline of slow sync to diagnose those problems. The ultimate solution is to join the trace of HBase and HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11240) Print hdfs pipeline when hlog's sync is slow
[ https://issues.apache.org/jira/browse/HBASE-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054300#comment-14054300 ] Enis Soztutar commented on HBASE-11240: --- This looks good for branch-1 as well. But in case of a slowdown (network, etc) this will put A LOT of entries to log. Can we safeguard it so that we do not oversaturate the log. Print hdfs pipeline when hlog's sync is slow Key: HBASE-11240 URL: https://issues.apache.org/jira/browse/HBASE-11240 Project: HBase Issue Type: Improvement Components: Operability, wal Reporter: Liu Shaohui Assignee: Liu Shaohui Fix For: 2.0.0 Attachments: HBASE-11240-trunk-v1.diff, HBASE-11240-trunk-v2.diff Sometimes the slow sync of hlog writer is because there is an abnormal datanode in the pipeline. So it will be helpful to print the pipeline of slow sync to diagnose those problems. The ultimate solution is to join the trace of HBase and HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11293) Master and Region servers fail to start when hbase.master.ipc.address=0.0.0.0, hbase.regionserver.ipc.address=0.0.0.0 and Kerberos is enabled
[
https://issues.apache.org/jira/browse/HBASE-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Harp updated HBASE-11293:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Master and Region servers fail to start when
hbase.master.ipc.address=0.0.0.0, hbase.regionserver.ipc.address=0.0.0.0 and
Kerberos is enabled
-
Key: HBASE-11293
URL: https://issues.apache.org/jira/browse/HBASE-11293
Project: HBase
Issue Type: Bug
Reporter: Michael Harp
Assignee: Devaraj Das
Attachments: 11293-1.txt
Setting
{code}
hbase.master.ipc.address=0.0.0.0
hbase.regionserver.ipc.address=0.0.0.0
{code}
causes the _HOST substitution in hbase/_h...@example.com to result in
hbase/0:0:0:0:0:0:0:0...@example.com which in turn causes kerberos
authentication to fail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[
https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054314#comment-14054314
]
Hadoop QA commented on HBASE-8:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654382/HBASE-8.01.patch
against trunk revision .
ATTACHMENT ID: 12654382
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9985//console
This message is automatically generated.
non environment variable solution for IllegalAccessError: class
com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass
com.google.protobuf.LiteralByteString
--
Key: HBASE-8
URL: https://issues.apache.org/jira/browse/HBASE-8
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: André Kelpe
Priority: Blocker
Fix For: 0.99.0
Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt,
1118.suggested.undoing.optimization.on.clientside.txt,
1118.suggested.undoing.optimization.on.clientside.txt,
HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch,
HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch,
HBASE-8.01.patch, shade_attempt.patch
I am running into the problem described in
https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a
newer version within cascading.hbase
(https://github.com/cascading/cascading.hbase).
One of the features of cascading.hbase is that you can use it from lingual
(http://www.cascading.org/projects/lingual/), our SQL layer for hadoop.
lingual has a notion of providers, which are fat jars that we pull down
dynamically at runtime. Those jars give users the ability to talk to any
system or format from SQL. They are added to the classpath programmatically
before we submit jobs to a hadoop cluster.
Since lingual does not know upfront , which providers are going to be used in
a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really
clunky and breaks the ease of use we had before. No other provider requires
this right now.
It would be great to have a programmatical
[jira] [Reopened] (HBASE-11293) Master and Region servers fail to start when hbase.master.ipc.address=0.0.0.0, hbase.regionserver.ipc.address=0.0.0.0 and Kerberos is enabled
[
https://issues.apache.org/jira/browse/HBASE-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das reopened HBASE-11293:
-
[~miharp], this is not done yet. Trunk needs to have a patch.
Master and Region servers fail to start when
hbase.master.ipc.address=0.0.0.0, hbase.regionserver.ipc.address=0.0.0.0 and
Kerberos is enabled
-
Key: HBASE-11293
URL: https://issues.apache.org/jira/browse/HBASE-11293
Project: HBase
Issue Type: Bug
Reporter: Michael Harp
Assignee: Devaraj Das
Attachments: 11293-1.txt
Setting
{code}
hbase.master.ipc.address=0.0.0.0
hbase.regionserver.ipc.address=0.0.0.0
{code}
causes the _HOST substitution in hbase/_h...@example.com to result in
hbase/0:0:0:0:0:0:0:0...@example.com which in turn causes kerberos
authentication to fail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054317#comment-14054317 ] Enis Soztutar commented on HBASE-8: --- +1 for branch-1. non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch, HBASE-8.01.patch, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11473) Add BaseWALObserver class
[ https://issues.apache.org/jira/browse/HBASE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-11473. --- Resolution: Fixed Hadoop Flags: Reviewed Committed this to 0.98+. Thanks Andrew for review. Add BaseWALObserver class - Key: HBASE-11473 URL: https://issues.apache.org/jira/browse/HBASE-11473 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11473_v1.patch We lack a BaseWALObserver class for WALObserver coprocessors. Other coprocessor interfaces come with a default base class which makes life a little bit easier for coprocessor implementors. I think we can add this to 0.98+. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11315) Keeping MVCC for configurable longer time
[ https://issues.apache.org/jira/browse/HBASE-11315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11315: -- Fix Version/s: 2.0.0 0.99.0 Keeping MVCC for configurable longer time -- Key: HBASE-11315 URL: https://issues.apache.org/jira/browse/HBASE-11315 Project: HBase Issue Type: Improvement Affects Versions: 0.99.0 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.99.0, 2.0.0 Attachments: hbase-11315-v1.patch, hbase-11315.patch After hbase-8763, we need keep mvcc number longer in hfile so that it can be used to order changes during writes. For example, the known put,delete,put,... scenario, cross region server scan, out of order puts(in recovery case). Current thinking is that we make the retention period configurable(below we're using 1 day to explain). During major compaction, we check hfile's creation time if a hfile creation time is older than 1 day then all mvcc of KVs in that hfile will be removed. If a hfile is created within 1 day, then all mvccs of KVs in that hfile will be kept. In case there are time clock skew, we can firstly sort hfiles based on its seqId in ascending order and find the first hfile's creation time stamp less than 1 day. Then mvcc of all hfiles before the found file will be removed during compaction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11315) Keeping MVCC for configurable longer time
[ https://issues.apache.org/jira/browse/HBASE-11315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054332#comment-14054332 ] Enis Soztutar commented on HBASE-11315: --- Please set the fix version when you commit and resolve for next time. Thanks. Keeping MVCC for configurable longer time -- Key: HBASE-11315 URL: https://issues.apache.org/jira/browse/HBASE-11315 Project: HBase Issue Type: Improvement Affects Versions: 0.99.0 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.99.0, 2.0.0 Attachments: hbase-11315-v1.patch, hbase-11315.patch After hbase-8763, we need keep mvcc number longer in hfile so that it can be used to order changes during writes. For example, the known put,delete,put,... scenario, cross region server scan, out of order puts(in recovery case). Current thinking is that we make the retention period configurable(below we're using 1 day to explain). During major compaction, we check hfile's creation time if a hfile creation time is older than 1 day then all mvcc of KVs in that hfile will be removed. If a hfile is created within 1 day, then all mvccs of KVs in that hfile will be kept. In case there are time clock skew, we can firstly sort hfiles based on its seqId in ascending order and find the first hfile's creation time stamp less than 1 day. Then mvcc of all hfiles before the found file will be removed during compaction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11476) Expand 'Conceptual View' section of Data Model chapter
Misty Stanley-Jones created HBASE-11476: --- Summary: Expand 'Conceptual View' section of Data Model chapter Key: HBASE-11476 URL: https://issues.apache.org/jira/browse/HBASE-11476 Project: HBase Issue Type: Bug Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Could use some updating and expansion to emphasize the differences between HBase and an RDBMS. I found http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_BigTable which is just excellent and we should link to it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11460) Deadlock in HMaster on masterAndZKLock in HConnectionManager
[
https://issues.apache.org/jira/browse/HBASE-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054412#comment-14054412
]
chunhui shen commented on HBASE-11460:
--
lgtm
+1 on patch
Deadlock in HMaster on masterAndZKLock in HConnectionManager
Key: HBASE-11460
URL: https://issues.apache.org/jira/browse/HBASE-11460
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.96.0
Reporter: Andrey Stepachev
Assignee: Ted Yu
Priority: Critical
Fix For: 0.99.0
Attachments: 11460-v1.txt, threads.tdump
On one of our clusters we got a deadlock in HMaster.
In a nutshell deadlock caused by using one HConnectionManager for serving
client-like calls and calls from HMaster RPC handlers.
HBaseAdmin uses HConnectionManager which takes a lock masterAndZKLock.
On the other side of this game sits TablesNamespaceManager (TNM). This class
uses HConnectionManager too (in my case for getting list of available
namespaces).
Problem is that HMaster class uses TNM for serving RPC requests.
If we look at TNM more closely, we can see, that this class is totally
synchronised.
Thats gives us a problem.
WebInterface calls request via HConnectionManager and locks masterAndZKLock.
Connection is blocking, so RpcClient will spin, awaiting for reply (while
holding lock).
That how it looks like in thread dump:
{code}
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xc8905430 (a
org.apache.hadoop.hbase.ipc.RpcClient$Call)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
- locked 0xc8905430 (a
org.apache.hadoop.hbase.ipc.RpcClient$Call)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(HConnectionManager.java:1467)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(HConnectionManager.java:2093)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(HConnectionManager.java:1819)
- locked 0xd15dc668 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3187)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:119)
- locked 0xcd0c1238 (a
org.apache.hadoop.hbase.client.RpcRetryingCaller)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
- locked 0xcd0c1238 (a
org.apache.hadoop.hbase.client.RpcRetryingCaller)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3214)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptorsByNamespace(HBaseAdmin.java:2265)
{code}
Some other client call any HMaster RPC, and it calls TablesNamespaceManager
methods, which in turn will block on HConnectionManager global lock
masterAndZKLock.
That how it looks like:
{code}
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1699)
- waiting to lock 0xd15dc668 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.isTableOnlineState(ZooKeeperRegistry.java:100)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isTableDisabled(HConnectionManager.java:874)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1027)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:852)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:72)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:119)
- locked 0xcd0ef108 (a
org.apache.hadoop.hbase.client.RpcRetryingCaller)
at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144)
at
[jira] [Updated] (HBASE-11476) Expand 'Conceptual View' section of Data Model chapter
[ https://issues.apache.org/jira/browse/HBASE-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11476: Component/s: documentation Expand 'Conceptual View' section of Data Model chapter --- Key: HBASE-11476 URL: https://issues.apache.org/jira/browse/HBASE-11476 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Could use some updating and expansion to emphasize the differences between HBase and an RDBMS. I found http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_BigTable which is just excellent and we should link to it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11437: --- Attachment: HBASE-11437.patch By mistake one line removal in Tag caused the tests to fail. This should fix it. Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch, HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[ https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054417#comment-14054417 ] Anoop Sam John commented on HBASE-11437: bq.Another option would be to deprecate both 'short Cell#getTagsLength' and 'int Cell#getTagsLengthUnsigned' in 0.98 and 0.99 and for 2.0.0 replace both with 'int Cell#getTagsLength'. Thus, in trunk we replace a short return type with an int and keep a reasonable method name, while respecting deprecation convention. What do you think? That will make the API clean in trunk atleast. Only thing is this will leave 98 and 1.0.0 with 2 deprecated APIs in Cell with out any replacements available there. That will be bit strange no? Modify cell tag handling code to treat the length as unsigned - Key: HBASE-11437 URL: https://issues.apache.org/jira/browse/HBASE-11437 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.5, 2.0.0 Attachments: HBASE-11437.patch, HBASE-11437.patch We store each tag's length and total tags length with 2 bytes in KeyValue buffer, HFiles etc and we treat these lengths as short through out the code. So the max length can be Short.MAX_VALUE. We can treat these lengths as unsigned and +ve always. So we can actually treat these lengths as int and store with 2 bytes so that the max length can reach 65535. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11467) New impl of Registry interface not using ZK + new RPCs on master protocol
[ https://issues.apache.org/jira/browse/HBASE-11467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054424#comment-14054424 ] Gary Helmling commented on HBASE-11467: --- If I understand the intent correctly, moving ClusterId discovery to a master RPC operation creates a sort of chicken-and-egg problem for token authentication. The RPC client needs to know the ClusterId of the cluster it is connecting to in order to select the correct authentication token to use. This was possible with ZK, as the ClusterId was stored in a public znode. If we move retrieval of the cluster ID to an RPC call on master, the client will not be able to authenticate, since, without the ClusterId, it does not know which token to select. I believe this will make token authentication unusable, or else we would have to special case that specific operation and make it _not_ require authentication on the master (which will be tricky in itself since authentication happens on the connection level). New impl of Registry interface not using ZK + new RPCs on master protocol - Key: HBASE-11467 URL: https://issues.apache.org/jira/browse/HBASE-11467 Project: HBase Issue Type: Sub-task Components: Client, Consensus, Zookeeper Reporter: Mikhail Antonov Fix For: 2.0.0 Currently there' only one implementation of Registry interface, which is using ZK to get info about meta. Need to create implementation which will be using RPC calls to master the client is connected to. That includes adding several new methods to master RPC protocol: - GetMetaRegionLocation - GetClusterId - IsTableOnlineState - GetCurrentNrHRS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralBy
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8: - Fix Version/s: 0.98.4 non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch, HBASE-8.01.patch, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralBy
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8: - Attachment: HBASE-8.02.patch HBASE-8-0.98.02.patch bq. Please fix imports on commit I went through all files touched by both patches, removing unused imports. Attaching the results as v02. Please let me know if something else was intended. non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt, 1118.suggested.undoing.optimization.on.clientside.txt, 1118.suggested.undoing.optimization.on.clientside.txt, HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch, HBASE-8-0.98.02.patch, HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, HBASE-8.00.patch, HBASE-8.01.patch, HBASE-8.02.patch, shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11476) Expand 'Conceptual View' section of Data Model chapter
[ https://issues.apache.org/jira/browse/HBASE-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11476: Attachment: HBASE-11476.patch Expand 'Conceptual View' section of Data Model chapter --- Key: HBASE-11476 URL: https://issues.apache.org/jira/browse/HBASE-11476 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11476.patch Could use some updating and expansion to emphasize the differences between HBase and an RDBMS. I found http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_BigTable which is just excellent and we should link to it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11477) book.xml has Docbook validity issues (again)
Misty Stanley-Jones created HBASE-11477: --- Summary: book.xml has Docbook validity issues (again) Key: HBASE-11477 URL: https://issues.apache.org/jira/browse/HBASE-11477 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11476) Expand 'Conceptual View' section of Data Model chapter
[ https://issues.apache.org/jira/browse/HBASE-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11476: Status: Patch Available (was: Open) I added some links to resources that explain it better (IMO) than the Ref Guide does. Then I tried to expand the existing doc (the Data Model chapter introduction, Conceptual View, and Physical View sections) to be more clear, including adding a second row to the table along with all the versions of com.cnn.www, because it wasn't very clear to me that the rows there were just versions of the same row. Review and feedback is appreciated. I definitely think we can do better here. I think we should lose the tabular view altogether -- I find the JSON-esque view to be much clearer. But it is also possible I have really misunderstood something and gotten something horribly wrong. By the way I started delving into this when I wanted to expand the docs about gets and scans and realized I didn't understand enough about the data model yet. I suspect I am not the only one! Expand 'Conceptual View' section of Data Model chapter --- Key: HBASE-11476 URL: https://issues.apache.org/jira/browse/HBASE-11476 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11476.patch Could use some updating and expansion to emphasize the differences between HBase and an RDBMS. I found http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_BigTable which is just excellent and we should link to it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11473) Add BaseWALObserver class
[ https://issues.apache.org/jira/browse/HBASE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054473#comment-14054473 ] Hudson commented on HBASE-11473: FAILURE: Integrated in HBase-1.0 #17 (See [https://builds.apache.org/job/HBase-1.0/17/]) HBASE-11473 Add BaseWALObserver class (enis: rev fef10e8413fa92d3d7a778197099d5657a81739c) * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.java Add BaseWALObserver class - Key: HBASE-11473 URL: https://issues.apache.org/jira/browse/HBASE-11473 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11473_v1.patch We lack a BaseWALObserver class for WALObserver coprocessors. Other coprocessor interfaces come with a default base class which makes life a little bit easier for coprocessor implementors. I think we can add this to 0.98+. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11473) Add BaseWALObserver class
[ https://issues.apache.org/jira/browse/HBASE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054472#comment-14054472 ] Hudson commented on HBASE-11473: SUCCESS: Integrated in HBase-TRUNK #5277 (See [https://builds.apache.org/job/HBase-TRUNK/5277/]) HBASE-11473 Add BaseWALObserver class (enis: rev 36d84a15158057f52c62b14d3bc4fa223f2693e6) * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java Add BaseWALObserver class - Key: HBASE-11473 URL: https://issues.apache.org/jira/browse/HBASE-11473 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11473_v1.patch We lack a BaseWALObserver class for WALObserver coprocessors. Other coprocessor interfaces come with a default base class which makes life a little bit easier for coprocessor implementors. I think we can add this to 0.98+. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11477) book.xml has Docbook validity issues (again)
[ https://issues.apache.org/jira/browse/HBASE-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11477: Attachment: HBASE-11477.patch No changes to content, only Docbook validity. book.xml has Docbook validity issues (again) Key: HBASE-11477 URL: https://issues.apache.org/jira/browse/HBASE-11477 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11477.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11477) book.xml has Docbook validity issues (again)
[ https://issues.apache.org/jira/browse/HBASE-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11477: Status: Patch Available (was: Open) book.xml has Docbook validity issues (again) Key: HBASE-11477 URL: https://issues.apache.org/jira/browse/HBASE-11477 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11477.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11458) NPEs if RegionServer cannot initialize
[ https://issues.apache.org/jira/browse/HBASE-11458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11458: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.98+. Thanks for reviews. NPEs if RegionServer cannot initialize -- Key: HBASE-11458 URL: https://issues.apache.org/jira/browse/HBASE-11458 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11458_v1-0.98.patch, hbase-11458_v1.patch If master aborts, or RS aborts before initialization is complete, we run into a lot of NPE's in region server abort / stop code path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11475) Distributed log replay should also replay compaction events
[ https://issues.apache.org/jira/browse/HBASE-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11475: -- Attachment: hbase-11475_v1.patch Here is a patch which makes log replay also replay compaction entries together with unit test coverage. [~jeffreyz] do you mind taking a look? Distributed log replay should also replay compaction events --- Key: HBASE-11475 URL: https://issues.apache.org/jira/browse/HBASE-11475 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11475_v1.patch Compaction events are written to WAL as of HBASE-2231 got committed. However, it seems that distributed log replay skips those entries without sending it to the replaying region. In contrast log split replays those. It seems that we should fix log replay to also replay the compaction events. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11475) Distributed log replay should also replay compaction events
[ https://issues.apache.org/jira/browse/HBASE-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-11475: -- Status: Patch Available (was: Open) Distributed log replay should also replay compaction events --- Key: HBASE-11475 URL: https://issues.apache.org/jira/browse/HBASE-11475 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: hbase-11475_v1.patch Compaction events are written to WAL as of HBASE-2231 got committed. However, it seems that distributed log replay skips those entries without sending it to the replaying region. In contrast log split replays those. It seems that we should fix log replay to also replay the compaction events. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11437) Modify cell tag handling code to treat the length as unsigned
[
https://issues.apache.org/jira/browse/HBASE-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054484#comment-14054484
]
Hadoop QA commented on HBASE-11437:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654436/HBASE-11437.patch
against trunk revision .
ATTACHMENT ID: 12654436
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.testMROnTable(TestImportTSVWithVisibilityLabels.java:162)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9987//console
This message is automatically generated.
Modify cell tag handling code to treat the length as unsigned
-
Key: HBASE-11437
URL: https://issues.apache.org/jira/browse/HBASE-11437
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0, 0.98.5, 2.0.0
Attachments: HBASE-11437.patch, HBASE-11437.patch
We store each tag's length and total tags length with 2 bytes in KeyValue
buffer, HFiles etc and we treat these lengths as short through out the code.
So the max length can be Short.MAX_VALUE. We can treat these lengths as
unsigned and +ve always. So we can actually treat these lengths as int and
store with 2 bytes so that the max length can reach 65535.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[
https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054506#comment-14054506
]
Hadoop QA commented on HBASE-8:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1265/HBASE-8.02.patch
against trunk revision .
ATTACHMENT ID: 1265
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9988//console
This message is automatically generated.
non environment variable solution for IllegalAccessError: class
com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass
com.google.protobuf.LiteralByteString
--
Key: HBASE-8
URL: https://issues.apache.org/jira/browse/HBASE-8
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: André Kelpe
Priority: Blocker
Fix For: 0.99.0, 0.98.4
Attachments: 8.098-0.txt, 8.098.txt, 8.bytestringer.txt,
1118.suggested.undoing.optimization.on.clientside.txt,
1118.suggested.undoing.optimization.on.clientside.txt,
HBASE-8-0.98.00.patch, HBASE-8-0.98.01.patch,
HBASE-8-0.98.02.patch, HBASE-8-0.98.patch.gz,
HBASE-8-trunk.patch.gz, HBASE-8.00.patch, HBASE-8.01.patch,
HBASE-8.02.patch, shade_attempt.patch
I am running into the problem described in
https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a
newer version within cascading.hbase
(https://github.com/cascading/cascading.hbase).
One of the features of cascading.hbase is that you can use it from lingual
(http://www.cascading.org/projects/lingual/), our SQL layer for hadoop.
lingual has a notion of providers, which are fat jars that we pull down
dynamically at runtime. Those jars give users the ability to talk to any
system or format from SQL. They are added to the classpath programmatically
before we submit jobs to a hadoop cluster.
Since lingual does not know upfront , which providers are going to be used in
a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really
clunky and breaks the ease of use we had before. No other
