[jira] [Commented] (HBASE-11873) Hbase Version CLI enhancement
[
https://issues.apache.org/jira/browse/HBASE-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128147#comment-14128147
]
Hadoop QA commented on HBASE-11873:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667602/HBASE-11873.patch
against trunk revision .
ATTACHMENT ID: 12667602
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.token.TestZKSecretWatcher
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10809//console
This message is automatically generated.
Hbase Version CLI enhancement
-
Key: HBASE-11873
URL: https://issues.apache.org/jira/browse/HBASE-11873
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Guo Ruijing
Priority: Minor
Labels: beginner
Attachments: HBASE-11873.patch
Hbase Version CLI enhancements:
1) include source code checksum.
2) change Subversion to Source code repository
Existing implementation:
hadoop@localhost p4_wspaces]$ hbase version
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: HBase 0.98.1-hadoop2
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Subversion ...
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Compiled by ...
Expected implematation:
hadoop@localhost p4_wspaces]$ hbase version
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: HBase 0.98.1-hadoop2
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Source code repository
...change Subversion to Source code repository
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Compiled by ...
2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: From source with
checksum eb1b9e8d63c302bed1168a7122d70 include source code checksum
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-11890) HBase REST Client is hard coded to http protocol
[ https://issues.apache.org/jira/browse/HBASE-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Tian reassigned HBASE-11890: -- Assignee: Qiang Tian HBase REST Client is hard coded to http protocol Key: HBASE-11890 URL: https://issues.apache.org/jira/browse/HBASE-11890 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.2 Reporter: Eric Yang Assignee: Qiang Tian HBase REST Client executePathOnly only supports http. It would be nice if there is a option to enable REST API client to connect through SSL. org.apache.hadoop.hbase.rest.client.Cluster class does not indicate which protocol can be used, we can either set flag in Cluster class or introduce a parameter in Client class to toggle SSL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11805) KeyValue to Cell Convert in WALEdit APIs
[ https://issues.apache.org/jira/browse/HBASE-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128176#comment-14128176 ] Lars Hofhansl commented on HBASE-11805: --- +1 on reverting this from 0.98 and adding the deprecation now. KeyValue to Cell Convert in WALEdit APIs Key: HBASE-11805 URL: https://issues.apache.org/jira/browse/HBASE-11805 Project: HBase Issue Type: Improvement Components: wal Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE-11805.patch, HBASE-11805_0.98.patch, HBASE-11805_0.98_V2.patch, HBASE-11805_0.99.patch, HBASE-11805_V2.patch, HBASE-11805_V3.patch In almost all other main interface class/APIs we have changed KeyValue to Cell. But missing in WALEdit. This is public marked for Replication (Well it should be for CP also) These 2 APIs deal with KVs add(KeyValue kv) ArrayListKeyValue getKeyValues() Suggest deprecate them and add for 0.98 add(Cell kv) ListCell getCells() And just replace from 1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11730) Document release managers for non-deprecated branches
[ https://issues.apache.org/jira/browse/HBASE-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128179#comment-14128179 ] Lars Hofhansl commented on HBASE-11730: --- BTW. a permanent release manager for a branch is something we informally made up. Other Apache project choose a RM for each release they do. I really think we should not make this role more than it is. An RM is not a decider about what goes somewhere and what not doesn't (not more than a committer would anyway), just somebody who keeps the a release (or branch) coherent. Document release managers for non-deprecated branches - Key: HBASE-11730 URL: https://issues.apache.org/jira/browse/HBASE-11730 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Attachments: HBASE-11730.patch New development goes against trunk and is backported as desired to existing release branches. From what I have seen on the jira, it looks like each branch's release manager makes the call on backporting a particular issue. We should document both this norm and who the relevant release manager is for each branch. In the current docs, I'd suggest adding the RM list to the Codelines section (18.11.1) and add a brief explanation of pinging the RM as a new section after submitting a patch again (18.12.6). Post HBASE-4593, the note about pinging a prior branch RM should just go as a bullet in the patch workflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11920: --- Attachment: HBASE-11920.patch This is the patch that would help in creating the new hooks. May be the change to Server from Stoppable is avoidable but passing Stoppable is more generic to retrive the CP host from it. But one thing to note is that this change is not possible in 0.98 just because AccessController implements RegionServerObserver. So any new hooks added would affect BC. Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Replication, security Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11920: --- Status: Patch Available (was: Open) Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Replication, security Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11920: --- Component/s: (was: security) Coprocessors Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128314#comment-14128314 ] ramkrishna.s.vasudevan commented on HBASE-11639: Adding new hooks to the REgionServerObserver will cause BC issues in 0.98 as ACL is implementing RegionServerObserver and not extending the BaseRegionServerObserver. [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 2.0.0, 0.98.7, 0.99.1 This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[
https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128330#comment-14128330
]
Hadoop QA commented on HBASE-11920:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667628/HBASE-11920.patch
against trunk revision .
ATTACHMENT ID: 12667628
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager
org.apache.hadoop.hbase.client.TestReplicaWithCluster
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.replication.TestPerTableCFReplication
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterHiddenColumn(TestRowCounter.java:137)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10810//console
This message is automatically generated.
Add CP hooks for ReplicationEndPoint
Key: HBASE-11920
URL: https://issues.apache.org/jira/browse/HBASE-11920
Project: HBase
Issue Type: Sub-task
Components: Coprocessors, Replication
Reporter: ramkrishna.s.vasudevan
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11920.patch
If we want to create internal replication endpoints other than the one
created thro configuration we may need new hooks. This is something like an
internal scanner that we create during compaction so that the actual
compaction scanner can be used as a delegator.
[~enis]
If I can give a patch by tomorrow will it be possible to include in the RC?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11805) KeyValue to Cell Convert in WALEdit APIs
[ https://issues.apache.org/jira/browse/HBASE-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11805: --- Attachment: HBASE-11805_0.98_addendum.patch Patch to revert the changes. In effect now in 0.98 the code level changes are 1. Addition of 2 new Cell based APIs in WALEdit 2. Deprecated APIs getKeyValues() and add(KeyValue) KeyValue to Cell Convert in WALEdit APIs Key: HBASE-11805 URL: https://issues.apache.org/jira/browse/HBASE-11805 Project: HBase Issue Type: Improvement Components: wal Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE-11805.patch, HBASE-11805_0.98.patch, HBASE-11805_0.98_V2.patch, HBASE-11805_0.98_addendum.patch, HBASE-11805_0.99.patch, HBASE-11805_V2.patch, HBASE-11805_V3.patch In almost all other main interface class/APIs we have changed KeyValue to Cell. But missing in WALEdit. This is public marked for Replication (Well it should be for CP also) These 2 APIs deal with KVs add(KeyValue kv) ArrayListKeyValue getKeyValues() Suggest deprecate them and add for 0.98 add(Cell kv) ListCell getCells() And just replace from 1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11761) Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+
[ https://issues.apache.org/jira/browse/HBASE-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128382#comment-14128382 ] Jonathan Hsieh commented on HBASE-11761: +1, lgtm. Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+ -- Key: HBASE-11761 URL: https://issues.apache.org/jira/browse/HBASE-11761 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Labels: beginner Attachments: HBASE-11761.patch In 0.96 we changed artifact structure, so that clients need to rely on an artifact specific to some module (hopefully hbase-client) instead of a single fat jar. We should add a FAQ item that points people towards hbase-client, to ease those updating downstream applications from 0.94 to 0.98+. Showing an example pom entry for e.g. org.apache.hbase:hbase:0.94.22 and one for e.g. org.apache.hbase:hbase-client:0.98.5 should be sufficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11933) Table level VisibilityLabelService
Anoop Sam John created HBASE-11933: -- Summary: Table level VisibilityLabelService Key: HBASE-11933 URL: https://issues.apache.org/jira/browse/HBASE-11933 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.98.5 Reporter: Anoop Sam John Assignee: Anoop Sam John HBASE-11553 added pluggable VisibilityLabelService, an abstraction layer for visibility services (add labels, set_auths , creation of visibility cell tags etc). This can be only cluster level single config now. Suggest we can enhance it to make it table level. (Might be useful in a multi-tenant scenario (?)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11933) Table level VisibilityLabelService
[ https://issues.apache.org/jira/browse/HBASE-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128386#comment-14128386 ] Anoop Sam John commented on HBASE-11933: Namespace level would be enough but we don't have namespace level configs support. (Would be nice to see whether we can add that. But not under this jira scope) Table level VisibilityLabelService -- Key: HBASE-11933 URL: https://issues.apache.org/jira/browse/HBASE-11933 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.98.5 Reporter: Anoop Sam John Assignee: Anoop Sam John HBASE-11553 added pluggable VisibilityLabelService, an abstraction layer for visibility services (add labels, set_auths , creation of visibility cell tags etc). This can be only cluster level single config now. Suggest we can enhance it to make it table level. (Might be useful in a multi-tenant scenario (?)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11920: --- Status: Open (was: Patch Available) Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11920: --- Attachment: HBASE-11920_1.patch Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch, HBASE-11920_1.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11920: --- Status: Patch Available (was: Open) Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch, HBASE-11920_1.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128402#comment-14128402 ] ramkrishna.s.vasudevan commented on HBASE-11920: bq/.org.apache.hadoop.hbase.client.TestReplicaWithCluster works with bigger timeout set for the failed test. All other test cases passes with the updated patch. Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch, HBASE-11920_1.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Affects Version/s: hbase-11339 Fix Version/s: hbase-11339 External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11644) External MOB compaction tools
[ https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11644: --- Status: Patch Available (was: Open) External MOB compaction tools - Key: HBASE-11644 URL: https://issues.apache.org/jira/browse/HBASE-11644 Project: HBase Issue Type: Sub-task Components: Compaction, master Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11644.diff From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools: 1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions. 2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones. Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11644) External MOB compaction tools
[
https://issues.apache.org/jira/browse/HBASE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128425#comment-14128425
]
Hadoop QA commented on HBASE-11644:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662941/HBASE-11644.diff
against trunk revision .
ATTACHMENT ID: 12662941
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10812//console
This message is automatically generated.
External MOB compaction tools
-
Key: HBASE-11644
URL: https://issues.apache.org/jira/browse/HBASE-11644
Project: HBase
Issue Type: Sub-task
Components: Compaction, master
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11644.diff
From the design doc, mob files are not involved in the normal HBase
compaction process. This means deleted mobs would still take up space and
that we never really merge mob files that accrue over time. Currently, MOBs
depend on two external tools:
1) A TTL cleaner that removes mobs that have passed their TTL or exceeded
minVersions.
2) A 'sweep tool' cleaner that remove mobs that have had their references
deleted and merges small files into larger ones.
Today the tools are triggered by admins. The longer term goal would be to
integrate them into hbase such that by default mobs are cleaned. The tools
will be preserved however so that advanced admins can disable automatic
cleanups and manually trigger these compaction like operaitons. #1 would
likely be a chore in the master while #2 requires some design work to
integrate into hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11721) jdiff script no longer works as usage instructions indicate
[ https://issues.apache.org/jira/browse/HBASE-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11721: --- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Commited to master. Thanks dima and thanks misty for the review. jdiff script no longer works as usage instructions indicate --- Key: HBASE-11721 URL: https://issues.apache.org/jira/browse/HBASE-11721 Project: HBase Issue Type: Bug Components: scripts Reporter: Misty Stanley-Jones Assignee: Dima Spivak Fix For: 2.0.0 Attachments: HBASE-11721-v2.patch, HBASE-11721-v3.patch, HBASE-11721-v4.patch, HBASE-11721.patch I pasted the command from the usage instructions embedded in the script, but it fails as follows: [misty@cheezel dev-support](master)$ bash ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 JDiff evaluation beginning: Determining if this is a local directory or a git repo. Looks like https://github.com/apache/hbase.git is a git repo Determining if this is a local directory or a git repo. Looks like https://github.com/MY_REPO/hbase.git is a git repo We are going to compare source 1 which is a git_repo and source 2, which is a git_repo 0.94 0.94 JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to /tmp/jdiff. % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 183 100 1830 0447 0 --:--:-- --:--:-- --:--:-- 448 Archive: jdiff-1.1.1-with-incompatible-option.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jdiff-1.1.1-with-incompatible-option.zip or jdiff-1.1.1-with-incompatible-option.zip.zip, and cannot find jdiff-1.1.1-with-incompatible-option.zip.ZIP, period. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11730) Document release managers for non-deprecated branches
[ https://issues.apache.org/jira/browse/HBASE-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128547#comment-14128547 ] stack commented on HBASE-11730: --- +1 on the [~lhofhansl] sentiment above though reading the patch, I do not see any violation. Maybe [~misty] on commit you could add a sentence derived from his formulation of what a RM does? Otherwise the patch lgtm. Fix 'maintaned' on commit. I was going to suggest that you add note that 0.96 is EOL'd but then you'd have to add a note for all before EOL'd too. Might not be bad to do? Document release managers for non-deprecated branches - Key: HBASE-11730 URL: https://issues.apache.org/jira/browse/HBASE-11730 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Attachments: HBASE-11730.patch New development goes against trunk and is backported as desired to existing release branches. From what I have seen on the jira, it looks like each branch's release manager makes the call on backporting a particular issue. We should document both this norm and who the relevant release manager is for each branch. In the current docs, I'd suggest adding the RM list to the Codelines section (18.11.1) and add a brief explanation of pinging the RM as a new section after submitting a patch again (18.12.6). Post HBASE-4593, the note about pinging a prior branch RM should just go as a bullet in the patch workflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11873) Hbase Version CLI enhancement
[ https://issues.apache.org/jira/browse/HBASE-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128550#comment-14128550 ] stack commented on HBASE-11873: --- Nice idea. How long does it take calculating the md5? Your VersionInfo is not portable. Its md5sum on some systems and md5 on others. Hbase Version CLI enhancement - Key: HBASE-11873 URL: https://issues.apache.org/jira/browse/HBASE-11873 Project: HBase Issue Type: Improvement Components: build Reporter: Guo Ruijing Priority: Minor Labels: beginner Attachments: HBASE-11873.patch Hbase Version CLI enhancements: 1) include source code checksum. 2) change Subversion to Source code repository Existing implementation: hadoop@localhost p4_wspaces]$ hbase version 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: HBase 0.98.1-hadoop2 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Subversion ... 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Compiled by ... Expected implematation: hadoop@localhost p4_wspaces]$ hbase version 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: HBase 0.98.1-hadoop2 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Source code repository ...change Subversion to Source code repository 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Compiled by ... 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: From source with checksum eb1b9e8d63c302bed1168a7122d70 include source code checksum -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11873) Hbase Version CLI enhancement
[ https://issues.apache.org/jira/browse/HBASE-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128554#comment-14128554 ] stack commented on HBASE-11873: --- Should probably show the checksum in the UI too. Hbase Version CLI enhancement - Key: HBASE-11873 URL: https://issues.apache.org/jira/browse/HBASE-11873 Project: HBase Issue Type: Improvement Components: build Reporter: Guo Ruijing Priority: Minor Labels: beginner Attachments: HBASE-11873.patch Hbase Version CLI enhancements: 1) include source code checksum. 2) change Subversion to Source code repository Existing implementation: hadoop@localhost p4_wspaces]$ hbase version 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: HBase 0.98.1-hadoop2 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Subversion ... 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Compiled by ... Expected implematation: hadoop@localhost p4_wspaces]$ hbase version 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: HBase 0.98.1-hadoop2 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Source code repository ...change Subversion to Source code repository 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: Compiled by ... 2014-09-01 03:29:40,773 INFO [main] util.VersionInfo: From source with checksum eb1b9e8d63c302bed1168a7122d70 include source code checksum -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11932) Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory
[ https://issues.apache.org/jira/browse/HBASE-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128564#comment-14128564 ] stack commented on HBASE-11932: --- This issue is not in the master branch, right [~misty] ? I tried apply and it fails. Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory - Key: HBASE-11932 URL: https://issues.apache.org/jira/browse/HBASE-11932 Project: HBase Issue Type: Bug Components: build, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11932.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11892) configs contain stale entries
[
https://issues.apache.org/jira/browse/HBASE-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128568#comment-14128568
]
stack commented on HBASE-11892:
---
What you need?
configs contain stale entries
-
Key: HBASE-11892
URL: https://issues.apache.org/jira/browse/HBASE-11892
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Sean Busbey
Assignee: Misty Stanley-Jones
Priority: Trivial
Labels: beginner
Attachments: HBASE-11892.patch
Configuration details that need to be cleaned up
* the documentation around configuring cleaner plugins have stale class names
for customizing behavior.
** {{hbase.master.logcleaner.plugins}} has LogCleanerDelegate and I think the
correct class is BaseLogCleanerDelegate.
** {{hbase.master.hfilecleaner.plugin}} has HFileCleanerDelegate instead of
BaseHFileCleanerDelegate.
* {{hbase.rpc.server.engine}} doesn't appear anywhere other than
hdfs-default.xml and the classes it references were removed by HBASE-8214
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Attachment: HBASE-7767.patch Reverted back HTableDescriptor to its original state. There is not additional key/values for TABLE_STATE. Table state now aggregated with HTD in new structure - TableDescriptor (pojo and protobuf versions). Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Patch Available (was: Open) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11892) configs contain stale entries
[
https://issues.apache.org/jira/browse/HBASE-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128601#comment-14128601
]
Sean Busbey commented on HBASE-11892:
-
I think clarification on your stated assumptions.
There is no LogCleanerDelegate interface in 0.94+ AFAICT. Same with the other
class that is mistakenly referred to. There's just the Base implementations I
mentioned.
+1 LGTM.
configs contain stale entries
-
Key: HBASE-11892
URL: https://issues.apache.org/jira/browse/HBASE-11892
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Sean Busbey
Assignee: Misty Stanley-Jones
Priority: Trivial
Labels: beginner
Attachments: HBASE-11892.patch
Configuration details that need to be cleaned up
* the documentation around configuring cleaner plugins have stale class names
for customizing behavior.
** {{hbase.master.logcleaner.plugins}} has LogCleanerDelegate and I think the
correct class is BaseLogCleanerDelegate.
** {{hbase.master.hfilecleaner.plugin}} has HFileCleanerDelegate instead of
BaseHFileCleanerDelegate.
* {{hbase.rpc.server.engine}} doesn't appear anywhere other than
hdfs-default.xml and the classes it references were removed by HBASE-8214
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11730) Document release managers for non-deprecated branches
[ https://issues.apache.org/jira/browse/HBASE-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128606#comment-14128606 ] Sean Busbey commented on HBASE-11730: - Could we just leave out EOL'd versions? Seems like if an issue is severe enough to warrant resurrecting one of them, it's going to require a dev@hbase email anyways. That would remove the entries for 0.96 and 0.94. Document release managers for non-deprecated branches - Key: HBASE-11730 URL: https://issues.apache.org/jira/browse/HBASE-11730 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Attachments: HBASE-11730.patch New development goes against trunk and is backported as desired to existing release branches. From what I have seen on the jira, it looks like each branch's release manager makes the call on backporting a particular issue. We should document both this norm and who the relevant release manager is for each branch. In the current docs, I'd suggest adding the RM list to the Codelines section (18.11.1) and add a brief explanation of pinging the RM as a new section after submitting a patch again (18.12.6). Post HBASE-4593, the note about pinging a prior branch RM should just go as a bullet in the patch workflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11761) Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+
[ https://issues.apache.org/jira/browse/HBASE-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128613#comment-14128613 ] Sean Busbey commented on HBASE-11761: - Could you reorder the examples so they're in increasing version #s? My thinking is that for people reading this entry because they need to upgrade, that will make things move from what they're familiar with to what they need to do. Is it worth including a pointer from the upgrade sections on 0.96 and 0.98 to this new FAQ entry? Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+ -- Key: HBASE-11761 URL: https://issues.apache.org/jira/browse/HBASE-11761 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Labels: beginner Attachments: HBASE-11761.patch In 0.96 we changed artifact structure, so that clients need to rely on an artifact specific to some module (hopefully hbase-client) instead of a single fat jar. We should add a FAQ item that points people towards hbase-client, to ease those updating downstream applications from 0.94 to 0.98+. Showing an example pom entry for e.g. org.apache.hbase:hbase:0.94.22 and one for e.g. org.apache.hbase:hbase-client:0.98.5 should be sufficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
Anoop Sam John created HBASE-11934:
--
Summary: Support KeyValueCodec to encode non KeyValue cells.
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.1
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This is
a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11934:
---
Attachment: HBASE-11934.patch
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11721) jdiff script no longer works as usage instructions indicate
[ https://issues.apache.org/jira/browse/HBASE-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128679#comment-14128679 ] Hudson commented on HBASE-11721: SUCCESS: Integrated in HBase-TRUNK #5488 (See [https://builds.apache.org/job/HBase-TRUNK/5488/]) HBASE-11721 jdiff script no longer works as usage instructions indicate (jmhsieh: rev 7066de63623f5c7c86f14884e44ef50a138a8330) * dev-support/jdiffHBasePublicAPI.sh jdiff script no longer works as usage instructions indicate --- Key: HBASE-11721 URL: https://issues.apache.org/jira/browse/HBASE-11721 Project: HBase Issue Type: Bug Components: scripts Reporter: Misty Stanley-Jones Assignee: Dima Spivak Fix For: 2.0.0 Attachments: HBASE-11721-v2.patch, HBASE-11721-v3.patch, HBASE-11721-v4.patch, HBASE-11721.patch I pasted the command from the usage instructions embedded in the script, but it fails as follows: [misty@cheezel dev-support](master)$ bash ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 JDiff evaluation beginning: Determining if this is a local directory or a git repo. Looks like https://github.com/apache/hbase.git is a git repo Determining if this is a local directory or a git repo. Looks like https://github.com/MY_REPO/hbase.git is a git repo We are going to compare source 1 which is a git_repo and source 2, which is a git_repo 0.94 0.94 JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to /tmp/jdiff. % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 183 100 1830 0447 0 --:--:-- --:--:-- --:--:-- 448 Archive: jdiff-1.1.1-with-incompatible-option.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jdiff-1.1.1-with-incompatible-option.zip or jdiff-1.1.1-with-incompatible-option.zip.zip, and cannot find jdiff-1.1.1-with-incompatible-option.zip.ZIP, period. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11934:
---
Status: Patch Available (was: Open)
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-9542) Have Get and MultiGet do cellblocks, currently they are pb all the time
[ https://issues.apache.org/jira/browse/HBASE-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9542. -- Resolution: Incomplete Resolving as incomplete. There is a mountain of work to do in here but needs better description and time. Putting off for now. Have Get and MultiGet do cellblocks, currently they are pb all the time --- Key: HBASE-9542 URL: https://issues.apache.org/jira/browse/HBASE-9542 Project: HBase Issue Type: Improvement Reporter: stack Priority: Critical Fix For: 0.99.1 Probably better if we cellblock Gets and MultiGets rather than pb the results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9542) Have Get and MultiGet do cellblocks, currently they are pb all the time
[ https://issues.apache.org/jira/browse/HBASE-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128761#comment-14128761 ] Anoop Sam John commented on HBASE-9542: --- Now any way we don't do bytes copy while PBing. So might not be required to do now. Have Get and MultiGet do cellblocks, currently they are pb all the time --- Key: HBASE-9542 URL: https://issues.apache.org/jira/browse/HBASE-9542 Project: HBase Issue Type: Improvement Reporter: stack Priority: Critical Fix For: 0.99.1 Probably better if we cellblock Gets and MultiGets rather than pb the results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9542) Have Get and MultiGet do cellblocks, currently they are pb all the time
[ https://issues.apache.org/jira/browse/HBASE-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128762#comment-14128762 ] Anoop Sam John commented on HBASE-9542: --- Now any way we don't do bytes copy while PBing. So might not be required to do now. Have Get and MultiGet do cellblocks, currently they are pb all the time --- Key: HBASE-9542 URL: https://issues.apache.org/jira/browse/HBASE-9542 Project: HBase Issue Type: Improvement Reporter: stack Priority: Critical Fix For: 0.99.1 Probably better if we cellblock Gets and MultiGets rather than pb the results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-9542) Have Get and MultiGet do cellblocks, currently they are pb all the time
[ https://issues.apache.org/jira/browse/HBASE-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9542: -- Comment: was deleted (was: Now any way we don't do bytes copy while PBing. So might not be required to do now.) Have Get and MultiGet do cellblocks, currently they are pb all the time --- Key: HBASE-9542 URL: https://issues.apache.org/jira/browse/HBASE-9542 Project: HBase Issue Type: Improvement Reporter: stack Priority: Critical Fix For: 0.99.1 Probably better if we cellblock Gets and MultiGets rather than pb the results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128799#comment-14128799
]
ramkrishna.s.vasudevan commented on HBASE-11934:
+1 on patch if test cases passes. This is a very important change.
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11934:
---
Component/s: Performance
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11934:
---
Priority: Critical (was: Major)
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128804#comment-14128804
]
Anoop Sam John commented on HBASE-11934:
Ping [~enis] for branch-1. The change is fully BC.
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128805#comment-14128805
]
ramkrishna.s.vasudevan commented on HBASE-11934:
CAn we create a new Codec itself for this and later raise JIRA to make this
codec default? Just aksing.
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11730) Document release managers for non-deprecated branches
[ https://issues.apache.org/jira/browse/HBASE-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128808#comment-14128808 ] stack commented on HBASE-11730: --- bq. Could we just leave out EOL'd versions? That'd work. Document release managers for non-deprecated branches - Key: HBASE-11730 URL: https://issues.apache.org/jira/browse/HBASE-11730 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Attachments: HBASE-11730.patch New development goes against trunk and is backported as desired to existing release branches. From what I have seen on the jira, it looks like each branch's release manager makes the call on backporting a particular issue. We should document both this norm and who the relevant release manager is for each branch. In the current docs, I'd suggest adding the RM list to the Codelines section (18.11.1) and add a brief explanation of pinging the RM as a new section after submitting a patch again (18.12.6). Post HBASE-4593, the note about pinging a prior branch RM should just go as a bullet in the patch workflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128811#comment-14128811
]
Nick Dimiduk commented on HBASE-11331:
--
{quote}
Left some comments at RB.
bq. Any other comments from reviewers? Are we happy with the config name
hbase.block.data.cachecompressed ?
Should this be {{hbase.block.cache.data.cachecompressed}}?
{quote}
There's not a lot of consistency around cache configurations. We also have:
- hbase.rs.cacheblocksonwrite
- hfile.block.index.cacheonwrite
- hfile.block.bloom.cacheonwrite
- hbase.rs.evictblocksonclose
- hbase.bucketcache.*
- hbase.rs.prefetchblocksonopen
- hbase.offheapcache.minblocksize
[blockcache] lazy block decompression
-
Key: HBASE-11331
URL: https://issues.apache.org/jira/browse/HBASE-11331
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 0.98.7, 0.99.1
Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch,
HBASE-11331.05-0.98.patch, HBASE-11331.05.patch, HBASE-11331.06-0.98.patch,
HBASE-11331.06.patch, HBASE-11331.07-0.98.patch, HBASE-11331.07.patch,
HBASE-11331LazyBlockDecompressperfcompare.pdf,
hbase-hbase-master-hor17n36.gq1.ygridcore.net.log, lazy-decompress.02.0.pdf,
lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf,
v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf
Maintaining data in its compressed form in the block cache will greatly
increase our effective blockcache size and should show a meaning improvement
in cache hit rates in well designed applications. The idea here is to lazily
decompress/decrypt blocks when they're consumed, rather than as soon as
they're pulled off of disk.
This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11892) configs contain stale entries
[
https://issues.apache.org/jira/browse/HBASE-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128810#comment-14128810
]
stack commented on HBASE-11892:
---
I checked. I withdraw my objection because the Interfaces I thought existed,
do not (Pardon my being lazy and not checking before filing a comment) +1 on
commit.
configs contain stale entries
-
Key: HBASE-11892
URL: https://issues.apache.org/jira/browse/HBASE-11892
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Sean Busbey
Assignee: Misty Stanley-Jones
Priority: Trivial
Labels: beginner
Attachments: HBASE-11892.patch
Configuration details that need to be cleaned up
* the documentation around configuring cleaner plugins have stale class names
for customizing behavior.
** {{hbase.master.logcleaner.plugins}} has LogCleanerDelegate and I think the
correct class is BaseLogCleanerDelegate.
** {{hbase.master.hfilecleaner.plugin}} has HFileCleanerDelegate instead of
BaseHFileCleanerDelegate.
* {{hbase.rpc.server.engine}} doesn't appear anywhere other than
hdfs-default.xml and the classes it references were removed by HBASE-8214
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128820#comment-14128820 ] Enis Soztutar commented on HBASE-11920: --- I am not sure that I get the motive for this one. Do you want to change the RE for inter cluster replication? What about other REs? Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch, HBASE-11920_1.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128821#comment-14128821
]
Hadoop QA commented on HBASE-11934:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667772/HBASE-11934.patch
against trunk revision .
ATTACHMENT ID: 12667772
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:137)
at
org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1042)
at org.apache.oozie.test.XTestCase.waitFor(XTestCase.java:686)
at
org.apache.oozie.TestCoordinatorEngineStreamLog.testCoordLogStreaming(TestCoordinatorEngineStreamLog.java:105)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10814//console
This message is automatically generated.
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian
[jira] [Commented] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128823#comment-14128823
]
Anoop Sam John commented on HBASE-11934:
May be not needed really Ram. The change is within the Codec implementation.
The bytes stream it writes to OutputStream remains the same. We can really
avoid need for one more codec and again work to make that default. Sounds ok?
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11920) Add CP hooks for ReplicationEndPoint
[ https://issues.apache.org/jira/browse/HBASE-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128828#comment-14128828 ] Anoop Sam John commented on HBASE-11920: Want to wrap the Inter cluster RE. The parent issue of this subtask speaks the motive/need Add CP hooks for ReplicationEndPoint Key: HBASE-11920 URL: https://issues.apache.org/jira/browse/HBASE-11920 Project: HBase Issue Type: Sub-task Components: Coprocessors, Replication Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0, 0.99.1 Attachments: HBASE-11920.patch, HBASE-11920_1.patch If we want to create internal replication endpoints other than the one created thro configuration we may need new hooks. This is something like an internal scanner that we create during compaction so that the actual compaction scanner can be used as a delegator. [~enis] If I can give a patch by tomorrow will it be possible to include in the RC? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128833#comment-14128833 ] stack commented on HBASE-11331: --- +1 Add to the simplification of block cache config a note that we need to unify the configuration args? Needs fat release note and on commit, add something to the refguide in the block cache section else I'm afraid folks won't come across this feature. [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch, HBASE-11331.05-0.98.patch, HBASE-11331.05.patch, HBASE-11331.06-0.98.patch, HBASE-11331.06.patch, HBASE-11331.07-0.98.patch, HBASE-11331.07.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, hbase-hbase-master-hor17n36.gq1.ygridcore.net.log, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[
https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128853#comment-14128853
]
Hadoop QA commented on HBASE-7767:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667762/HBASE-7767.patch
against trunk revision .
ATTACHMENT ID: 12667762
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 45 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+compactStoreFiles(tableDir, htd.getHTableDescriptor(), hri,
path.getName(), compactOnce, major);
+ if (scc.getTableState(table).inStates(TableState.State.DISABLED,
TableState.State.DISABLING)) {
+ * @see
org.apache.hadoop.hbase.TableDescriptors#getTableDescriptors(org.apache.hadoop.fs.FileSystem,
org.apache.hadoop.fs.Path)
+ assertEquals(Meta should be not in transition, metaState.getState(),
RegionState.State.OPEN);
+ assertEquals(Meta should be not in transition, metaState.getState(),
RegionState.State.OPEN);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient
org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook
org.apache.hadoop.hbase.coprocessor.TestBigDecimalColumnInterpreter
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.coprocessor.TestDoubleColumnInterpreter
org.apache.hadoop.hbase.zookeeper.TestTableStateManager
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.coprocessor.TestWALObserver
org.apache.hadoop.hbase.TestClusterBootOrder
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.coprocessor.TestHTableWrapper
org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.client.TestReplicaWithCluster
org.apache.hadoop.hbase.coprocessor.TestOpenTableInCoprocessor
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.client.TestSnapshotMetadata
org.apache.hadoop.hbase.snapshot.TestExportSnapshot
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
org.apache.hadoop.hbase.replication.TestPerTableCFReplication
org.apache.hadoop.hbase.TestJMXListener
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.coprocessor.TestBatchCoprocessorEndpoint
org.apache.hadoop.hbase.TestGlobalMemStoreSize
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
org.apache.hadoop.hbase.master.handler.TestCreateTableHandler
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.zookeeper.TestZooKeeperACL
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
[jira] [Created] (HBASE-11935) Unbounded creation of Replication Failover workers
Lars Hofhansl created HBASE-11935: - Summary: Unbounded creation of Replication Failover workers Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11935: -- Assignee: Jesse Yates Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-11935: Attachment: hbase-11935-0.98-v0.patch Attaching fix for 0.98. Worked up with [~lhofhansl], [~apurtell] and rest of the folks at Salesforce. Not sure if there is a good way to test for this behavior that isn't completely pointless. Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[
https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128924#comment-14128924
]
Andrew Purtell commented on HBASE-11935:
We could get unbounded ReplicationSource allocation in
ReplicationSourceManager.NodeFailoverWorker.run:
{noformat}
ReplicationTrackerZKImpl.OtherRegionServerWatcher.nodeDeleted -
ReplicationSourceManager.regionServerRemoved -
ReplicationSourceManager.transferQueues -
NodeFailoverWorker.run -
ReplicationSourceManager.getReplicationSource -
new ReplicationSource
{noformat}
Unbounded creation of Replication Failover workers
--
Key: HBASE-11935
URL: https://issues.apache.org/jira/browse/HBASE-11935
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Priority: Critical
Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1
Attachments: hbase-11935-0.98-v0.patch
We just ran into a production incident with TCP SYN storms on port 2181
(zookeeper).
In our case the slave cluster was not running. When we bounced the primary
cluster we saw an unbounded number of failover threads all hammering the
hosts on the slave ZK machines (which did not run ZK at the time)... Causing
overall degradation of network performance between datacenters.
Looking at the code we noticed that the thread pool handling of the Failover
workers was probably unintended.
Patch coming soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11935: --- Affects Version/s: 2.0.0 0.99.0 0.94.23 0.98.6 Status: Patch Available (was: Open) Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Affects Versions: 0.98.6, 0.94.23, 0.99.0, 2.0.0 Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch, hbase-11935-trunk-v0.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[
https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11935:
---
Attachment: hbase-11935-trunk-v0.patch
Patch for trunk
Replication tests pass locally:
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.replication.TestMasterReplication
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.946 sec - in
org.apache.hadoop.hbase.replication.TestMasterReplication
Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.37 sec - in
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
Running org.apache.hadoop.hbase.replication.TestPerTableCFReplication
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.85 sec - in
org.apache.hadoop.hbase.replication.TestPerTableCFReplication
Running
org.apache.hadoop.hbase.replication.TestReplicationChangingPeerRegionservers
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.904 sec - in
org.apache.hadoop.hbase.replication.TestReplicationChangingPeerRegionservers
Running org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.352 sec - in
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.599 sec - in
org.apache.hadoop.hbase.replication.TestReplicationEndpoint
Running org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.607 sec - in
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS
Running
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.881 sec - in
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
Running org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.436 sec - in
org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
Running org.apache.hadoop.hbase.replication.TestReplicationSmallTests
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.888 sec - in
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
Running org.apache.hadoop.hbase.replication.TestReplicationSource
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.846 sec - in
org.apache.hadoop.hbase.replication.TestReplicationSource
Running org.apache.hadoop.hbase.replication.TestReplicationStateZKImpl
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.465 sec - in
org.apache.hadoop.hbase.replication.TestReplicationStateZKImpl
Running org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.017 sec - in
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
Running org.apache.hadoop.hbase.replication.TestReplicationTrackerZKImpl
Tests run: 4, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 1.738 sec - in
org.apache.hadoop.hbase.replication.TestReplicationTrackerZKImpl
Running org.apache.hadoop.hbase.replication.TestReplicationWALEntryFilters
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.122 sec - in
org.apache.hadoop.hbase.replication.TestReplicationWALEntryFilters
Running org.apache.hadoop.hbase.replication.TestReplicationWithTags
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.966 sec - in
org.apache.hadoop.hbase.replication.TestReplicationWithTags
Results :
Tests run: 40, Failures: 0, Errors: 0, Skipped: 2
{noformat}
Unbounded creation of Replication Failover workers
--
Key: HBASE-11935
URL: https://issues.apache.org/jira/browse/HBASE-11935
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Priority: Critical
Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1
Attachments: hbase-11935-0.98-v0.patch, hbase-11935-trunk-v0.patch
We just ran into a production incident with TCP SYN storms on port 2181
(zookeeper).
In our case the slave cluster was not running. When we bounced the primary
cluster we saw an unbounded number of failover threads all hammering the
hosts on the slave ZK machines (which did not run ZK at the time)... Causing
overall degradation of network performance between datacenters.
Looking at the code we noticed that the thread pool handling of the Failover
[jira] [Updated] (HBASE-11930) Document new permission check to roll WAL writer
[ https://issues.apache.org/jira/browse/HBASE-11930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11930: Component/s: wal Document new permission check to roll WAL writer Key: HBASE-11930 URL: https://issues.apache.org/jira/browse/HBASE-11930 Project: HBase Issue Type: Sub-task Components: regionserver, security, wal Reporter: Andrew Purtell Fix For: 2.0.0, 0.98.7, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128976#comment-14128976 ] Lars Hofhansl commented on HBASE-11935: --- Let's add some logging too... The queue failover if very quiet. Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6 Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch, hbase-11935-trunk-v0.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[
https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129064#comment-14129064
]
Hadoop QA commented on HBASE-11935:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667811/hbase-11935-trunk-v0.patch
against trunk revision .
ATTACHMENT ID: 12667811
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.http.TestHttpServerLifecycle.testStoppedServerIsNotAlive(TestHttpServerLifecycle.java:115)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10815//console
This message is automatically generated.
Unbounded creation of Replication Failover workers
--
Key: HBASE-11935
URL: https://issues.apache.org/jira/browse/HBASE-11935
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Priority: Critical
Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1
Attachments: hbase-11935-0.98-v0.patch, hbase-11935-trunk-v0.patch
We just ran into a production incident with TCP SYN storms on port 2181
(zookeeper).
In our case the slave cluster was not running. When we bounced the primary
cluster we saw an unbounded number of failover threads all hammering the
hosts on the slave ZK machines (which did not run ZK at the time)... Causing
overall degradation of network performance between datacenters.
Looking at the code we noticed that the thread pool handling of the Failover
workers was probably unintended.
Patch coming soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-11935: Attachment: hbase-11935-0.98-v1.patch Patch for 0.98 w/ a log message every time we start a new ReplicationSource and the logs its taking over, at debug level Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6 Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch, hbase-11935-trunk-v0.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-11935: Attachment: hbase-11935-trunk-v1.patch Attaching patch for trunk with same log line added Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6 Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch, hbase-11935-trunk-v0.patch, hbase-11935-trunk-v1.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[
https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129087#comment-14129087
]
Hadoop QA commented on HBASE-11935:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667845/hbase-11935-trunk-v1.patch
against trunk revision .
ATTACHMENT ID: 12667845
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java:[590,18]
error: cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project hbase-server: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java:[590,18]
error: cannot find symbol
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10816//console
This message is automatically generated.
Unbounded creation of Replication Failover workers
--
Key: HBASE-11935
URL: https://issues.apache.org/jira/browse/HBASE-11935
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Priority: Critical
Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1
Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch,
hbase-11935-trunk-v0.patch, hbase-11935-trunk-v1.patch
We just ran into a production incident with TCP SYN storms on port 2181
(zookeeper).
In our case the slave cluster was not running. When we bounced the primary
cluster we saw an unbounded number of failover threads all hammering the
hosts on the slave ZK machines (which did not run ZK at the time)... Causing
overall degradation of network performance between datacenters.
Looking at the code we noticed that the thread pool handling of the Failover
workers was probably unintended.
Patch coming soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11935: --- Attachment: hbase-11935-trunk-v2.patch Fixed patch for trunk Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6 Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch, hbase-11935-trunk-v0.patch, hbase-11935-trunk-v1.patch, hbase-11935-trunk-v2.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[ https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129100#comment-14129100 ] Jesse Yates commented on HBASE-11935: - That's what i get for doing my coding in vim. Thanks [~apurtell] Unbounded creation of Replication Failover workers -- Key: HBASE-11935 URL: https://issues.apache.org/jira/browse/HBASE-11935 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6 Reporter: Lars Hofhansl Assignee: Jesse Yates Priority: Critical Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch, hbase-11935-trunk-v0.patch, hbase-11935-trunk-v1.patch, hbase-11935-trunk-v2.patch We just ran into a production incident with TCP SYN storms on port 2181 (zookeeper). In our case the slave cluster was not running. When we bounced the primary cluster we saw an unbounded number of failover threads all hammering the hosts on the slave ZK machines (which did not run ZK at the time)... Causing overall degradation of network performance between datacenters. Looking at the code we noticed that the thread pool handling of the Failover workers was probably unintended. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11761) Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+
[ https://issues.apache.org/jira/browse/HBASE-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11761: Attachment: HBASE-11761-v1.patch Better? Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+ -- Key: HBASE-11761 URL: https://issues.apache.org/jira/browse/HBASE-11761 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Labels: beginner Attachments: HBASE-11761-v1.patch, HBASE-11761.patch In 0.96 we changed artifact structure, so that clients need to rely on an artifact specific to some module (hopefully hbase-client) instead of a single fat jar. We should add a FAQ item that points people towards hbase-client, to ease those updating downstream applications from 0.94 to 0.98+. Showing an example pom entry for e.g. org.apache.hbase:hbase:0.94.22 and one for e.g. org.apache.hbase:hbase-client:0.98.5 should be sufficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129126#comment-14129126
]
Enis Soztutar commented on HBASE-11934:
---
KVCodec spiiling bytes in KV format makes sense I think.
Patch looks good. Two minor questions:
Maybe keep the arg as short, and force explicit cast. It seems safer:
{code}
- public static void writeShort(OutputStream out, short v) throws IOException {
+ public static void writeShort(OutputStream out, int v) throws IOException {
{code}
Also if KV serialization changes (like adding more stuff like tags, etc), we
should make sure that the serialization in the KVUtil has to change as well.
Maybe add a short comment in KV.oswrite() pointing to there. Otherwise +1 for
0.99+. It seems that RC is still blocked. We can get this in 0.99.0.
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 2.0.0, 0.99.1
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Attachment: HBASE-7767.patch fixing bug with not updated descriptor. fixing review comments from [~stack] Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Open (was: Patch Available) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11934) Support KeyValueCodec to encode non KeyValue cells.
[
https://issues.apache.org/jira/browse/HBASE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-11934:
--
Fix Version/s: (was: 0.99.1)
0.99.0
Support KeyValueCodec to encode non KeyValue cells.
---
Key: HBASE-11934
URL: https://issues.apache.org/jira/browse/HBASE-11934
Project: HBase
Issue Type: Sub-task
Components: Performance, regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 0.99.0, 2.0.0
Attachments: HBASE-11934.patch
We have KeyValueCodec as the default RPC codec now. This can only encode
KeyValue types and need to convert any non KV cell to KeyValue format. This
is a expensive op.
{code}
// This is crass and will not work when KV changes. Also if passed a non-kv
Cell, it will
// make expensive copy.
// Do not write tags over RPC
KeyValue.oswrite((KeyValue) KeyValueUtil.ensureKeyValue(cell), this.out,
false);
{code}
Now we want Cell to be in entire read path and don't want any copy until we
write the cell to client.
The optimization done in the DBE of avoiding copying of value bytes was also
not getting real advantage bacause of this ensureKeyValue() call at the Codec.
We can encode and write non KeyValue cells with KV serialization format and
avoid the recreate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Patch Available (was: Open) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
Vladimir Rodionov created HBASE-11936: - Summary: IsolationLevel must be attribute of a Query not a Scan Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11932) Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory
[ https://issues.apache.org/jira/browse/HBASE-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11932: Attachment: HBASE-11932-v1.patch Not sure what I did wrong. Trying again. Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory - Key: HBASE-11932 URL: https://issues.apache.org/jira/browse/HBASE-11932 Project: HBase Issue Type: Bug Components: build, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11932-v1.patch, HBASE-11932.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11932) Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory
[
https://issues.apache.org/jira/browse/HBASE-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129154#comment-14129154
]
Sean Busbey commented on HBASE-11932:
-
I tried on current master. Normal apply failed, but {{git apply -p0
../patches/HBASE-11932.patch}} worked.
Stop the html-single from building a html-single of every chapter and
cluttering the docbkx directory
-
Key: HBASE-11932
URL: https://issues.apache.org/jira/browse/HBASE-11932
Project: HBase
Issue Type: Bug
Components: build, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Attachments: HBASE-11932-v1.patch, HBASE-11932.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11761) Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+
[ https://issues.apache.org/jira/browse/HBASE-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129158#comment-14129158 ] Sean Busbey commented on HBASE-11761: - +1 lgtm Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+ -- Key: HBASE-11761 URL: https://issues.apache.org/jira/browse/HBASE-11761 Project: HBase Issue Type: Task Components: documentation Reporter: Sean Busbey Assignee: Misty Stanley-Jones Labels: beginner Attachments: HBASE-11761-v1.patch, HBASE-11761.patch In 0.96 we changed artifact structure, so that clients need to rely on an artifact specific to some module (hopefully hbase-client) instead of a single fat jar. We should add a FAQ item that points people towards hbase-client, to ease those updating downstream applications from 0.94 to 0.98+. Showing an example pom entry for e.g. org.apache.hbase:hbase:0.94.22 and one for e.g. org.apache.hbase:hbase-client:0.98.5 should be sufficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11936: -- Attachment: HBASE_11936.patch Patch available. IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11936: -- Fix Version/s: 0.98.7 Labels: features (was: ) Affects Version/s: 0.98.6 Status: Patch Available (was: Open) IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Labels: features Fix For: 0.98.7 Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11761) Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+
[
https://issues.apache.org/jira/browse/HBASE-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129190#comment-14129190
]
Hadoop QA commented on HBASE-11761:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667852/HBASE-11761-v1.patch
against trunk revision .
ATTACHMENT ID: 12667852
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10818//console
This message is automatically generated.
Add a FAQ item for updating a maven-managed application from 0.94 - 0.96+
--
Key: HBASE-11761
URL: https://issues.apache.org/jira/browse/HBASE-11761
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Sean Busbey
Assignee: Misty Stanley-Jones
Labels: beginner
Attachments: HBASE-11761-v1.patch, HBASE-11761.patch
In 0.96 we changed artifact structure, so that clients need to rely on an
artifact specific to some module (hopefully hbase-client) instead of a single
fat jar.
We should add a FAQ item that points people towards hbase-client, to ease
those updating downstream applications from 0.94 to 0.98+.
Showing an example pom entry for e.g. org.apache.hbase:hbase:0.94.22 and one
for e.g. org.apache.hbase:hbase-client:0.98.5 should be sufficient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11932) Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory
[
https://issues.apache.org/jira/browse/HBASE-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129191#comment-14129191
]
Hadoop QA commented on HBASE-11932:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667858/HBASE-11932-v1.patch
against trunk revision .
ATTACHMENT ID: 12667858
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestClassFinder
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10820//console
This message is automatically generated.
Stop the html-single from building a html-single of every chapter and
cluttering the docbkx directory
-
Key: HBASE-11932
URL: https://issues.apache.org/jira/browse/HBASE-11932
Project: HBase
Issue Type: Bug
Components: build, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Attachments: HBASE-11932-v1.patch, HBASE-11932.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11467) New impl of Registry interface not using ZK + new RPCs on master protocol
[ https://issues.apache.org/jira/browse/HBASE-11467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129198#comment-14129198 ] Mikhail Antonov commented on HBASE-11467: - pinging [~stack] - any thoughts on the latest patch up on RB? New impl of Registry interface not using ZK + new RPCs on master protocol - Key: HBASE-11467 URL: https://issues.apache.org/jira/browse/HBASE-11467 Project: HBase Issue Type: Sub-task Components: Client, Consensus, Zookeeper Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Fix For: 2.0.0 Attachments: HBASE-11467.patch, HBASE-11467.patch, HBASE-11467.patch Currently there' only one implementation of Registry interface, which is using ZK to get info about meta. Need to create implementation which will be using RPC calls to master the client is connected to. Review of early version of patch is here: https://reviews.apache.org/r/24296/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11937) Update patch submission guidelines to stop using --no-prefix
Sean Busbey created HBASE-11937:
---
Summary: Update patch submission guidelines to stop using
--no-prefix
Key: HBASE-11937
URL: https://issues.apache.org/jira/browse/HBASE-11937
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Sean Busbey
Priority: Minor
Right now the submission guidelines include the use of --no-prefix in the
Methods to Create Patches section:
{quote}
Git
git format-patch is preferred because it preserves commit messages. Use git
squash first, to combine smaller commits into a single larger one.
* git format-patch --no-prefix origin/master --stdout HBASE-.patch
* git diff --no-prefix origin/master HBASE-.patch
{quote}
The use of --no-prefix means that users of {{git apply}} and {{git am}} have to
know to specify {{-p0}} and (which strips 0 levels of prefix). Both of those
tools default to stripping the 1 level of prefix git makes by default.
The guide for HBase committers section on reviewing doesn't give any
suggestions on how to apply patches from contributors. It should probably give
examples for using {{git am}} and {{git apply}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11932) Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory
[ https://issues.apache.org/jira/browse/HBASE-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129202#comment-14129202 ] stack commented on HBASE-11932: --- My fault. I was trying to apply to the wrong branch. Testing, I have this in the docbkx directory now which seems like an improvement: kalashnikov:docbkx stack$ ls -la total 3360 drwxr-xr-x 17 stack staff 578 Sep 10 14:47 . drwxr-xr-x4 stack staff 136 Sep 10 14:47 .. drwxr-xr-x 228 stack staff 7752 Sep 10 14:47 book -rw-r--r--1 stack staff 1586207 Sep 10 14:47 book.html drwxr-xr-x4 stack staff 136 Sep 10 14:47 css -rw-r--r--1 stack staff88080 Sep 10 14:47 hbase-default.xml drwxr-xr-x 26 stack staff 884 Sep 10 14:47 images -rw-r--r--1 stack staff 613 Sep 10 14:47 java.html -rw-r--r--1 stack staff 571 Sep 10 14:47 ld-d0e14048.html -rw-r--r--1 stack staff 432 Sep 10 14:47 ld-d0e18269.html -rw-r--r--1 stack staff 440 Sep 10 14:47 ld-d0e18278.html -rw-r--r--1 stack staff 454 Sep 10 14:47 ld-d0e18287.html -rw-r--r--1 stack staff 447 Sep 10 14:47 ld-d0e18296.html -rw-r--r--1 stack staff 500 Sep 10 14:47 ld-d0e18339.html -rw-r--r--1 stack staff 402 Sep 10 14:47 ld-d0e23028.html -rw-r--r--1 stack staff 402 Sep 10 14:47 ld-d0e23038.html -rw-r--r--1 stack staff 402 Sep 10 14:47 ld-d0e23053.html Anything to be done about the ld-*.html stuff? +1 Stop the html-single from building a html-single of every chapter and cluttering the docbkx directory - Key: HBASE-11932 URL: https://issues.apache.org/jira/browse/HBASE-11932 Project: HBase Issue Type: Bug Components: build, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11932-v1.patch, HBASE-11932.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129204#comment-14129204 ] Andrew Purtell commented on HBASE-11936: Java binary compatibility supports moving a method upward in the class hierarchy, provided a forwarding method is left in its place. Searched around for java moving method upward. IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Labels: features Fix For: 0.98.7 Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11936: --- Fix Version/s: 0.99.1 2.0.0 Assignee: (was: Vladimir Rodionov) IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Labels: features Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11936: --- Assignee: Vladimir Rodionov IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Labels: features Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129211#comment-14129211
]
Enis Soztutar commented on HBASE-11331:
---
Is this right? Assuming HeapByteBuffer here.
{code}
@Override
public void
serialize(ByteBuffer destination) {
-ByteBuffer
dupBuf = this.buf.duplicate();
-
dupBuf.rewind();
-
destination.put(dupBuf);
+// assumes
HeapByteBuffer
+
destination.put(this.buf.array(), this.buf.arrayOffset()
+
getSerializedLength() - EXTRA_SERIALIZATION_SPACE);
serializeExtraInfo(destination);
}
{code}
On naming conf, I'll go with whatever you think is better. Agreed with Stack.
Fat release note would be good.
[blockcache] lazy block decompression
-
Key: HBASE-11331
URL: https://issues.apache.org/jira/browse/HBASE-11331
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 0.98.7, 0.99.1
Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch,
HBASE-11331.05-0.98.patch, HBASE-11331.05.patch, HBASE-11331.06-0.98.patch,
HBASE-11331.06.patch, HBASE-11331.07-0.98.patch, HBASE-11331.07.patch,
HBASE-11331LazyBlockDecompressperfcompare.pdf,
hbase-hbase-master-hor17n36.gq1.ygridcore.net.log, lazy-decompress.02.0.pdf,
lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf,
v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf
Maintaining data in its compressed form in the block cache will greatly
increase our effective blockcache size and should show a meaning improvement
in cache hit rates in well designed applications. The idea here is to lazily
decompress/decrypt blocks when they're consumed, rather than as soon as
they're pulled off of disk.
This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129214#comment-14129214 ] Vladimir Rodionov commented on HBASE-11936: --- [~apurtell] Found nothing, actually. Are you saying that it is better to keep these methods in Scan for client backward compatibility? IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Labels: features Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
[ https://issues.apache.org/jira/browse/HBASE-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129229#comment-14129229 ] Andrew Purtell commented on HBASE-11936: Sorry. This says moving a method upward in the class hierarchy is fine: http://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html This much older one - http://www.cs.cornell.edu/andru/javaspec/13.doc.html - says the same but additionally provided a forwarding method is left in its place. I'm not an expert on Java binary compatibility. IsolationLevel must be attribute of a Query not a Scan -- Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Labels: features Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE_11936.patch The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11914) YARN tests currently use more than 4GB of memory
[ https://issues.apache.org/jira/browse/HBASE-11914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129230#comment-14129230 ] Andrew Purtell commented on HBASE-11914: Missing patch? YARN tests currently use more than 4GB of memory Key: HBASE-11914 URL: https://issues.apache.org/jira/browse/HBASE-11914 Project: HBase Issue Type: Sub-task Reporter: Alex Newman Assignee: Alex Newman Much of the org.apache.hadoop.hbase.mapreduce patches use more than 4GB of memory. There's no reason for this. This patch lowers the memory required so that we can run the tests in wussy CI environments (circleci/travis-ci) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11914) YARN tests currently use more than 4GB of memory
[ https://issues.apache.org/jira/browse/HBASE-11914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129231#comment-14129231 ] Alex Newman commented on HBASE-11914: - I am still trying to figure out exactly how to do this. Soon though. It's needed for tests to pass in travisci and circle-ci YARN tests currently use more than 4GB of memory Key: HBASE-11914 URL: https://issues.apache.org/jira/browse/HBASE-11914 Project: HBase Issue Type: Sub-task Reporter: Alex Newman Assignee: Alex Newman Much of the org.apache.hadoop.hbase.mapreduce patches use more than 4GB of memory. There's no reason for this. This patch lowers the memory required so that we can run the tests in wussy CI environments (circleci/travis-ci) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[
https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129238#comment-14129238
]
Hadoop QA commented on HBASE-11935:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667848/hbase-11935-trunk-v2.patch
against trunk revision .
ATTACHMENT ID: 12667848
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10817//console
This message is automatically generated.
Unbounded creation of Replication Failover workers
--
Key: HBASE-11935
URL: https://issues.apache.org/jira/browse/HBASE-11935
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Priority: Critical
Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1
Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch,
hbase-11935-trunk-v0.patch, hbase-11935-trunk-v1.patch,
hbase-11935-trunk-v2.patch
We just ran into a production incident with TCP SYN storms on port 2181
(zookeeper).
In our case the slave cluster was not running. When we bounced the primary
cluster we saw an unbounded number of failover threads all hammering the
hosts on the slave ZK machines (which did not run ZK at the time)... Causing
overall degradation of network performance between datacenters.
Looking at the code we noticed that the thread pool handling of the Failover
workers was probably unintended.
Patch coming soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129242#comment-14129242
]
Nick Dimiduk commented on HBASE-11331:
--
Thanks [~enis], [~stack] for having a look.
bq. Add to the simplification of block cache config a note that we need to
unify the configuration args?
Alright, I'll open a ticket.
bq. Needs fat release note and on commit, add something to the refguide in the
block cache section else I'm afraid folks won't come across this feature.
Release note I'd planned, just what's in the commit message; you want more than
this?
{noformat}
When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are
cached in the BlockCache in their on-disk format. This is different from the
default behavior, which decompresses and decrypts a block before caching.
{noformat}
I'll open a docs ticket for updating the book.
bq. Is this right? Assuming HeapByteBuffer here.
Alas, yes. There's a number of assumptions baked into HFileBlock about
HeapByteArrays. The whole read-ahead of the next block's header feature is
based on this supposition (look for reads beyond limit without checking
capacity; this is how I ran into the serialization bug in the first place).
[blockcache] lazy block decompression
-
Key: HBASE-11331
URL: https://issues.apache.org/jira/browse/HBASE-11331
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 0.98.7, 0.99.1
Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch,
HBASE-11331.05-0.98.patch, HBASE-11331.05.patch, HBASE-11331.06-0.98.patch,
HBASE-11331.06.patch, HBASE-11331.07-0.98.patch, HBASE-11331.07.patch,
HBASE-11331LazyBlockDecompressperfcompare.pdf,
hbase-hbase-master-hor17n36.gq1.ygridcore.net.log, lazy-decompress.02.0.pdf,
lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf,
v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf
Maintaining data in its compressed form in the block cache will greatly
increase our effective blockcache size and should show a meaning improvement
in cache hit rates in well designed applications. The idea here is to lazily
decompress/decrypt blocks when they're consumed, rather than as soon as
they're pulled off of disk.
This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11937) Update patch submission guidelines to stop using --no-prefix
[
https://issues.apache.org/jira/browse/HBASE-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129251#comment-14129251
]
Sean Busbey commented on HBASE-11937:
-
A committer should first see if the patch applies cleanly to master. This works
the same regardless of which git command created the patch.
{noformat}
$ git fetch origin
$ git checkout master
$ git rebase origin/master
$ git checkout example-checking-HBASE-11891
$ git apply --check /some/path/to/patches/HBASE-11891.patch
{noformat}
No output means things are fine. Failures look like this:
{noformat}
$ git apply --check /some/path/to/patches/HBASE-11891.old.patch
error: patch failed:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterStatusListener.java:34
error:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterStatusListener.java:
patch does not apply
error: patch failed:
hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseInterfaceAudience.java:29
error:
hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseInterfaceAudience.java:
patch does not apply
error: patch failed:
hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodec.java:25
error: hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodec.java:
patch does not apply
error: patch failed:
hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodecWithTags.java:25
error:
hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodecWithTags.java:
patch does not apply
{noformat}
If the patch doesn't apply, the contributor should updated it for the current
master.
A committer can tell if the patch was made with git format-patch (rather than
git diff) if it begins with a header that looks like an email message:
{noformat}
From b7893a4e07cbff333d48065dad9810f9f03cf159 Mon Sep 17 00:00:00 2001
From: Sean Busbey bus...@apache.org
Date: Wed, 3 Sep 2014 23:23:16 -0500
Subject: [PATCH] HBASE-11891 Introduce an HBaseInterfaceAudience level to
denote class names that appear in configs.
---
{noformat}
If the patch was made with format patch, then they should use {{git am}} to
apply.
{noformat}
$ git am /some/path/to/format-patch/patches/HBASE-11891.patch
Applying: HBASE-11891 Introduce an HBaseInterfaceAudience level to denote class
names that appear in configs.
{noformat}
This will place a commit in the current local branch, which you can interact
with normally via git commands.
If the patch was mad with git diff, then they should use {{git apply}} to apply.
{noformat}
$ git apply /some/path/to/diff/patches/HBASE-11891.patch
{noformat}
No output means things worked fine. This approach won't add a commit; instead
all changes will be on the current working directory.
{noformat}
$ git status
# On branch example-checking-HBASE-11891
# Changes not staged for commit:
# (use git add file... to update what will be committed)
# (use git checkout -- file... to discard changes in working directory)
#
# modified:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterStatusListener.java
# modified:
hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseInterfaceAudience.java
# modified:
hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodec.java
{noformat}
For patches created with {{git diff}} a committer can also use the {{patch}}
command to apply changes locally.
{noformat}
$ patch -p1 /some/path/to/diff/patches/HBASE-11891.patch
patching file
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterStatusListener.java
patching file
hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseInterfaceAudience.java
patching file
hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodec.java
patching file
hbase-common/src/main/java/org/apache/hadoop/hbase/codec/CellCodecWithTags.java
...
{noformat}
This should give you a working directory that is the same as the one created by
{{git apply}}.
At this point you can interact with the local changes as you would with local
modifications you made.
Update patch submission guidelines to stop using --no-prefix
Key: HBASE-11937
URL: https://issues.apache.org/jira/browse/HBASE-11937
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Sean Busbey
Priority: Minor
Right now the submission guidelines include the use of --no-prefix in the
Methods to Create Patches section:
{quote}
Git
git format-patch is preferred because it preserves commit messages. Use git
squash first, to combine smaller commits into a single larger one.
* git format-patch --no-prefix origin/master --stdout HBASE-.patch
* git diff --no-prefix origin/master HBASE-.patch
{quote}
The use of --no-prefix means that users of {{git
[jira] [Commented] (HBASE-11935) Unbounded creation of Replication Failover workers
[
https://issues.apache.org/jira/browse/HBASE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129254#comment-14129254
]
Andrew Purtell commented on HBASE-11935:
TestReplicationThrottler passed 10 out of 10 times locally for me, e.g.
{noformat}
Running
org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.537 sec - in
org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
{noformat}
Unbounded creation of Replication Failover workers
--
Key: HBASE-11935
URL: https://issues.apache.org/jira/browse/HBASE-11935
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0, 0.94.23, 0.98.6
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Priority: Critical
Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1
Attachments: hbase-11935-0.98-v0.patch, hbase-11935-0.98-v1.patch,
hbase-11935-trunk-v0.patch, hbase-11935-trunk-v1.patch,
hbase-11935-trunk-v2.patch
We just ran into a production incident with TCP SYN storms on port 2181
(zookeeper).
In our case the slave cluster was not running. When we bounced the primary
cluster we saw an unbounded number of failover threads all hammering the
hosts on the slave ZK machines (which did not run ZK at the time)... Causing
overall degradation of network performance between datacenters.
Looking at the code we noticed that the thread pool handling of the Failover
workers was probably unintended.
Patch coming soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129270#comment-14129270 ] Andrew Purtell commented on HBASE-11639: bq. Adding new hooks to the RegionServerObserver will cause BC issues in 0.98 as AC is implementing RegionServerObserver and not extending the BaseRegionServerObserver. Just update the AC at the same time the new hooks go in? [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 2.0.0, 0.98.7, 0.99.1 This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11847) HFile tool should be able to print block headers
[
https://issues.apache.org/jira/browse/HBASE-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129275#comment-14129275
]
Andrew Purtell commented on HBASE-11847:
{code}
+ * TODO: this same/similar block iteration logic is used in
HFileBlock#blockRange and
+ * TestLazyDataBlockDecompression. Refactor?
{code}
Looks familiar. I think something similar is done in HFileV2 prefetch? If you
like. Or could be a follow up task.
+1
HFile tool should be able to print block headers
Key: HBASE-11847
URL: https://issues.apache.org/jira/browse/HBASE-11847
Project: HBase
Issue Type: Improvement
Components: HFile
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Fix For: 2.0.0, 0.98.7, 0.99.1
Attachments: HBASE-11847.00.patch, HBASE-11847.01.patch
Printing the block index is helpful, but sometimes you want to see more info
about the blocks themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11938) Unify cache configuration
Nick Dimiduk created HBASE-11938:
Summary: Unify cache configuration
Key: HBASE-11938
URL: https://issues.apache.org/jira/browse/HBASE-11938
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Nick Dimiduk
Priority: Minor
As pointed out in a
[comment|https://issues.apache.org/jira/browse/HBASE-11331?focusedCommentId=14128811page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14128811]
on HBASE-11331, our configuration keys for the blockcache are all over the
place.
{quote}
There's not a lot of consistency around cache configurations. We also have:
- hbase.rs.cacheblocksonwrite
- hfile.block.index.cacheonwrite
- hfile.block.bloom.cacheonwrite
- hbase.rs.evictblocksonclose
- hbase.bucketcache.*
- hbase.rs.prefetchblocksonopen
- hbase.offheapcache.minblocksize
{quote}
We should rename them to a consistent scheme.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-9719) Premptive Call Me Maybe HBase
[ https://issues.apache.org/jira/browse/HBASE-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129277#comment-14129277 ] Robert Yokota commented on HBASE-9719: -- Here is an addendum to my blog post: http://eng.yammer.com/call-me-maybe-hbase-addendum/ Premptive Call Me Maybe HBase - Key: HBASE-9719 URL: https://issues.apache.org/jira/browse/HBASE-9719 Project: HBase Issue Type: Bug Reporter: stack Assignee: Robert Yokota Fix For: 0.98.1 Aphyr wrote an interesting article on C* [1]. Some awkward-looking issues were turned up though it seems the author is purportedly doing nothing but exercising the software within spec; he is just paying close attention to what is being returned. It does not look like Aphyr will be coming our way any time soon [2] -- thanks Ian Varley -- but he could change his mind. Wouldn't it be coolio if we'd already run his test suite and found any bugs and fixed them before he came by? This issue is about running his article against hbase so we find the embarrassing before he does. 1. http://aphyr.com/posts/294-call-me-maybe-cassandra 2. https://twitter.com/aphyr/status/335082835868254209 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
