[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[
https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048555#comment-14048555
]
Hadoop QA commented on HBASE-11316:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653316/HBASE-11316-5.patch
against trunk revision .
ATTACHMENT ID: 12653316
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ can try raising this value. Default: the value of LONG.MAX_VALUE,
expressed in bytes./description
+drop (filter out) deletes or expired versions, because of
potential side effects. See xref
+
filenamehbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java/filename.
+ will compact larger StoreFiles. However, during
reads, HBase will need to seek
+ through fewer StpreFo;es to accomplish the read.
Consider this approach if you
+
xlink:href=http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836;Store
+userinputalter 'orders_table', CONFIGURATION = {'hbase.hstore.engine.class'
= 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine',
'hbase.hstore.blockingStoreFiles' = '100'}/userinput
+userinputalter 'orders_table', {NAME = 'blobs_cf', CONFIGURATION =
{'hbase.hstore.engine.class' =
'org.apache.hadoop.hbase.regionserver.StripeStoreEngine',
'hbase.hstore.blockingStoreFiles' = '100'}}/userinput
+userinputcreate 'orders_table', 'blobs_cf', CONFIGURATION =
{'hbase.hstore.engine.class' =
'org.apache.hadoop.hbase.regionserver.StripeStoreEngine',
'hbase.hstore.blockingStoreFiles' = '100'}/userinput
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy
org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9912//console
This message is automatically generated.
Expand info about compactions beyond HBASE-11120
Key: HBASE-11316
URL: https://issues.apache.org/jira/browse/HBASE-11316
Project: HBase
Issue Type: Bug
Components: Compaction, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Attachments: HBASE-11316-1.patch,
[jira] [Commented] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[
https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048559#comment-14048559
]
Anoop Sam John commented on HBASE-11434:
{code}
+// set up the list of users with superuser privilege
+superusers = Lists.asList(userProvider.getCurrentUserName(),
+ conf.getStrings(AccessControlLists.SUPERUSER_CONF_KEY, new String[0]));
{code}
And UserProvider#getCurrentUserName() returns user.getName().
We should use user.getShortName() (?)
And previously this List of users were creating by
{code}
-String currentUser = user.getShortName();
-ListString superusers = Lists.asList(currentUser, conf.getStrings(
- AccessControlLists.SUPERUSER_CONF_KEY, new String[0]));
{code}
So we used getShortName().
[AccessController] Disallow inbound cells with reserved tags
Key: HBASE-11434
URL: https://issues.apache.org/jira/browse/HBASE-11434
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.99.0, 0.98.4
Attachments: HBASE-11434.patch, HBASE-11434.patch
The AccessController allows users to store cells with ACL tags encoded by the
client. This isn't a security issue currently, because in order to store the
cell the user must have a relevant WRITE grant, and the user is allowed to
specify whatever ACL for the cell they'd like. However it could become a
correctness problem in the future, if we introduce format sanity checking or
the like, so let's disallow inbound mutations containing cells with reserved
tags like the VisibilityController does.
The check is skipped if the active user is a superuser. First, superusers are
allowed to do anything. Second, replication (as superuser) must be able to
store incoming cells with ACL tags.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11448) Fix javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048560#comment-14048560 ] Hudson commented on HBASE-11448: SUCCESS: Integrated in HBase-TRUNK #5253 (See [https://builds.apache.org/job/HBase-TRUNK/5253/]) HBASE-11448 Fix javadoc warnings (enis: rev 0167558eb31ff48308d592ef70b6d005ba6d21fb) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RackManager.java Fix javadoc warnings Key: HBASE-11448 URL: https://issues.apache.org/jira/browse/HBASE-11448 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Trivial Fix For: 0.99.0 Attachments: hbase-11448_v1.patch Master has a couple of new javadoc warnings because of the hbase-10070. We should fix them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11422) Specification of scope is missing for certain Hadoop dependencies
[
https://issues.apache.org/jira/browse/HBASE-11422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048564#comment-14048564
]
stack commented on HBASE-11422:
---
Did the first patch have classifiers in it? Classifying hasn't worked
consistently across mvn phases in my experience.
Patch looks good. If I were testing, I'd be looking at the tgz assembly making
sure it had hadoop jars all of same version (I'm pretty sure I've looked at
this in the past and IIRC it used to work properly only putting hadoop versions
of a particular type into assembly... but hey... stuff changes). Would also
try publishing a snapshot to make sure all went up to the repo w/ correct
references. I can try it if you just say the word [~cos], np.
On hbase-testing-util, let me look over at the other issue you raise. I'm not
sure I grok its need myself (the original hbase-downstreamer seems to be able
to use testing utils w/o need of hbase-testing-util but I must be missing
something...)
Specification of scope is missing for certain Hadoop dependencies
-
Key: HBASE-11422
URL: https://issues.apache.org/jira/browse/HBASE-11422
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Konstantin Boudnik
Fix For: 0.99.0, 0.98.4
Attachments: HBASE-11422.0.98.patch
[~cos] reported in the mailing thread, 'HBase 0.98.x dependency problems',
that specifying hadoop-two.version with value other than 2.2.0 pulls in
incorrect dependencies such as:
{code}
[INFO] | \- org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.2.0:compile
{code}
This is due to missing specification of scope in the pom.xml files.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11344) Hide row keys and such from the web UIs
[ https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-11344: Assignee: Devaraj Das Status: Patch Available (was: Open) Hide row keys and such from the web UIs --- Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0 Attachments: 11344-1.txt The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11344) Hide row keys and such from the web UIs
[ https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-11344: Attachment: 11344-1.txt Attaching a patch. Adds a class that replaces the real start/end keys with hidden strings when the webUI is about to display the respective details. Hide row keys and such from the web UIs --- Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Fix For: 0.99.0 Attachments: 11344-1.txt The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048600#comment-14048600
]
Hadoop QA commented on HBASE-4495:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653320/HBASE-4495-v5.patch
against trunk revision .
ATTACHMENT ID: 12653320
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 132
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+#org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan,org.apache.hadoop.hbase.catalog.TestMetaTableAccessorNoCluster,org.apache.hadoop.hbase.catalog.TestMetaTableAccessor,org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat,org.apache.hadoop.hbase.mapred.TestTableMapReduce,org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort,org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove,org.apache.hadoop.hbase.client.TestAdmin,org.apache.hadoop.hbase.master.TestMasterFailover,org.apache.hadoop.hbase.regionserver.wal.TestLogRolling,org.apache.hadoop.hbase.master.TestDistributedLogSplitting,org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable,org.apache.hadoop.hbase.TestGlobalMemStoreSize,
+ public static void removeRegionReplicasFromMeta(Setbyte[] metaRows, int
replicaIndexToDeleteFrom,
+PairHRegionInfo, ServerName pair =
MetaTableAccessor.getRegion(connection, tableNameOrRegionName);
+ sb.append(\nRegion server holding hbase:meta: + new
MetaTableLocator().getMetaRegionLocation(zkw));
+ public DisableTableHandler(Server server, TableName tableName,
AssignmentManager assignmentManager,
+MetaTableAccessor.mergeRegions(server.getShortCircuitConnection(),
mergedRegion.getRegionInfo(), region_a
+mergeRegionsAndPutMetaEntries(server.getShortCircuitConnection(),
mergedRegion.getRegionInfo(),
+ return
HConnectionTestingUtility.getMockedConnectionAndDecorate(TESTUTIL.getConfiguration(),
+ return
HConnectionTestingUtility.getMockedConnectionAndDecorate(TESTUTIL.getConfiguration(),
+deleteOneReplicaLocation.deleteColumns(HConstants.CATALOG_FAMILY,
MetaTableAccessor.getServerColumn(1));
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9914//console
This message is automatically generated.
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL:
[jira] [Commented] (HBASE-11435) Visibility labelled cells fail to getting replicated
[
https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048604#comment-14048604
]
Hadoop QA commented on HBASE-11435:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653318/HBASE-11435.patch
against trunk revision .
ATTACHMENT ID: 12653318
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication.testBlockReceived(TestPendingReplication.java:243)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3289)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9913//console
This message is automatically generated.
Visibility labelled cells fail to getting replicated
Key: HBASE-11435
URL: https://issues.apache.org/jira/browse/HBASE-11435
Project: HBase
Issue Type: Bug
Components: Replication, security
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
Fix For: 0.99.0, 0.98.4
Attachments: HBASE-11435.patch
Thanks [~apurtell] for finding out the issue.
We have the reserved tags check in the preBatchMutation hook in VC. In case
of the replication the cells are coming into the sink cluster with visibility
tags with it. Those are created by VC in the source cluster and should get
added into sink cluster. But the current check will throw exception and we
will fail to replicate these cells.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Status: Open (was: Patch Available) Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11355) a couple of callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-11355: Resolution: Fixed Fix Version/s: 0.98.4 0.99.0 Status: Resolved (was: Patch Available) a couple of callQueue related improvements -- Key: HBASE-11355 URL: https://issues.apache.org/jira/browse/HBASE-11355 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Affects Versions: 0.99.0, 0.94.20 Reporter: Liang Xie Assignee: Matteo Bertozzi Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11355-v0.patch, HBASE-11355-v1.patch, HBASE-11355-v2.patch, gets.png In one of my in-memory read only testing(100% get requests), one of the top scalibility bottleneck came from the single callQueue. A tentative sharing this callQueue according to the rpc handler number showed a big throughput improvement(the original get() qps is around 60k, after this one and other hotspot tunning, i got 220k get() qps in the same single region server) in a YCSB read only scenario. Another stuff we can do is seperating the queue into read call queue and write call queue, we had done it in our internal branch, it would helpful in some outages, to avoid all read or all write requests ran out of all handler threads. One more stuff is changing the current blocking behevior once the callQueue is full, considering the full callQueue almost means the backend processing is slow somehow, so a fail-fast here should be more reasonable if we using HBase as a low latency processing system. see callQueue.put(call) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11440:
---
Attachment: HBASE-11440_1.patch
Updated patch with more descriptions.
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11440:
---
Status: Patch Available (was: Open)
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11440:
---
Status: Open (was: Patch Available)
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Attachment: HBASE-4495-v6.patch
v6 patch for QA run.
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Status: Patch Available (was: Open)
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Status: Open (was: Patch Available)
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11449) IntegrationTestIngestWithACL fails to use different users after HBASE-10810
ramkrishna.s.vasudevan created HBASE-11449: -- Summary: IntegrationTestIngestWithACL fails to use different users after HBASE-10810 Key: HBASE-11449 URL: https://issues.apache.org/jira/browse/HBASE-11449 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Im MultiThreadedReaderWithACL we needed the Table instances per user and the same was getting cached. Now after HBASE-10810 the table instance is obtained from the connection maintained which is the connection created by the user and not by the user intended. So LTT does not work in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11449) IntegrationTestIngestWithACL fails to use different users after HBASE-10810
[ https://issues.apache.org/jira/browse/HBASE-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11449: --- Description: Im MultiThreadedReaderWithACL we needed the Table instances per user and the same was getting cached. Now after HBASE-10810 the table instance is obtained from the connection maintained which is the connection created by the 'root' user and not by the user intended. So IT does not work in trunk. (was: Im MultiThreadedReaderWithACL we needed the Table instances per user and the same was getting cached. Now after HBASE-10810 the table instance is obtained from the connection maintained which is the connection created by the user and not by the user intended. So LTT does not work in trunk.) IntegrationTestIngestWithACL fails to use different users after HBASE-10810 --- Key: HBASE-11449 URL: https://issues.apache.org/jira/browse/HBASE-11449 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Im MultiThreadedReaderWithACL we needed the Table instances per user and the same was getting cached. Now after HBASE-10810 the table instance is obtained from the connection maintained which is the connection created by the 'root' user and not by the user intended. So IT does not work in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11344) Hide row keys and such from the web UIs
[
https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048674#comment-14048674
]
Hadoop QA commented on HBASE-11344:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653334/11344-1.txt
against trunk revision .
ATTACHMENT ID: 12653334
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 15 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ td%=
escapeXml(Bytes.toStringBinary(RegionDetailsForDisplay.getRegionNameForDisplay(regionInfo,
conf))) %/td
+ td%=
escapeXml(Bytes.toStringBinary(RegionDetailsForDisplay.getStartKey(regionInfo,
conf))) %/td
+ td%=
escapeXml(Bytes.toStringBinary(RegionDetailsForDisplay.getEndKey(regionInfo,
conf))) %/td
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9915//console
This message is automatically generated.
Hide row keys and such from the web UIs
---
Key: HBASE-11344
URL: https://issues.apache.org/jira/browse/HBASE-11344
Project: HBase
Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.99.0
Attachments: 11344-1.txt
The table details on the master UI lists the start row keys of the regions.
The row keys might have sensitive data. We should hide them based on whether
or not the user accessing has the required authorization to view the table..
To start with, we could make the display of row keys and such based on a
configuration being true or false. If it is false, such potentially sensitive
data is never displayed on the web UI.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11449) IntegrationTestIngestWithACL fails to use different users after HBASE-10810
[
https://issues.apache.org/jira/browse/HBASE-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11449:
---
Attachment: HBASE-11449.patch
This is for trunk. Now the IT passes and also when we run using
{code}
mvn verify -Dtest.output.tofile=false -Dit.test=IntegrationTestIngestWithACL
{code}
IntegrationTestIngestWithACL fails to use different users after HBASE-10810
---
Key: HBASE-11449
URL: https://issues.apache.org/jira/browse/HBASE-11449
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11449.patch
Im MultiThreadedReaderWithACL we needed the Table instances per user and the
same was getting cached. Now after HBASE-10810 the table instance is
obtained from the connection maintained which is the connection created by
the 'root' user and not by the user intended. So IT does not work in trunk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11449) IntegrationTestIngestWithACL fails to use different users after HBASE-10810
[ https://issues.apache.org/jira/browse/HBASE-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11449: --- Status: Patch Available (was: Open) IntegrationTestIngestWithACL fails to use different users after HBASE-10810 --- Key: HBASE-11449 URL: https://issues.apache.org/jira/browse/HBASE-11449 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-11449.patch Im MultiThreadedReaderWithACL we needed the Table instances per user and the same was getting cached. Now after HBASE-10810 the table instance is obtained from the connection maintained which is the connection created by the 'root' user and not by the user intended. So IT does not work in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11294) IntegrationTestIngestWithACL should automatically set the superuser when running on local minicluster
[
https://issues.apache.org/jira/browse/HBASE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11294:
---
Attachment: HBASE-11294_2.patch
This is good to go for 0.98.
[~apurtell]
Can you check this once. If fine we can commit this to 0.98.4
IntegrationTestIngestWithACL should automatically set the superuser when
running on local minicluster
-
Key: HBASE-11294
URL: https://issues.apache.org/jira/browse/HBASE-11294
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.5
Attachments: HBASE-11294_1.patch, HBASE-11294_2.patch
To reproduce:
{noformat}
$ mvn -DskipTests clean install
$ cd hbase-it
$ mvn verify -Dit.test=IntegrationTestIngestWithACL
{noformat}
This should execute successfully according to
http://hbase.apache.org/book/hbase.tests.html section 16.7.5.1.
Instead no tables can deploy because the superuser is not automatically set
to the running user, as what used to happen once upon a time:
{noformat}
2014-06-03 20:15:10,067 WARN [htable-pool12-t1] client.AsyncProcess(675):
#7, table=hbase:meta, attempt=1/350 failed 1 ops, last exception:
org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (user=apurtell, scope=hbase:meta, family=info:regioninfo,
action=WRITE)
at
org.apache.hadoop.hbase.security.access.AccessController.prePut(AccessController.java:1447)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:1122)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2269)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2244)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2200)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2204)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4263)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3479)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3369)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
at java.lang.Thread.run(Thread.java:745)
on localhost,59092,1401826507071, tracking started Tue Jun 03 20:15:10 UTC
2014 - FAILED, NOT RETRYING ANYMORE
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Status: Patch Available (was: Open) Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v15.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Attachment: HBASE-10885_v15.patch Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: 10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt, HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch, HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch, HBASE-10885_v15.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch, HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch, HBASE-10885_v9.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11449) IntegrationTestIngestWithACL fails to use different users after HBASE-10810
[
https://issues.apache.org/jira/browse/HBASE-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048728#comment-14048728
]
Hadoop QA commented on HBASE-11449:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653359/HBASE-11449.patch
against trunk revision .
ATTACHMENT ID: 12653359
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9918//console
This message is automatically generated.
IntegrationTestIngestWithACL fails to use different users after HBASE-10810
---
Key: HBASE-11449
URL: https://issues.apache.org/jira/browse/HBASE-11449
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11449.patch
Im MultiThreadedReaderWithACL we needed the Table instances per user and the
same was getting cached. Now after HBASE-10810 the table instance is
obtained from the connection maintained which is the connection created by
the 'root' user and not by the user intended. So IT does not work in trunk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11294) IntegrationTestIngestWithACL should automatically set the superuser when running on local minicluster
[
https://issues.apache.org/jira/browse/HBASE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048733#comment-14048733
]
Hadoop QA commented on HBASE-11294:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653364/HBASE-11294_2.patch
against trunk revision .
ATTACHMENT ID: 12653364
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9919//console
This message is automatically generated.
IntegrationTestIngestWithACL should automatically set the superuser when
running on local minicluster
-
Key: HBASE-11294
URL: https://issues.apache.org/jira/browse/HBASE-11294
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.5
Attachments: HBASE-11294_1.patch, HBASE-11294_2.patch
To reproduce:
{noformat}
$ mvn -DskipTests clean install
$ cd hbase-it
$ mvn verify -Dit.test=IntegrationTestIngestWithACL
{noformat}
This should execute successfully according to
http://hbase.apache.org/book/hbase.tests.html section 16.7.5.1.
Instead no tables can deploy because the superuser is not automatically set
to the running user, as what used to happen once upon a time:
{noformat}
2014-06-03 20:15:10,067 WARN [htable-pool12-t1] client.AsyncProcess(675):
#7, table=hbase:meta, attempt=1/350 failed 1 ops, last exception:
org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (user=apurtell, scope=hbase:meta, family=info:regioninfo,
action=WRITE)
at
org.apache.hadoop.hbase.security.access.AccessController.prePut(AccessController.java:1447)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:1122)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2269)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2244)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2200)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2204)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4263)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3479)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3369)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
at java.lang.Thread.run(Thread.java:745)
on localhost,59092,1401826507071, tracking started Tue Jun 03 20:15:10 UTC
2014 - FAILED, NOT RETRYING ANYMORE
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048743#comment-14048743
]
Hadoop QA commented on HBASE-11440:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653350/HBASE-11440_1.patch
against trunk revision .
ATTACHMENT ID: 12653350
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.client.TestReplicaWithCluster.testChangeTable(TestReplicaWithCluster.java:217)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9916//console
This message is automatically generated.
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11294) IntegrationTestIngestWithACL should automatically set the superuser when running on local minicluster
[
https://issues.apache.org/jira/browse/HBASE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048744#comment-14048744
]
ramkrishna.s.vasudevan commented on HBASE-11294:
This is a 0.98 patch.
IntegrationTestIngestWithACL should automatically set the superuser when
running on local minicluster
-
Key: HBASE-11294
URL: https://issues.apache.org/jira/browse/HBASE-11294
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.5
Attachments: HBASE-11294_1.patch, HBASE-11294_2.patch
To reproduce:
{noformat}
$ mvn -DskipTests clean install
$ cd hbase-it
$ mvn verify -Dit.test=IntegrationTestIngestWithACL
{noformat}
This should execute successfully according to
http://hbase.apache.org/book/hbase.tests.html section 16.7.5.1.
Instead no tables can deploy because the superuser is not automatically set
to the running user, as what used to happen once upon a time:
{noformat}
2014-06-03 20:15:10,067 WARN [htable-pool12-t1] client.AsyncProcess(675):
#7, table=hbase:meta, attempt=1/350 failed 1 ops, last exception:
org.apache.hadoop.hbase.security.AccessDeniedException:
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions (user=apurtell, scope=hbase:meta, family=info:regioninfo,
action=WRITE)
at
org.apache.hadoop.hbase.security.access.AccessController.prePut(AccessController.java:1447)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:1122)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2269)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2244)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2200)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2204)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4263)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3479)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3369)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
at java.lang.Thread.run(Thread.java:745)
on localhost,59092,1401826507071, tracking started Tue Jun 03 20:15:10 UTC
2014 - FAILED, NOT RETRYING ANYMORE
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11449) IntegrationTestIngestWithACL fails to use different users after HBASE-10810
[ https://issues.apache.org/jira/browse/HBASE-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048746#comment-14048746 ] ramkrishna.s.vasudevan commented on HBASE-11449: Tested the patch in a cluster. IntegrationTestIngestWithACL fails to use different users after HBASE-10810 --- Key: HBASE-11449 URL: https://issues.apache.org/jira/browse/HBASE-11449 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-11449.patch Im MultiThreadedReaderWithACL we needed the Table instances per user and the same was getting cached. Now after HBASE-10810 the table instance is obtained from the connection maintained which is the connection created by the 'root' user and not by the user intended. So IT does not work in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11447) Proposal for a generic transaction API for HBase
[ https://issues.apache.org/jira/browse/HBASE-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048749#comment-14048749 ] Lars Hofhansl commented on HBASE-11447: --- Cool. * I would like to discuss associating a TransactionHTable object with a Transaction upon construction and then use the normal put/delete/etc methods. HTables are lightweight now and hence can be created for each transaction. That way the API is the same (besides construction) and we won't have two implementations of each operation. The Transaction could even have a getter for HTableInterface instances. * PREPARED/PREPARING implies 2PC transactions only? * For optimistic concurrency control (OCC) and snapshot isolation (SI) a transaction would need to include: the write TS, the read TS, potentially uncommitted transactions newer than the read point, etc. I.e. Transaction maybe should just be a marker interface to be passed around, but details are left to the implementation. * We should allow distinguishing between read only and read write transactions. (important for OCC) * We should also be able to indicate isolation level. Full serializable transactions are very expensive in a distributed environment when many rows are read/updated. Proposal for a generic transaction API for HBase Key: HBASE-11447 URL: https://issues.apache.org/jira/browse/HBASE-11447 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 1.0.0 Environment: Any. Reporter: John de Roo Priority: Minor Labels: features, newbie Fix For: 1.0.0 Attachments: Open Transaction Interface.pdf HBase transaction management today is provided by a number of products, each implementing a different API, each having different strengths. The lack of a common API for transactional interfaces means that applications need to be coded to work with a specific Transaction Manager. This proposal outlines an API which, if implemented by the different Transaction Manager vendors would provide stability and choice to HBase application developers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11355) a couple of callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048771#comment-14048771 ] Hudson commented on HBASE-11355: SUCCESS: Integrated in HBase-TRUNK #5254 (See [https://builds.apache.org/job/HBase-TRUNK/5254/]) HBASE-11355 a couple of callQueue related improvements (matteo.bertozzi: rev 0e8e41d0ef5985aa3d424621eb8fcb1ca68838ab) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MultipleQueueRpcExecutor.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ReflectionUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SingleQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-common/src/main/resources/hbase-default.xml a couple of callQueue related improvements -- Key: HBASE-11355 URL: https://issues.apache.org/jira/browse/HBASE-11355 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Affects Versions: 0.99.0, 0.94.20 Reporter: Liang Xie Assignee: Matteo Bertozzi Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11355-v0.patch, HBASE-11355-v1.patch, HBASE-11355-v2.patch, gets.png In one of my in-memory read only testing(100% get requests), one of the top scalibility bottleneck came from the single callQueue. A tentative sharing this callQueue according to the rpc handler number showed a big throughput improvement(the original get() qps is around 60k, after this one and other hotspot tunning, i got 220k get() qps in the same single region server) in a YCSB read only scenario. Another stuff we can do is seperating the queue into read call queue and write call queue, we had done it in our internal branch, it would helpful in some outages, to avoid all read or all write requests ran out of all handler threads. One more stuff is changing the current blocking behevior once the callQueue is full, considering the full callQueue almost means the backend processing is slow somehow, so a fail-fast here should be more reasonable if we using HBase as a low latency processing system. see callQueue.put(call) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-11450:
Attachment: HBASE-11450-v0.patch
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-11450-v0.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11450) Improve file size info in SnapshotInfo tool
Matteo Bertozzi created HBASE-11450:
---
Summary: Improve file size info in SnapshotInfo tool
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-11450-v0.patch
Add a -size-in-bytes flag to print the file size in byte instead of the human
readable format.
and add a check on the file length between the manifest and the hfile, marking
as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-11450:
Status: Patch Available (was: Open)
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-11450-v0.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11355) a couple of callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048782#comment-14048782 ] Hudson commented on HBASE-11355: SUCCESS: Integrated in HBase-1.0 #2 (See [https://builds.apache.org/job/HBase-1.0/2/]) HBASE-11355 a couple of callQueue related improvements (matteo.bertozzi: rev 9a6a59c7b7d8357c50fd32f01d0ca21911db3da2) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ReflectionUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SingleQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MultipleQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java a couple of callQueue related improvements -- Key: HBASE-11355 URL: https://issues.apache.org/jira/browse/HBASE-11355 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Affects Versions: 0.99.0, 0.94.20 Reporter: Liang Xie Assignee: Matteo Bertozzi Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11355-v0.patch, HBASE-11355-v1.patch, HBASE-11355-v2.patch, gets.png In one of my in-memory read only testing(100% get requests), one of the top scalibility bottleneck came from the single callQueue. A tentative sharing this callQueue according to the rpc handler number showed a big throughput improvement(the original get() qps is around 60k, after this one and other hotspot tunning, i got 220k get() qps in the same single region server) in a YCSB read only scenario. Another stuff we can do is seperating the queue into read call queue and write call queue, we had done it in our internal branch, it would helpful in some outages, to avoid all read or all write requests ran out of all handler threads. One more stuff is changing the current blocking behevior once the callQueue is full, considering the full callQueue almost means the backend processing is slow somehow, so a fail-fast here should be more reasonable if we using HBase as a low latency processing system. see callQueue.put(call) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11355) a couple of callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048787#comment-14048787 ] Hudson commented on HBASE-11355: SUCCESS: Integrated in HBase-0.98 #368 (See [https://builds.apache.org/job/HBase-0.98/368/]) HBASE-11355 a couple of callQueue related improvements (matteo.bertozzi: rev f8d50681f628bac50a0082c4c7db52cc0f96ae6a) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SingleQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ReflectionUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MultipleQueueRpcExecutor.java * hbase-common/src/main/resources/hbase-default.xml a couple of callQueue related improvements -- Key: HBASE-11355 URL: https://issues.apache.org/jira/browse/HBASE-11355 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Affects Versions: 0.99.0, 0.94.20 Reporter: Liang Xie Assignee: Matteo Bertozzi Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11355-v0.patch, HBASE-11355-v1.patch, HBASE-11355-v2.patch, gets.png In one of my in-memory read only testing(100% get requests), one of the top scalibility bottleneck came from the single callQueue. A tentative sharing this callQueue according to the rpc handler number showed a big throughput improvement(the original get() qps is around 60k, after this one and other hotspot tunning, i got 220k get() qps in the same single region server) in a YCSB read only scenario. Another stuff we can do is seperating the queue into read call queue and write call queue, we had done it in our internal branch, it would helpful in some outages, to avoid all read or all write requests ran out of all handler threads. One more stuff is changing the current blocking behevior once the callQueue is full, considering the full callQueue almost means the backend processing is slow somehow, so a fail-fast here should be more reasonable if we using HBase as a low latency processing system. see callQueue.put(call) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048789#comment-14048789
]
Hadoop QA commented on HBASE-4495:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653355/HBASE-4495-v6.patch
against trunk revision .
ATTACHMENT ID: 12653355
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 145
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+#org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan,org.apache.hadoop.hbase.catalog.TestMetaTableAccessorNoCluster,org.apache.hadoop.hbase.catalog.TestMetaTableAccessor,org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat,org.apache.hadoop.hbase.mapred.TestTableMapReduce,org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort,org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove,org.apache.hadoop.hbase.client.TestAdmin,org.apache.hadoop.hbase.master.TestMasterFailover,org.apache.hadoop.hbase.regionserver.wal.TestLogRolling,org.apache.hadoop.hbase.master.TestDistributedLogSplitting,org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable,org.apache.hadoop.hbase.TestGlobalMemStoreSize,
+ * Callers should call close on the returned {@link
org.apache.hadoop.hbase.client.HTable} instance.
+ public static Result getMetaTableRowResultAsSplitRegion(final HRegionInfo
hri, final ServerName sn)
+//windows fix: tgz file has hbase:meta directory renamed as -META- since
the original is an illegal
+//name under windows. So we rename it back. See
src/test/data//TestMetaMigrationConvertingToPB.README and
+ * Does {@link
org.apache.hadoop.hbase.MetaTableAccessor#getRegion(org.apache.hadoop.hbase.client.HConnection,
+ Put putA =
MetaTableAccessor.makePutFromRegionInfo(daughterRegions.getFirst().getRegionInfo());
+ Put putB =
MetaTableAccessor.makePutFromRegionInfo(daughterRegions.getSecond().getRegionInfo());
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat.testScan(TestMultiTableInputFormat.java:244)
at
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat.testScanOBBToOPP(TestMultiTableInputFormat.java:189)
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testDurability(TestImportExport.java:562)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9917//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[
https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048803#comment-14048803
]
Hadoop QA commented on HBASE-10885:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653366/HBASE-10885_v15.patch
against trunk revision .
ATTACHMENT ID: 12653366
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testSecondaryFailsWithErrorBeforeSettingHeaders(TestCheckpoint.java:570)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:3500)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9920//console
This message is automatically generated.
Support visibility expressions on Deletes
-
Key: HBASE-10885
URL: https://issues.apache.org/jira/browse/HBASE-10885
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.99.0, 0.98.4
Attachments:
10885-org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes-output.txt,
HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch,
HBASE-10885_new_tag_type_2.patch, HBASE-10885_v1.patch,
HBASE-10885_v12.patch, HBASE-10885_v12.patch, HBASE-10885_v13.patch,
HBASE-10885_v15.patch, HBASE-10885_v2.patch, HBASE-10885_v2.patch,
HBASE-10885_v2.patch, HBASE-10885_v3.patch, HBASE-10885_v4.patch,
HBASE-10885_v5.patch, HBASE-10885_v7.patch, HBASE-10885_v8.patch,
HBASE-10885_v9.patch
Accumulo can specify visibility expressions for delete markers. During
compaction the cells covered by the tombstone are determined in part by
matching the visibility expression. This is useful for the use case of data
set coalescing, where entries from multiple data sets carrying different
labels are combined into one common large table. Later, a subset of entries
can be conveniently removed using visibility expressions.
Currently doing the same in HBase would only be possible with a custom
coprocessor. Otherwise, a Delete will affect all cells covered by the
tombstone regardless of any visibility expression scoping. This is
[jira] [Updated] (HBASE-11423) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11423: --- Attachment: HBASE-11423.patch Reattaching same patch for a QA run Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint -- Key: HBASE-11423 URL: https://issues.apache.org/jira/browse/HBASE-11423 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11423.patch This is because pre/postBatchMutate() APIs are not getting called from HRegion#processRowsWithLocks() -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11423) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11423: --- Status: Open (was: Patch Available) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint -- Key: HBASE-11423 URL: https://issues.apache.org/jira/browse/HBASE-11423 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11423.patch This is because pre/postBatchMutate() APIs are not getting called from HRegion#processRowsWithLocks() -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11423) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11423: --- Status: Patch Available (was: Open) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint -- Key: HBASE-11423 URL: https://issues.apache.org/jira/browse/HBASE-11423 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11423.patch This is because pre/postBatchMutate() APIs are not getting called from HRegion#processRowsWithLocks() -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048832#comment-14048832 ] Anoop Sam John commented on HBASE-11435: -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication.testBlockReceived(TestPendingReplication.java:243) Why it is running hdfs testcases ! Other test issue also seems not related to this patch and same with the findbug warns. Will commit shortly Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11423) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11423: --- Attachment: (was: HBASE-11423.patch) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint -- Key: HBASE-11423 URL: https://issues.apache.org/jira/browse/HBASE-11423 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11423.patch This is because pre/postBatchMutate() APIs are not getting called from HRegion#processRowsWithLocks() -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11435: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.98, branch-1 and master Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11435: --- Fix Version/s: 2.0.0 Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048861#comment-14048861
]
Hadoop QA commented on HBASE-11450:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653380/HBASE-11450-v0.patch
against trunk revision .
ATTACHMENT ID: 12653380
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testFlushCacheWhileScanning(TestHRegion.java:3162)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9921//console
This message is automatically generated.
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-11450-v0.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
[jira] [Commented] (HBASE-11355) a couple of callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048868#comment-14048868 ] Hudson commented on HBASE-11355: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #348 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/348/]) HBASE-11355 a couple of callQueue related improvements (matteo.bertozzi: rev f8d50681f628bac50a0082c4c7db52cc0f96ae6a) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MultipleQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SingleQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ReflectionUtils.java a couple of callQueue related improvements -- Key: HBASE-11355 URL: https://issues.apache.org/jira/browse/HBASE-11355 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Affects Versions: 0.99.0, 0.94.20 Reporter: Liang Xie Assignee: Matteo Bertozzi Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11355-v0.patch, HBASE-11355-v1.patch, HBASE-11355-v2.patch, gets.png In one of my in-memory read only testing(100% get requests), one of the top scalibility bottleneck came from the single callQueue. A tentative sharing this callQueue according to the rpc handler number showed a big throughput improvement(the original get() qps is around 60k, after this one and other hotspot tunning, i got 220k get() qps in the same single region server) in a YCSB read only scenario. Another stuff we can do is seperating the queue into read call queue and write call queue, we had done it in our internal branch, it would helpful in some outages, to avoid all read or all write requests ran out of all handler threads. One more stuff is changing the current blocking behevior once the callQueue is full, considering the full callQueue almost means the backend processing is slow somehow, so a fail-fast here should be more reasonable if we using HBase as a low latency processing system. see callQueue.put(call) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11423) Visibility label and per cell ACL feature not working with HTable#mutateRow() and MultiRowMutationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048948#comment-14048948
]
Hadoop QA commented on HBASE-11423:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653386/HBASE-11423.patch
against trunk revision .
ATTACHMENT ID: 12653386
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.client.TestReplicasClient.testSmallScanWithReplicas(TestReplicasClient.java:541)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9922//console
This message is automatically generated.
Visibility label and per cell ACL feature not working with HTable#mutateRow()
and MultiRowMutationEndpoint
--
Key: HBASE-11423
URL: https://issues.apache.org/jira/browse/HBASE-11423
Project: HBase
Issue Type: Bug
Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.99.0, 0.98.4
Attachments: HBASE-11423.patch
This is because pre/postBatchMutate() APIs are not getting called from
HRegion#processRowsWithLocks()
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048987#comment-14048987 ] Hudson commented on HBASE-11435: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #349 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/349/]) HBASE-11435 Visibility labelled cells fail to getting replicated. (Anoop) (anoopsamjohn: rev dde62f16d244a6d313232eb548fcd6e3c94a0a60) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049005#comment-14049005 ] Hudson commented on HBASE-11435: SUCCESS: Integrated in HBase-0.98 #369 (See [https://builds.apache.org/job/HBase-0.98/369/]) HBASE-11435 Visibility labelled cells fail to getting replicated. (Anoop) (anoopsamjohn: rev dde62f16d244a6d313232eb548fcd6e3c94a0a60) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049017#comment-14049017 ] Hudson commented on HBASE-11435: SUCCESS: Integrated in HBase-TRUNK #5255 (See [https://builds.apache.org/job/HBase-TRUNK/5255/]) HBASE-11435 Visibility labelled cells fail to getting replicated. (Anoop) (anoopsamjohn: rev ba044f010ee5a9f8f07986136317c77ce6ff9e80) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11439: --- Summary: StripeCompaction may not obey the OffPeak rule to compaction (was: StripeCompaction may not obey the OffPeak rule to compact.) StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11435) Visibility labelled cells fail to getting replicated
[ https://issues.apache.org/jira/browse/HBASE-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049025#comment-14049025 ] Hudson commented on HBASE-11435: FAILURE: Integrated in HBase-1.0 #3 (See [https://builds.apache.org/job/HBase-1.0/3/]) HBASE-11435 Visibility labelled cells fail to getting replicated. (Anoop) (anoopsamjohn: rev a02716fb45206f4d419acb46112446e4cf0db90e) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java Visibility labelled cells fail to getting replicated Key: HBASE-11435 URL: https://issues.apache.org/jira/browse/HBASE-11435 Project: HBase Issue Type: Bug Components: Replication, security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Critical Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11435.patch Thanks [~apurtell] for finding out the issue. We have the reserved tags check in the preBatchMutation hook in VC. In case of the replication the cells are coming into the sink cluster with visibility tags with it. Those are created by VC in the source cluster and should get added into sink cluster. But the current check will throw exception and we will fail to replicate these cells. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11439: --- Fix Version/s: 2.0.0 0.98.4 0.99.0 Hadoop Flags: Reviewed Integrated to 0.98, branch-1 and trunk. Thanks for the patch, Victor. StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11451) HBase MapReduce job loads configuration (hbase-site.xml), but actually doesn't use the values read.
Arun Allamsetty created HBASE-11451:
---
Summary: HBase MapReduce job loads configuration (hbase-site.xml),
but actually doesn't use the values read.
Key: HBASE-11451
URL: https://issues.apache.org/jira/browse/HBASE-11451
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.96.1.1
Environment: Linux Debain 7.4 Wheezy
Reporter: Arun Allamsetty
I am writing a MapReduce job which reads from (a) HBase table(s). Almost
everything works as it is supposed to except the {{Configuration}} class. So I
did this,
{code:java}
Configuration config = HBaseConfiguration.create();
GenericOptionsParser parser = new GenericOptionsParser(config, args);
// This should work but is not working.
config.addResource(new Path(parser.getCommandLine().getOptionValue(conf,
DEFAULT_HBASE_CONF)));
{code}
When I run the job like this (passing the path to `hbase-site.xml` correctly),
I get this error.
{code:java}
14/06/30 23:02:30 WARN zookeeper.ClientCnxn: Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
14/06/30 23:02:30 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
{code}
But when I add the following two lines, it works like a charm (even though it
seems completely ridiculous).
{code:java}
// So these are the workarounds.
config.set(hbase.rootdir, config.get(hbase.rootdir));
config.set(hbase.zookeeper.quorum, config.get(hbase.zookeeper.quorum));
{code}
Basically, read the parameters back from the {{Configuration}} object and set
them back in the same object, which is bonkers.
I read a bug raised about it
[HBASE-11066|https://issues.apache.org/jira/browse/HBASE-11066], but it seems
to have been closed citing local configuration problem (I think not). I use CDH
5.0.2 with HBase 0.96.1.1.
I don't know if it is an issue with the Cloudera build or the general one.
Please let me know if you need any more details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11439: --- Resolution: Fixed Status: Resolved (was: Patch Available) StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11451) HBase MapReduce job loads configuration (hbase-site.xml), but actually doesn't use the values read.
[
https://issues.apache.org/jira/browse/HBASE-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049059#comment-14049059
]
Ted Yu commented on HBASE-11451:
Seems to be related to HBASE-11378
HBase MapReduce job loads configuration (hbase-site.xml), but actually
doesn't use the values read.
---
Key: HBASE-11451
URL: https://issues.apache.org/jira/browse/HBASE-11451
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.96.1.1
Environment: Linux Debain 7.4 Wheezy
Reporter: Arun Allamsetty
I am writing a MapReduce job which reads from (a) HBase table(s). Almost
everything works as it is supposed to except the {{Configuration}} class. So
I did this,
{code:java}
Configuration config = HBaseConfiguration.create();
GenericOptionsParser parser = new GenericOptionsParser(config, args);
// This should work but is not working.
config.addResource(new Path(parser.getCommandLine().getOptionValue(conf,
DEFAULT_HBASE_CONF)));
{code}
When I run the job like this (passing the path to `hbase-site.xml`
correctly), I get this error.
{code:java}
14/06/30 23:02:30 WARN zookeeper.ClientCnxn: Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
14/06/30 23:02:30 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
{code}
But when I add the following two lines, it works like a charm (even though it
seems completely ridiculous).
{code:java}
// So these are the workarounds.
config.set(hbase.rootdir, config.get(hbase.rootdir));
config.set(hbase.zookeeper.quorum, config.get(hbase.zookeeper.quorum));
{code}
Basically, read the parameters back from the {{Configuration}} object and set
them back in the same object, which is bonkers.
I read a bug raised about it
[HBASE-11066|https://issues.apache.org/jira/browse/HBASE-11066], but it seems
to have been closed citing local configuration problem (I think not). I use
CDH 5.0.2 with HBase 0.96.1.1.
I don't know if it is an issue with the Cloudera build or the general one.
Please let me know if you need any more details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049087#comment-14049087
]
ramkrishna.s.vasudevan commented on HBASE-11440:
Should this be committed to branch-1 also?
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049089#comment-14049089
]
ramkrishna.s.vasudevan commented on HBASE-11440:
Pushed to master for now.
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Status: Patch Available (was: Open)
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Status: Open (was: Patch Available)
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Labels: 1.0.0 (was: )
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: 1.0.0
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049114#comment-14049114 ] stack commented on HBASE-11316: --- bq. Anyway I played around with this and tried to be consistent in using HFile instead of StoreFile, Column Family instead of Store. This is the way to go IMO. Looking at PDF, you have this in spite of the above: it flushes its contents to a StoreFile. The number of StoreFiles in a Store increases over time. What you thinking given the above? Expand info about compactions beyond HBASE-11120 Key: HBASE-11316 URL: https://issues.apache.org/jira/browse/HBASE-11316 Project: HBase Issue Type: Bug Components: Compaction, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, HBASE-11316.patch, ch9_compactions.pdf Round 2 - expand info about the algorithms, talk about stripe compaction, and talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Attachment: HBASE-4495-v7.patch
All tests which failed in v6 patch pass on my local. Reattaching patch, v7,
with fixed long lines.
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: 1.0.0
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495-v7.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch,
HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Status: Patch Available (was: Open)
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: 1.0.0
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495-v7.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch,
HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-4495:
---
Status: Open (was: Patch Available)
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: 1.0.0
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495-v7.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch,
HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11401) Issue with seqNo binding for KV mvcc
[
https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049126#comment-14049126
]
stack commented on HBASE-11401:
---
Changing order makes things way slower:
No patch going via region api instead of calling appendnosync on HLog directly:
{code}
nopatch3.3.1.txt:2014-07-01 10:26:26,700 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=3, iterations=100,
syncInterval=0 took 747.846s 4011.521ops/s
nopatch3.10.1.txt:2014-07-01 10:30:55,546 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=10, iterations=100,
syncInterval=0 took 260.533s 38382.855ops/s
nopatch3.25.1.txt:2014-07-01 10:37:23,266 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=25, iterations=100,
syncInterval=0 took 379.455s 65883.969ops/s
{code}
With patch applied
{code}
wpatch3.3.1.txt:2014-07-01 07:08:52,169 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=3, iterations=100,
syncInterval=0 took 1283.630s 2337.122ops/s
wpatch3.10.1.txt:2014-07-01 07:36:36,312 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=10, iterations=100,
syncInterval=0 took 1660.901s 6020.829ops/s
wpatch3.25.1.txt:2014-07-01 09:06:33,298 INFO [main]
wal.HLogPerformanceEvaluation: Summary: threads=25, iterations=100,
syncInterval=0 took 5393.763s 4634.983ops/s
{code}
Issue with seqNo binding for KV mvcc
Key: HBASE-11401
URL: https://issues.apache.org/jira/browse/HBASE-11401
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Priority: Critical
Fix For: 0.99.0
Attachments: memstore.txt
After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is
implemented in a tricky way now.
In HRegion on write path, we first write to memstore and then write to HLog
finally sync log. So at the time of write to memstore we dont know the WAL
seqNo. To overcome this, we hold the ref to the KV objects just added to
memstore and pass those also to write to wal call. Once the seqNo is
obtained, we will reset the mvcc is those KVs with this seqNo. (While write
to memstore we wrote kvs with a very high temp value for mvcc so that
concurrent readers wont see them)
This model works well with the DefaultMemstore. During the write there wont
be any concurrent call to snapshot().
But now we have memstore as a pluggable interface. The above model of late
binding assumes that the memstore internal datastructure continue to refer to
same java objects. This might not be true always. Like in HBASE-10713, in
btw the kvs can be converted into a CellBlock. If we discontinue to refer to
same KV java objects, we will fail in getting the seqNo assigned as kv mvcc.
If we were doing write and sync to wal and then write to memstore, this would
have get solved. But this model we changed (in 94 I believe) for better perf.
Under HRegion level lock, we write to memstore and then to wal. Finally out
of lock we do the the log sync. So we can not change it now
I tried changing the order of ops within the lock (ie. write to log and then
to memstore) so that we can get the seqNo when write to memstore. But because
of the new HLog write model, we are not guarenteed to get the write to done
immediately.
One possible way can be add a new API in Log level, to get a next seqNo
alone. Call this first and then using which write to memstore and then to wal
(using this seqNo). Just a random thought. Not tried.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049129#comment-14049129
]
Andrew Purtell commented on HBASE-11440:
bq. Should this be committed to branch-1 also?
Ping [~enis]
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11401) Issue with seqNo binding for KV mvcc
[ https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11401: -- Attachment: 11401.changing.order.txt Here is my patch. Changes order applying to memory after appending. Waits on seqid to be set into HLogKey before adding to memory. Also adds a flag to HLogPE which designates using hregion api to append rather than call HLog append and sync directly. Issue with seqNo binding for KV mvcc Key: HBASE-11401 URL: https://issues.apache.org/jira/browse/HBASE-11401 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Anoop Sam John Priority: Critical Fix For: 0.99.0 Attachments: 11401.changing.order.txt, memstore.txt After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is implemented in a tricky way now. In HRegion on write path, we first write to memstore and then write to HLog finally sync log. So at the time of write to memstore we dont know the WAL seqNo. To overcome this, we hold the ref to the KV objects just added to memstore and pass those also to write to wal call. Once the seqNo is obtained, we will reset the mvcc is those KVs with this seqNo. (While write to memstore we wrote kvs with a very high temp value for mvcc so that concurrent readers wont see them) This model works well with the DefaultMemstore. During the write there wont be any concurrent call to snapshot(). But now we have memstore as a pluggable interface. The above model of late binding assumes that the memstore internal datastructure continue to refer to same java objects. This might not be true always. Like in HBASE-10713, in btw the kvs can be converted into a CellBlock. If we discontinue to refer to same KV java objects, we will fail in getting the seqNo assigned as kv mvcc. If we were doing write and sync to wal and then write to memstore, this would have get solved. But this model we changed (in 94 I believe) for better perf. Under HRegion level lock, we write to memstore and then to wal. Finally out of lock we do the the log sync. So we can not change it now I tried changing the order of ops within the lock (ie. write to log and then to memstore) so that we can get the seqNo when write to memstore. But because of the new HLog write model, we are not guarenteed to get the write to done immediately. One possible way can be add a new API in Log level, to get a next seqNo alone. Call this first and then using which write to memstore and then to wal (using this seqNo). Just a random thought. Not tried. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049135#comment-14049135 ] Hudson commented on HBASE-11439: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #350 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/350/]) HBASE-11439 StripeCompaction may not obey the OffPeak rule to compaction (tedyu: rev c8d1d7cef8be88304770e6d461b5574039346d9b) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10800) Use CellComparator instead of KVComparator
[ https://issues.apache.org/jira/browse/HBASE-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10800: --- Attachment: HBASE-10800_1.patch Changes KVComparator to implement CellComparator. All core methods are in CellComparator. MetaComparator is still in KV because it is handling special cases. All changes done in the read path mostly. Removed some of the methods from KeyValue and moved to KeyValueUtil. So KeyValue class is cleaned up. SamePrefixCompartor is removed as it was not getting used except in testcase. Still running testcases, most of them are passing. This would help in adding comparator methods in the CellComparator that could even compare offheap bytes. Will it be better to make CellComparator an interface? Thoughts? Use CellComparator instead of KVComparator -- Key: HBASE-10800 URL: https://issues.apache.org/jira/browse/HBASE-10800 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10800_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11413) [findbugs] RV: Negating the result of compareTo()/compare()
[
https://issues.apache.org/jira/browse/HBASE-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated HBASE-11413:
--
Status: Open (was: Patch Available)
[findbugs] RV: Negating the result of compareTo()/compare()
---
Key: HBASE-11413
URL: https://issues.apache.org/jira/browse/HBASE-11413
Project: HBase
Issue Type: Bug
Reporter: Mike Drob
Priority: Trivial
Attachments: HBASE-11413.patch
Finbugs warns that {{CatalogJanitor.java}}, {{NamespaceUpgrade.java}} and
{{FSTableDescriptors.java}} have the following warning:
{quote}
RV: Negating the result of compareTo()/compare()
(RV_NEGATING_RESULT_OF_COMPARETO)
This code negatives the return value of a compareTo or compare method. This
is a questionable or bad programming practice, since if the return value is
Integer.MIN_VALUE, negating the return value won't negate the sign of the
result. You can achieve the same intended result by reversing the order of
the operands rather than by negating the results.
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11413) [findbugs] RV: Negating the result of compareTo()/compare()
[
https://issues.apache.org/jira/browse/HBASE-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated HBASE-11413:
--
Status: Patch Available (was: Open)
Resubmitting for QA Bot
[findbugs] RV: Negating the result of compareTo()/compare()
---
Key: HBASE-11413
URL: https://issues.apache.org/jira/browse/HBASE-11413
Project: HBase
Issue Type: Bug
Reporter: Mike Drob
Priority: Trivial
Attachments: HBASE-11413.patch
Finbugs warns that {{CatalogJanitor.java}}, {{NamespaceUpgrade.java}} and
{{FSTableDescriptors.java}} have the following warning:
{quote}
RV: Negating the result of compareTo()/compare()
(RV_NEGATING_RESULT_OF_COMPARETO)
This code negatives the return value of a compareTo or compare method. This
is a questionable or bad programming practice, since if the return value is
Integer.MIN_VALUE, negating the return value won't negate the sign of the
result. You can achieve the same intended result by reversing the order of
the operands rather than by negating the results.
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10800) Use CellComparator instead of KVComparator
[ https://issues.apache.org/jira/browse/HBASE-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10800: --- Status: Patch Available (was: In Progress) Use CellComparator instead of KVComparator -- Key: HBASE-10800 URL: https://issues.apache.org/jira/browse/HBASE-10800 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10800_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10866) Decouple HLogSplitterHandler from ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10866: Status: Open (was: Patch Available) Decouple HLogSplitterHandler from ZooKeeper --- Key: HBASE-10866 URL: https://issues.apache.org/jira/browse/HBASE-10866 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Reporter: Mikhail Antonov Assignee: Mikhail Antonov Attachments: HBASE-10866.patch, HBASE-10866.patch, HBASE-10866.patch, HBASE-10866.patch As some sort of follow-up or initial step towards HBASE-10296... Whatever consensus algorithm/library may be the chosen, perhaps on of first practical steps towards this goal would be to better abstract ZK-related API and details, which are now throughout the codebase (mostly leaked throuth ZkUtil, ZooKeeperWatcher and listeners). I'd like to propose a series of patches to help better abstract out zookeeper (and then help develop consensus APIs). Here is first version of patch for initial review (then I'm planning to work on another handlers in regionserver, and then perhaps start working on abstracting listeners). Note - there used to be an attachment with PDF, a write-up, it's now moved up to the parent JIRA. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11401) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment
[ https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11401: -- Summary: Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment (was: Issue with seqNo binding for KV mvcc) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment --- Key: HBASE-11401 URL: https://issues.apache.org/jira/browse/HBASE-11401 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Anoop Sam John Priority: Critical Fix For: 0.99.0 Attachments: 11401.changing.order.txt, memstore.txt After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is implemented in a tricky way now. In HRegion on write path, we first write to memstore and then write to HLog finally sync log. So at the time of write to memstore we dont know the WAL seqNo. To overcome this, we hold the ref to the KV objects just added to memstore and pass those also to write to wal call. Once the seqNo is obtained, we will reset the mvcc is those KVs with this seqNo. (While write to memstore we wrote kvs with a very high temp value for mvcc so that concurrent readers wont see them) This model works well with the DefaultMemstore. During the write there wont be any concurrent call to snapshot(). But now we have memstore as a pluggable interface. The above model of late binding assumes that the memstore internal datastructure continue to refer to same java objects. This might not be true always. Like in HBASE-10713, in btw the kvs can be converted into a CellBlock. If we discontinue to refer to same KV java objects, we will fail in getting the seqNo assigned as kv mvcc. If we were doing write and sync to wal and then write to memstore, this would have get solved. But this model we changed (in 94 I believe) for better perf. Under HRegion level lock, we write to memstore and then to wal. Finally out of lock we do the the log sync. So we can not change it now I tried changing the order of ops within the lock (ie. write to log and then to memstore) so that we can get the seqNo when write to memstore. But because of the new HLog write model, we are not guarenteed to get the write to done immediately. One possible way can be add a new API in Log level, to get a next seqNo alone. Call this first and then using which write to memstore and then to wal (using this seqNo). Just a random thought. Not tried. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11446) Reduce the frequency of RNG calls in SecureWALCellCodec#EncryptedKvEncoder
[ https://issues.apache.org/jira/browse/HBASE-11446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049162#comment-14049162 ] Andrew Purtell commented on HBASE-11446: Any concerns about this [~hitliuyi] ? Reduce the frequency of RNG calls in SecureWALCellCodec#EncryptedKvEncoder -- Key: HBASE-11446 URL: https://issues.apache.org/jira/browse/HBASE-11446 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11446.patch By reducing the frequency of RNG calls in SecureWALCellCodec#EncryptedKvEncoder we can save 37% of on CPU time in that method and 3% of total on CPU time during an ingest test. WAL processing is a critical latency sensitive area. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10866) Decouple HLogSplitterHandler from ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov resolved HBASE-10866. - Resolution: Duplicate Superseded by HBASE-11072 to abstract whole distributed log splitting. Decouple HLogSplitterHandler from ZooKeeper --- Key: HBASE-10866 URL: https://issues.apache.org/jira/browse/HBASE-10866 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Reporter: Mikhail Antonov Assignee: Mikhail Antonov Attachments: HBASE-10866.patch, HBASE-10866.patch, HBASE-10866.patch, HBASE-10866.patch As some sort of follow-up or initial step towards HBASE-10296... Whatever consensus algorithm/library may be the chosen, perhaps on of first practical steps towards this goal would be to better abstract ZK-related API and details, which are now throughout the codebase (mostly leaked throuth ZkUtil, ZooKeeperWatcher and listeners). I'd like to propose a series of patches to help better abstract out zookeeper (and then help develop consensus APIs). Here is first version of patch for initial review (then I'm planning to work on another handlers in regionserver, and then perhaps start working on abstracting listeners). Note - there used to be an attachment with PDF, a write-up, it's now moved up to the parent JIRA. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11444) Remove use of reflection in the User facade
[ https://issues.apache.org/jira/browse/HBASE-11444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049164#comment-14049164 ] Andrew Purtell commented on HBASE-11444: As an alternative half-measure, I wonder if we can cache the resolved implementation class and method handles after the first lookup. Remove use of reflection in the User facade --- Key: HBASE-11444 URL: https://issues.apache.org/jira/browse/HBASE-11444 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Fix For: 0.99.0 The time we spend looking up a user's short name with User#getShortName in hot security code paths is mostly spent in Class#getMethod, ~70%. This is only ~1% of overall CPU time but is the bulk of time spent in AccessController#prePut, for example. Since HBase 1.0 is only going to support Hadoop 2, we can remove the use of reflection and replace with direct calls to UserGroupInformation, or remove the User facade and use UGI directly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11401) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment
[ https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049185#comment-14049185 ] Jeffrey Zhong commented on HBASE-11401: --- [~saint@gmail.com] I checked your patch which seems do right thing. I guess the time is spent in wait for SeqId part. Not sure how easy for you to try two ring buffer strategy to decrease seqId assignment wait time. If we can't re-order the AppendNoSync and write to memstore, would it be possible 1) for plug-able memstore to only deal with KVs within memstore read point 2) add a listener inside KV updateMVCC call in order to give plugable Memstore a chance to change related MVCC as well. Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment --- Key: HBASE-11401 URL: https://issues.apache.org/jira/browse/HBASE-11401 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Anoop Sam John Priority: Critical Fix For: 0.99.0 Attachments: 11401.changing.order.txt, memstore.txt After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is implemented in a tricky way now. In HRegion on write path, we first write to memstore and then write to HLog finally sync log. So at the time of write to memstore we dont know the WAL seqNo. To overcome this, we hold the ref to the KV objects just added to memstore and pass those also to write to wal call. Once the seqNo is obtained, we will reset the mvcc is those KVs with this seqNo. (While write to memstore we wrote kvs with a very high temp value for mvcc so that concurrent readers wont see them) This model works well with the DefaultMemstore. During the write there wont be any concurrent call to snapshot(). But now we have memstore as a pluggable interface. The above model of late binding assumes that the memstore internal datastructure continue to refer to same java objects. This might not be true always. Like in HBASE-10713, in btw the kvs can be converted into a CellBlock. If we discontinue to refer to same KV java objects, we will fail in getting the seqNo assigned as kv mvcc. If we were doing write and sync to wal and then write to memstore, this would have get solved. But this model we changed (in 94 I believe) for better perf. Under HRegion level lock, we write to memstore and then to wal. Finally out of lock we do the the log sync. So we can not change it now I tried changing the order of ops within the lock (ie. write to log and then to memstore) so that we can get the seqNo when write to memstore. But because of the new HLog write model, we are not guarenteed to get the write to done immediately. One possible way can be add a new API in Log level, to get a next seqNo alone. Call this first and then using which write to memstore and then to wal (using this seqNo). Just a random thought. Not tried. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10800) Use CellComparator instead of KVComparator
[
https://issues.apache.org/jira/browse/HBASE-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049203#comment-14049203
]
Hadoop QA commented on HBASE-10800:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653437/HBASE-10800_1.patch
against trunk revision .
ATTACHMENT ID: 12653437
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 69 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 14 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ comparisonResult = CellComparator.CELL_COMPARATOR.compareRows(left,
loffset + KeyValue.ROW_LENGTH_SIZE,
+ comp = samePrefixComparator.compareCommonRowPrefix(seekCell,
currentCell, rowCommonPrefix);
+ public EncodedSeeker createSeeker(CellComparator comparator,
HFileBlockDecodingContext decodingCtx) {
+ conf, cacheConf, (KVComparator)comparator, bloomType, maxKeyCount,
favoredNodes, fileContext);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestKeyValue
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:360)
at
org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:134)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9924//console
This message is automatically generated.
Use CellComparator instead of KVComparator
--
Key: HBASE-10800
URL: https://issues.apache.org/jira/browse/HBASE-10800
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-10800_1.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11413) [findbugs] RV: Negating the result of compareTo()/compare()
[
https://issues.apache.org/jira/browse/HBASE-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049221#comment-14049221
]
stack commented on HBASE-11413:
---
lgtm
Lets see what hadoopqa says. Thanks Mike.
[findbugs] RV: Negating the result of compareTo()/compare()
---
Key: HBASE-11413
URL: https://issues.apache.org/jira/browse/HBASE-11413
Project: HBase
Issue Type: Bug
Reporter: Mike Drob
Priority: Trivial
Attachments: HBASE-11413.patch
Finbugs warns that {{CatalogJanitor.java}}, {{NamespaceUpgrade.java}} and
{{FSTableDescriptors.java}} have the following warning:
{quote}
RV: Negating the result of compareTo()/compare()
(RV_NEGATING_RESULT_OF_COMPARETO)
This code negatives the return value of a compareTo or compare method. This
is a questionable or bad programming practice, since if the return value is
Integer.MIN_VALUE, negating the return value won't negate the sign of the
result. You can achieve the same intended result by reversing the order of
the operands rather than by negating the results.
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10800) Use CellComparator instead of KVComparator
[ https://issues.apache.org/jira/browse/HBASE-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049220#comment-14049220 ] stack commented on HBASE-10800: --- It is not you but what is compareStatic? (when the method is not static) Why not compareKey rather than compareOnlyKeyPortion? Why we have to take a flag on compareStatic when Key only? Should there be a compareKey only method that is called when you are comparing the whole Cell? (Do we ever compare Value? If not, why we have to say compare Key at all?) Should this be a method on its own? +int c = Bytes.compareTo(a.getRowArray(), a.getRowOffset(), a.getRowLength(), +b.getRowArray(), b.getRowOffset(), b.getRowLength()); My sense is that it is done lots of times throughout the codebase. Hmm... doesn't compareRows do the above? Call lit? Why you have non-static methods? You ask if should be an Interface. You thinking different CellComparators? I'd think that it'd be the Cell that would change implementation. Would be trouble if the CellComparator itself changed... If we come up w/ faster CellComparator lets just use it in place of the old? Or what you thinking Ram? I've asked before but forgot, what is a flat key? compareFlatKey Should CellComparator only have compareTo in it? And then CellUtil has the rest? Maybe KVComparator didn't do this so you have CellComparator not doing it. MatchingRow was in KVComparator? Should it be in CellUtil? Hmm... seems like you have put a bunch of the compares into CellUtil. Good. Make the separation cleaner? Feels like these methods should be static again Ram. If you can get away with purging that much from KeyValue, more power to you! Thanks for digging in on this Ram. Use CellComparator instead of KVComparator -- Key: HBASE-10800 URL: https://issues.apache.org/jira/browse/HBASE-10800 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10800_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049233#comment-14049233 ] Hudson commented on HBASE-11439: SUCCESS: Integrated in HBase-1.0 #4 (See [https://builds.apache.org/job/HBase-1.0/4/]) HBASE-11439 StripeCompaction may not obey the OffPeak rule to compaction (Victor Xu) (tedyu: rev b59d9315900345fcba5c5e52758d975d386400a7) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049239#comment-14049239
]
Hadoop QA commented on HBASE-4495:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653428/HBASE-4495-v7.patch
against trunk revision .
ATTACHMENT ID: 12653428
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 145
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.client.TestReplicaWithCluster.testCreateDeleteTable(TestReplicaWithCluster.java:138)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9923//console
This message is automatically generated.
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: 1.0.0
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495-v7.patch, HBASE-4495.patch, HBASE-4495.patch, HBASE-4495.patch,
HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
[jira] [Updated] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[ https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11434: --- Status: Open (was: Patch Available) [AccessController] Disallow inbound cells with reserved tags Key: HBASE-11434 URL: https://issues.apache.org/jira/browse/HBASE-11434 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11434.patch, HBASE-11434.patch The AccessController allows users to store cells with ACL tags encoded by the client. This isn't a security issue currently, because in order to store the cell the user must have a relevant WRITE grant, and the user is allowed to specify whatever ACL for the cell they'd like. However it could become a correctness problem in the future, if we introduce format sanity checking or the like, so let's disallow inbound mutations containing cells with reserved tags like the VisibilityController does. The check is skipped if the active user is a superuser. First, superusers are allowed to do anything. Second, replication (as superuser) must be able to store incoming cells with ACL tags. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049254#comment-14049254 ] Hudson commented on HBASE-11439: FAILURE: Integrated in HBase-0.98 #370 (See [https://builds.apache.org/job/HBase-0.98/370/]) HBASE-11439 StripeCompaction may not obey the OffPeak rule to compaction (tedyu: rev c8d1d7cef8be88304770e6d461b5574039346d9b) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11439) StripeCompaction may not obey the OffPeak rule to compaction
[ https://issues.apache.org/jira/browse/HBASE-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049255#comment-14049255 ] Hudson commented on HBASE-11439: SUCCESS: Integrated in HBase-TRUNK #5256 (See [https://builds.apache.org/job/HBase-TRUNK/5256/]) HBASE-11439 StripeCompaction may not obey the OffPeak rule to compaction (Victor Xu) (tedyu: rev 5f3c0b7f9be00430e8ea53b7d4afae5f6f3a8b3b) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java StripeCompaction may not obey the OffPeak rule to compaction Key: HBASE-11439 URL: https://issues.apache.org/jira/browse/HBASE-11439 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.2 Environment: jdk1.7.0_45, hadoop-cdh5, hbase-0.98.2 Reporter: Victor Xu Assignee: Victor Xu Priority: Minor Fix For: 0.99.0, 0.98.4, 2.0.0 Attachments: HBASE-11439-v2.patch, HBASE-11439.patch, HBASE-11439.patch This is just a written mistake. The parameters in stripePolicy.applyCompactionPolicy() of StripeCompactionPolicy.java is wrong. Just swap the 'mightBeStuck' and 'mayUseOffPeak'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11401) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment
[ https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11401: -- Attachment: wpatch.traces.svg nopatch.traces.svg Here are flame graphs running ten threads with and without patch. If you dig you can see 7% CPU or so doing get of sequence number. We seem to be doing more waitFor inside in disruptor too when patch is in place. On other hand, I don't see add-to-memstore in the patched version (There is no mention in the svg of DefaultMemStore nor in the collected traces -- it is 'fast' in patched version?) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment --- Key: HBASE-11401 URL: https://issues.apache.org/jira/browse/HBASE-11401 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Anoop Sam John Priority: Critical Fix For: 0.99.0 Attachments: 11401.changing.order.txt, memstore.txt, nopatch.traces.svg, wpatch.traces.svg After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is implemented in a tricky way now. In HRegion on write path, we first write to memstore and then write to HLog finally sync log. So at the time of write to memstore we dont know the WAL seqNo. To overcome this, we hold the ref to the KV objects just added to memstore and pass those also to write to wal call. Once the seqNo is obtained, we will reset the mvcc is those KVs with this seqNo. (While write to memstore we wrote kvs with a very high temp value for mvcc so that concurrent readers wont see them) This model works well with the DefaultMemstore. During the write there wont be any concurrent call to snapshot(). But now we have memstore as a pluggable interface. The above model of late binding assumes that the memstore internal datastructure continue to refer to same java objects. This might not be true always. Like in HBASE-10713, in btw the kvs can be converted into a CellBlock. If we discontinue to refer to same KV java objects, we will fail in getting the seqNo assigned as kv mvcc. If we were doing write and sync to wal and then write to memstore, this would have get solved. But this model we changed (in 94 I believe) for better perf. Under HRegion level lock, we write to memstore and then to wal. Finally out of lock we do the the log sync. So we can not change it now I tried changing the order of ops within the lock (ie. write to log and then to memstore) so that we can get the seqNo when write to memstore. But because of the new HLog write model, we are not guarenteed to get the write to done immediately. One possible way can be add a new API in Log level, to get a next seqNo alone. Call this first and then using which write to memstore and then to wal (using this seqNo). Just a random thought. Not tried. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[ https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11434: --- Status: Patch Available (was: Open) [AccessController] Disallow inbound cells with reserved tags Key: HBASE-11434 URL: https://issues.apache.org/jira/browse/HBASE-11434 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11434.patch, HBASE-11434.patch, HBASE-11434.patch The AccessController allows users to store cells with ACL tags encoded by the client. This isn't a security issue currently, because in order to store the cell the user must have a relevant WRITE grant, and the user is allowed to specify whatever ACL for the cell they'd like. However it could become a correctness problem in the future, if we introduce format sanity checking or the like, so let's disallow inbound mutations containing cells with reserved tags like the VisibilityController does. The check is skipped if the active user is a superuser. First, superusers are allowed to do anything. Second, replication (as superuser) must be able to store incoming cells with ACL tags. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11401) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment
[ https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049269#comment-14049269 ] stack commented on HBASE-11401: --- [~jeffreyz] Seems like its the wait on seqid and less disruptor batching (flame graphs raise a couple of new questions though). Was thinking like yourself whether pluggable memstore could work around Cells that have not yet gotten a sequenceid (Compressing memstore or encoding as a cellblock I'd suppose you'd compress a snapshot; if a Cell does not yet have a sequenceid, wait on the Cell sequenceid?) Otherwise, we'll be back here to do some gymnastics (double ringbuffer, etc.). Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment --- Key: HBASE-11401 URL: https://issues.apache.org/jira/browse/HBASE-11401 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Anoop Sam John Priority: Critical Fix For: 0.99.0 Attachments: 11401.changing.order.txt, memstore.txt, nopatch.traces.svg, wpatch.traces.svg After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is implemented in a tricky way now. In HRegion on write path, we first write to memstore and then write to HLog finally sync log. So at the time of write to memstore we dont know the WAL seqNo. To overcome this, we hold the ref to the KV objects just added to memstore and pass those also to write to wal call. Once the seqNo is obtained, we will reset the mvcc is those KVs with this seqNo. (While write to memstore we wrote kvs with a very high temp value for mvcc so that concurrent readers wont see them) This model works well with the DefaultMemstore. During the write there wont be any concurrent call to snapshot(). But now we have memstore as a pluggable interface. The above model of late binding assumes that the memstore internal datastructure continue to refer to same java objects. This might not be true always. Like in HBASE-10713, in btw the kvs can be converted into a CellBlock. If we discontinue to refer to same KV java objects, we will fail in getting the seqNo assigned as kv mvcc. If we were doing write and sync to wal and then write to memstore, this would have get solved. But this model we changed (in 94 I believe) for better perf. Under HRegion level lock, we write to memstore and then to wal. Finally out of lock we do the the log sync. So we can not change it now I tried changing the order of ops within the lock (ie. write to log and then to memstore) so that we can get the seqNo when write to memstore. But because of the new HLog write model, we are not guarenteed to get the write to done immediately. One possible way can be add a new API in Log level, to get a next seqNo alone. Call this first and then using which write to memstore and then to wal (using this seqNo). Just a random thought. Not tried. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[ https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11434: --- Attachment: HBASE-11434.patch Yes. New patch, with unit test. [AccessController] Disallow inbound cells with reserved tags Key: HBASE-11434 URL: https://issues.apache.org/jira/browse/HBASE-11434 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11434.patch, HBASE-11434.patch, HBASE-11434.patch The AccessController allows users to store cells with ACL tags encoded by the client. This isn't a security issue currently, because in order to store the cell the user must have a relevant WRITE grant, and the user is allowed to specify whatever ACL for the cell they'd like. However it could become a correctness problem in the future, if we introduce format sanity checking or the like, so let's disallow inbound mutations containing cells with reserved tags like the VisibilityController does. The check is skipped if the active user is a superuser. First, superusers are allowed to do anything. Second, replication (as superuser) must be able to store incoming cells with ACL tags. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[
https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049277#comment-14049277
]
Hadoop QA commented on HBASE-11434:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653456/HBASE-11434.patch
against trunk revision .
ATTACHMENT ID: 12653456
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9925//console
This message is automatically generated.
[AccessController] Disallow inbound cells with reserved tags
Key: HBASE-11434
URL: https://issues.apache.org/jira/browse/HBASE-11434
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.99.0, 0.98.4
Attachments: HBASE-11434.patch, HBASE-11434.patch, HBASE-11434.patch
The AccessController allows users to store cells with ACL tags encoded by the
client. This isn't a security issue currently, because in order to store the
cell the user must have a relevant WRITE grant, and the user is allowed to
specify whatever ACL for the cell they'd like. However it could become a
correctness problem in the future, if we introduce format sanity checking or
the like, so let's disallow inbound mutations containing cells with reserved
tags like the VisibilityController does.
The check is skipped if the active user is a superuser. First, superusers are
allowed to do anything. Second, replication (as superuser) must be able to
store incoming cells with ACL tags.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11401) Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment
[ https://issues.apache.org/jira/browse/HBASE-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049283#comment-14049283 ] stack commented on HBASE-11401: --- Absence of memstore adds in the patched version must mean we are not applying edits to memstore in the patch. Would need to dig in when we come back here on why not. Late-binding sequenceid presumes a particular KeyValue mvcc format hampering experiment --- Key: HBASE-11401 URL: https://issues.apache.org/jira/browse/HBASE-11401 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Anoop Sam John Priority: Critical Fix For: 0.99.0 Attachments: 11401.changing.order.txt, memstore.txt, nopatch.traces.svg, wpatch.traces.svg After HBASE-8763, we have combined KV mvcc and HLog seqNo. This is implemented in a tricky way now. In HRegion on write path, we first write to memstore and then write to HLog finally sync log. So at the time of write to memstore we dont know the WAL seqNo. To overcome this, we hold the ref to the KV objects just added to memstore and pass those also to write to wal call. Once the seqNo is obtained, we will reset the mvcc is those KVs with this seqNo. (While write to memstore we wrote kvs with a very high temp value for mvcc so that concurrent readers wont see them) This model works well with the DefaultMemstore. During the write there wont be any concurrent call to snapshot(). But now we have memstore as a pluggable interface. The above model of late binding assumes that the memstore internal datastructure continue to refer to same java objects. This might not be true always. Like in HBASE-10713, in btw the kvs can be converted into a CellBlock. If we discontinue to refer to same KV java objects, we will fail in getting the seqNo assigned as kv mvcc. If we were doing write and sync to wal and then write to memstore, this would have get solved. But this model we changed (in 94 I believe) for better perf. Under HRegion level lock, we write to memstore and then to wal. Finally out of lock we do the the log sync. So we can not change it now I tried changing the order of ops within the lock (ie. write to log and then to memstore) so that we can get the seqNo when write to memstore. But because of the new HLog write model, we are not guarenteed to get the write to done immediately. One possible way can be add a new API in Log level, to get a next seqNo alone. Call this first and then using which write to memstore and then to wal (using this seqNo). Just a random thought. Not tried. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-4495) CatalogTracker has an identity crisis; needs to be cut-back in scope
[
https://issues.apache.org/jira/browse/HBASE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4495:
-
Attachment: HBASE-4495-v7.patch
Failures seem unrelated. Retry.
CatalogTracker has an identity crisis; needs to be cut-back in scope
Key: HBASE-4495
URL: https://issues.apache.org/jira/browse/HBASE-4495
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: 1.0.0
Fix For: 0.99.0
Attachments: HBASE-4495-v2.patch, HBASE-4495-v3.patch,
HBASE-4495-v5.patch, HBASE-4495-v5.patch, HBASE-4495-v6.patch,
HBASE-4495-v7.patch, HBASE-4495-v7.patch, HBASE-4495.patch, HBASE-4495.patch,
HBASE-4495.patch, HBASE-4495.patch
CT needs a good reworking. I'd suggest its scope be cut way down to only
deal in zk transactions rather than zk and reading meta location in hbase
(over an HConnection) and being a purveyor of HRegionInterfaces on meta and
root servers and being an Abortable and a verifier of catalog locations.
Once this is done, I would suggest it then better belongs over under the zk
package and that the Meta* classes then move to client package.
Here's some messy notes I added to head of CT class in hbase-3446 where I
spent some time trying to make out what it was CT did.
{code}
// TODO: This class needs a rethink. The original intent was that it would
be
// the one-stop-shop for root and meta locations and that it would get this
// info from reading and watching zk state. The class was to be used by
// servers when they needed to know of root and meta movement but also by
// client-side (inside in HTable) so rather than figure root and meta
// locations on fault, the client would instead get notifications out of zk.
//
// But this original intent is frustrated by the fact that this class has to
// read an hbase table, the -ROOT- table, to figure out the .META. region
// location which means we depend on an HConnection. HConnection will do
// retrying but also, it has its own mechanism for finding root and meta
// locations (and for 'verifying'; it tries the location and if it fails,
does
// new lookup, etc.). So, at least for now, HConnection (or HTable) can't
// have a CT since CT needs a HConnection (Even then, do want HT to have a
CT?
// For HT keep up a session with ZK? Rather, shouldn't we do like
asynchbase
// where we'd open a connection to zk, read what we need then let the
// connection go?). The 'fix' is make it so both root and meta addresses
// are wholey up in zk -- not in zk (root) -- and in an hbase table (meta).
//
// But even then, this class does 'verification' of the location and it does
// this by making a call over an HConnection (which will do its own root
// and meta lookups). Isn't this verification 'useless' since when we
// return, whatever is dependent on the result of this call then needs to
// use HConnection; what we have verified may change in meantime
(HConnection
// uses the CT primitives, the root and meta trackers finding root
locations).
//
// When meta is moved to zk, this class may make more sense. In the
// meantime, it does not cohere. It should just watch meta and root and
// NOT do verification -- let that be out in HConnection since its going to
// be done there ultimately anyways.
//
// This class has spread throughout the codebase. It needs to be reigned
in.
// This class should be used server-side only, even if we move meta location
// up into zk. Currently its used over in the client package. Its used in
// MetaReader and MetaEditor classes usually just to get the Configuration
// its using (It does this indirectly by asking its HConnection for its
// Configuration and even then this is just used to get an HConnection out
on
// the other end). St.Ack 10/23/2011.
//
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11372) Remove SlabCache
[ https://issues.apache.org/jira/browse/HBASE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049290#comment-14049290 ] Enis Soztutar commented on HBASE-11372: --- +1 for branch-1 as well. Remove SlabCache Key: HBASE-11372 URL: https://issues.apache.org/jira/browse/HBASE-11372 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.99.0 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0 Attachments: HBASE-11372.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk
[
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049288#comment-14049288
]
Enis Soztutar commented on HBASE-11440:
---
+1 for branch-1. We would need the tags to be carried with replication.
Make KeyValueCodecWithTags as the default codec for replication in trunk
Key: HBASE-11440
URL: https://issues.apache.org/jira/browse/HBASE-11440
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-11440.patch, HBASE-11440_1.patch
Set
{code}
property
namehbase.replication.rpc.codec/name
valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
/property
{code}
in hbase-default.xml
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[ https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11434: --- Status: Open (was: Patch Available) [AccessController] Disallow inbound cells with reserved tags Key: HBASE-11434 URL: https://issues.apache.org/jira/browse/HBASE-11434 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11434.patch, HBASE-11434.patch, HBASE-11434.patch The AccessController allows users to store cells with ACL tags encoded by the client. This isn't a security issue currently, because in order to store the cell the user must have a relevant WRITE grant, and the user is allowed to specify whatever ACL for the cell they'd like. However it could become a correctness problem in the future, if we introduce format sanity checking or the like, so let's disallow inbound mutations containing cells with reserved tags like the VisibilityController does. The check is skipped if the active user is a superuser. First, superusers are allowed to do anything. Second, replication (as superuser) must be able to store incoming cells with ACL tags. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11434) [AccessController] Disallow inbound cells with reserved tags
[ https://issues.apache.org/jira/browse/HBASE-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11434: --- Status: Patch Available (was: Open) [AccessController] Disallow inbound cells with reserved tags Key: HBASE-11434 URL: https://issues.apache.org/jira/browse/HBASE-11434 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11434.patch, HBASE-11434.patch, HBASE-11434.patch, HBASE-11434.patch The AccessController allows users to store cells with ACL tags encoded by the client. This isn't a security issue currently, because in order to store the cell the user must have a relevant WRITE grant, and the user is allowed to specify whatever ACL for the cell they'd like. However it could become a correctness problem in the future, if we introduce format sanity checking or the like, so let's disallow inbound mutations containing cells with reserved tags like the VisibilityController does. The check is skipped if the active user is a superuser. First, superusers are allowed to do anything. Second, replication (as superuser) must be able to store incoming cells with ACL tags. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11344) Hide row keys and such from the web UIs
[ https://issues.apache.org/jira/browse/HBASE-11344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-11344: Attachment: 11344-2.txt This fixes issues reported by hadoopqa. Hide row keys and such from the web UIs --- Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0 Attachments: 11344-1.txt, 11344-2.txt The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
