[jira] [Commented] (HBASE-13183) Make ZK tickTime configurable in standalone HBase
[ https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354391#comment-14354391 ] Hudson commented on HBASE-13183: FAILURE: Integrated in HBase-0.98 #888 (See [https://builds.apache.org/job/HBase-0.98/888/]) HBASE-13183 Make ZK tickTime configurable in standalone HBase (Alex Araujo) (apurtell: rev 1ea43070c241a4887d1f7affaccaea1602f33f10) * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java Make ZK tickTime configurable in standalone HBase - Key: HBASE-13183 URL: https://issues.apache.org/jira/browse/HBASE-13183 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.98.10.1 Reporter: Alex Araujo Assignee: Alex Araujo Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-13183-0.98.patch, HBASE-13183.patch Standalone HBase does not allow the ZooKeeper tickTime to be configured for the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 2000 ms, which is too low for some use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13185) Document hbase.regionserver.thrift.framed.max_frame_size_in_mb more clearly
[
https://issues.apache.org/jira/browse/HBASE-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354400#comment-14354400
]
Hudson commented on HBASE-13185:
FAILURE: Integrated in HBase-TRUNK #6230 (See
[https://builds.apache.org/job/HBase-TRUNK/6230/])
HBASE-13185 Document hbase.regionserver.thrift.framed.max_frame_size_in_mb more
clearly (stack: rev 0654d13403dc8c739d2cb9ec6820aacda2eb3017)
* hbase-common/src/main/resources/hbase-default.xml
Document hbase.regionserver.thrift.framed.max_frame_size_in_mb more clearly
---
Key: HBASE-13185
URL: https://issues.apache.org/jira/browse/HBASE-13185
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Daisuke Kobayashi
Assignee: Daisuke Kobayashi
Priority: Trivial
Fix For: 2.0.0
Attachments: HBASE-13185.patch
Current document does not make sense how much is it going to allocate for
{{hbase.regionserver.thrift.framed.max_frame_size_in_mb}}:
http://hbase.apache.org/book.html#hbase.regionserver.thrift.framed.max_frame_size_in_mb
It is 2MB by default per the code, so that it would be fine to document this:
https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java#L450-L451
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13186) HBase build error due to checkstyle
[ https://issues.apache.org/jira/browse/HBASE-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354399#comment-14354399 ] Hudson commented on HBASE-13186: FAILURE: Integrated in HBase-TRUNK #6230 (See [https://builds.apache.org/job/HBase-TRUNK/6230/]) HBASE-13186 HBase build error due to checkstyle (mstanleyjones: rev c3c2039dd81e8c2d667f1a31d6cff7bfcb2c0bc6) * pom.xml HBase build error due to checkstyle --- Key: HBASE-13186 URL: https://issues.apache.org/jira/browse/HBASE-13186 Project: HBase Issue Type: Bug Components: build Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 Attachments: HBASE-13186.patch See logs in https://builds.apache.org/job/PreCommit-HBASE-Build/13138//consoleFull. Root cause is that checkstyle is looking in target/. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13167) The check for balancerCutoffTime is buggy
[
https://issues.apache.org/jira/browse/HBASE-13167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354406#comment-14354406
]
Mikhail Antonov commented on HBASE-13167:
-
pinging [~eclark]
The check for balancerCutoffTime is buggy
-
Key: HBASE-13167
URL: https://issues.apache.org/jira/browse/HBASE-13167
Project: HBase
Issue Type: Bug
Components: Balancer
Affects Versions: 0.98.8
Reporter: Tianyin Xu
Attachments: HBASE-13167.patch
In HMaster, the checking logic for balancerCutoffTime in buggy.
See the following code:
{code:title=HMaster.java|borderStyle=solid}
1419 private int getBalancerCutoffTime() {
1420 int balancerCutoffTime =
1421 getConfiguration().getInt(hbase.balancer.max.balancing, -1);
1422 if (balancerCutoffTime == -1) {
1423 // No time period set so create one
1424 int balancerPeriod =
1425 getConfiguration().getInt(hbase.balancer.period, 30);
1426 balancerCutoffTime = balancerPeriod;
1427 // If nonsense period, set it to balancerPeriod
1428 if (balancerCutoffTime = 0) balancerCutoffTime = balancerPeriod;
1429 }
1430 return balancerCutoffTime;
1431 }
{code}
Basically, the check for nonsense period is no effect, because
balancerCutoffTime is always assigned to be balancerPeriod whatever the value
is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13162) Add capability for cleaning hbase acls to hbase cleanup script.
[ https://issues.apache.org/jira/browse/HBASE-13162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354638#comment-14354638 ] Anoop Sam John commented on HBASE-13162: Fine no issues. I was just saying. :-) Add capability for cleaning hbase acls to hbase cleanup script. --- Key: HBASE-13162 URL: https://issues.apache.org/jira/browse/HBASE-13162 Project: HBase Issue Type: Improvement Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-13162.patch Adding a new option --cleanAcls for cleaning up the acl related stuff in zookeeper. This comes handy when the user wants to rollback security. This script created by [~mbertozzi] and was used couple of times internally. Thought it would be a nice addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky
[ https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354777#comment-14354777 ] Hudson commented on HBASE-13182: SUCCESS: Integrated in HBase-TRUNK #6233 (See [https://builds.apache.org/job/HBase-TRUNK/6233/]) HBASE-13182 Test NamespaceAuditor/AccessController create/delete table is flaky (matteo.bertozzi: rev eeb8c846d99970f590c695d00732f8c4f02defda) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/namespace/TestNamespaceAuditor.java Test NamespaceAuditor/AccessController create/delete table is flaky --- Key: HBASE-13182 URL: https://issues.apache.org/jira/browse/HBASE-13182 Project: HBase Issue Type: Test Components: test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-13182-v0.patch Similar to HBASE-13179 TestNamespaceAuditor and the two test AccessController fail when create or delete table takes more time.and it is because the sync version of the Admin operation is not sync, but relies on the last operation on the server e.g. delete meta. but the post-operation method of the coprocessor is called after meta is deleted.. long story short, the client is not really sync and it will be fixed by HBASE-12439. similar to what we do in TestMasterObserver we can add an observer with a countDownLatch to wait the postOpHandler call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13189) PrefixTreecell should implement HeapSize
ramkrishna.s.vasudevan created HBASE-13189: -- Summary: PrefixTreecell should implement HeapSize Key: HBASE-13189 URL: https://issues.apache.org/jira/browse/HBASE-13189 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0, 1.0.1, 1.1.0 PrefixTreeCell should implement HeapSize. The CellUtil.estimatedHeapSize getting used in scanner estimates. With PrefixTreecell not implementing HeapSize we may get an incorrect result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13188) java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
[
https://issues.apache.org/jira/browse/HBASE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354749#comment-14354749
]
Hadoop QA commented on HBASE-13188:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703632/HBASE-13188.patch
against master branch at commit eeb8c846d99970f590c695d00732f8c4f02defda.
ATTACHMENT ID: 12703632
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13153//console
This message is automatically generated.
java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
--
Key: HBASE-13188
URL: https://issues.apache.org/jira/browse/HBASE-13188
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13188.patch
Running a range scan with PE tool with 25 threads getting this error
{code}
java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.hbase.io.BoundedByteBufferPool.putBuffer(BoundedByteBufferPool.java:104)
at org.apache.hadoop.hbase.ipc.RpcServer$Call.done(RpcServer.java:325)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.processResponse(RpcServer.java:1078)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.processAllResponses(RpcServer.java:1103)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doAsyncWrite(RpcServer.java:1036)
at
[jira] [Commented] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354796#comment-14354796
]
Hudson commented on HBASE-13063:
FAILURE: Integrated in HBase-1.1 #270 (See
[https://builds.apache.org/job/HBase-1.1/270/])
HBASE-13063 Allow to turn off memstore replication for region replicas
(matteo.bertozzi: rev eb2193afa0d266063b705d8743a940a06a5c2e9f)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
Allow to turn off memstore replication for region replicas
--
Key: HBASE-13063
URL: https://issues.apache.org/jira/browse/HBASE-13063
Project: HBase
Issue Type: New Feature
Components: regionserver, Replication
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
HBASE-13063-v2.patch
HBASE-11568 allows to use replication to send wal edits from the primary to
the replicas.
sometimes the memstore replication is not required, so it will be nice have a
flag to disable it.
the result will be more or less the same to what we have in phase-1, but with
the row-level consistency provided by the flush edit transfered via
replication to refresh the hfiles.
create 't1', 'f', {REGION_REPLICATION = 2, REGION_MEMSTORE_REPLICATION =
false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354593#comment-14354593 ] Hudson commented on HBASE-12405: SUCCESS: Integrated in HBase-TRUNK #6232 (See [https://builds.apache.org/job/HBase-TRUNK/6232/]) HBASE-12405 addendum fix HRegion heap size (zhangduo: rev 105f9b328221eef30c61a36593927be9a61200b7) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java WAL accounting by Store --- Key: HBASE-12405 URL: https://issues.apache.org/jira/browse/HBASE-12405 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 2.0.0, 1.1.0 Reporter: zhangduo Assignee: zhangduo Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12405-addendum.patch, HBASE-12405.patch, HBASE-12405_1.patch, HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch, HBASE-12405_5.patch HBASE-10201 has made flush decisions per Store, but has not done enough work on HLog, so there are two problems: 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. 2. There maybe holes in WAL accounting. For example, assume family A with sequence id 1 and 3, family B with seqId 2. If we flush family A, we can only record that WAL before sequence id 1 can be removed safely. If we do a replay at this point, sequence id 3 will also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky
[ https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354594#comment-14354594 ] Hudson commented on HBASE-13182: FAILURE: Integrated in HBase-1.1 #269 (See [https://builds.apache.org/job/HBase-1.1/269/]) HBASE-13182 TestAccessController create/delete table is flaky (matteo.bertozzi: rev 52ff5485a8d0997f58514d4ef807faccc7378e5a) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java Test NamespaceAuditor/AccessController create/delete table is flaky --- Key: HBASE-13182 URL: https://issues.apache.org/jira/browse/HBASE-13182 Project: HBase Issue Type: Test Components: test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-13182-v0.patch Similar to HBASE-13179 TestNamespaceAuditor and the two test AccessController fail when create or delete table takes more time.and it is because the sync version of the Admin operation is not sync, but relies on the last operation on the server e.g. delete meta. but the post-operation method of the coprocessor is called after meta is deleted.. long story short, the client is not really sync and it will be fixed by HBASE-12439. similar to what we do in TestMasterObserver we can add an observer with a countDownLatch to wait the postOpHandler call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi reopened HBASE-13181:
-
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-13181:
Status: Patch Available (was: Reopened)
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13181-v0.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-13063:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Allow to turn off memstore replication for region replicas
--
Key: HBASE-13063
URL: https://issues.apache.org/jira/browse/HBASE-13063
Project: HBase
Issue Type: New Feature
Components: regionserver, Replication
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
HBASE-13063-v2.patch
HBASE-11568 allows to use replication to send wal edits from the primary to
the replicas.
sometimes the memstore replication is not required, so it will be nice have a
flag to disable it.
the result will be more or less the same to what we have in phase-1, but with
the row-level consistency provided by the flush edit transfered via
replication to refresh the hfiles.
create 't1', 'f', {REGION_REPLICATION = 2, REGION_MEMSTORE_REPLICATION =
false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-13181:
Attachment: HBASE-13181-v0.patch
The problem is a missing close() of the Output stream.
we are running with hfile.format = v2 by default
and since the size of the v2 file is small (700bytes vs the 4.5k of v3) the
file doesn't get flushed, so we end up with a 0 byte file that throws the
corrupted file exception.
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13181-v0.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354830#comment-14354830
]
Hadoop QA commented on HBASE-13181:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703638/HBASE-13181-v0.patch
against master branch at commit cbf03787ea04da8d1c2ceef8fbb4ccbabb7d67ed.
ATTACHMENT ID: 12703638
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13154//console
This message is automatically generated.
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13181-v0.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
[jira] [Commented] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354856#comment-14354856
]
Hudson commented on HBASE-13063:
SUCCESS: Integrated in HBase-TRUNK #6234 (See
[https://builds.apache.org/job/HBase-TRUNK/6234/])
HBASE-13063 Allow to turn off memstore replication for region replicas
(matteo.bertozzi: rev cbf03787ea04da8d1c2ceef8fbb4ccbabb7d67ed)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
Allow to turn off memstore replication for region replicas
--
Key: HBASE-13063
URL: https://issues.apache.org/jira/browse/HBASE-13063
Project: HBase
Issue Type: New Feature
Components: regionserver, Replication
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
HBASE-13063-v2.patch
HBASE-11568 allows to use replication to send wal edits from the primary to
the replicas.
sometimes the memstore replication is not required, so it will be nice have a
flag to disable it.
the result will be more or less the same to what we have in phase-1, but with
the row-level consistency provided by the flush edit transfered via
replication to refresh the hfiles.
create 't1', 'f', {REGION_REPLICATION = 2, REGION_MEMSTORE_REPLICATION =
false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13183) Make ZK tickTime configurable in standalone HBase
[ https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354893#comment-14354893 ] Alex Araujo commented on HBASE-13183: - Thanks for committing [~apurtell]. I looked at the build and test failures above and they appear to be unrelated to this change. Make ZK tickTime configurable in standalone HBase - Key: HBASE-13183 URL: https://issues.apache.org/jira/browse/HBASE-13183 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.98.10.1 Reporter: Alex Araujo Assignee: Alex Araujo Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-13183-0.98.patch, HBASE-13183.patch Standalone HBase does not allow the ZooKeeper tickTime to be configured for the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 2000 ms, which is too low for some use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7126) Update website with info on how to report security bugs
[ https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-7126: --- Status: Open (was: Patch Available) Update website with info on how to report security bugs Key: HBASE-7126 URL: https://issues.apache.org/jira/browse/HBASE-7126 Project: HBase Issue Type: Task Components: documentation Reporter: Eli Collins Assignee: Misty Stanley-Jones Priority: Critical Labels: website Attachments: HBASE-7126-v1.patch, HBASE-7126.patch The HBase website should be updated with information on how to report potential security vulnerabilities. In Hadoop land we have a private security list that anyone case post to that we point to on our list page: Hadoop example http://hadoop.apache.org/general_lists.html#Security. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7126) Update website with info on how to report security bugs
[ https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-7126: --- Attachment: HBASE-7126-v1.patch Attempted to address feedback in the latest patch. Update website with info on how to report security bugs Key: HBASE-7126 URL: https://issues.apache.org/jira/browse/HBASE-7126 Project: HBase Issue Type: Task Components: documentation Reporter: Eli Collins Assignee: Misty Stanley-Jones Priority: Critical Labels: website Attachments: HBASE-7126-v1.patch, HBASE-7126.patch The HBase website should be updated with information on how to report potential security vulnerabilities. In Hadoop land we have a private security list that anyone case post to that we point to on our list page: Hadoop example http://hadoop.apache.org/general_lists.html#Security. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356184#comment-14356184 ] ramkrishna.s.vasudevan commented on HBASE-11425: Adding to what Anoop says, we are not trying to focus on where we do compression. That part remains the same. It is only after we start using an Hfileblock we tend to use it in a offheap mode and particularly avoid the copy that happens in the BucketCache every time a block needs to be used. (In this case it is going to be a decompressed block only). But one point to note here is that in the case of DBE it is an encoded block - and we still go on with the encoded block only and the existing logic of decoding the block still works the same way. In the existing code there are two copies that happen here - one from the BucketCache and other in the DBE algo. Now we try to avoid the first one. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356187#comment-14356187 ] ramkrishna.s.vasudevan commented on HBASE-11425: bq.Would be good to get some answers on his questions too. I could take that IA. Get some tests in this area to be more clear on this. bq.If only for illustration of where you are focused, suggest you add to the doc Sure. Would also ensure the other points that were specifically discussed gets added to the doc. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356185#comment-14356185 ] ramkrishna.s.vasudevan commented on HBASE-11425: Adding to what Anoop says, we are not trying to focus on where we do compression. That part remains the same. It is only after we start using an Hfileblock we tend to use it in a offheap mode and particularly avoid the copy that happens in the BucketCache every time a block needs to be used. (In this case it is going to be a decompressed block only). But one point to note here is that in the case of DBE it is an encoded block - and we still go on with the encoded block only and the existing logic of decoding the block still works the same way. In the existing code there are two copies that happen here - one from the BucketCache and other in the DBE algo. Now we try to avoid the first one. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13063:
--
Attachment: hbase-13063-addendum.patch
I think the problem is that the flush call after the writes may not flush
because the primary region may already be flushing (saw from failed test logs
running in a loop).
This may cause a problem with the test since it expects that the last flush
will flush everything. I have fixed the problem with this addendum patch by not
requesting the flush when the seconday region replica comes up when memstore
replication is set to false (because we do not need this for correctness).
Reviews welcome.
Allow to turn off memstore replication for region replicas
--
Key: HBASE-13063
URL: https://issues.apache.org/jira/browse/HBASE-13063
Project: HBase
Issue Type: New Feature
Components: regionserver, Replication
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
HBASE-13063-v2.patch, hbase-13063-addendum.patch
HBASE-11568 allows to use replication to send wal edits from the primary to
the replicas.
sometimes the memstore replication is not required, so it will be nice have a
flag to disable it.
the result will be more or less the same to what we have in phase-1, but with
the row-level consistency provided by the flush edit transfered via
replication to refresh the hfiles.
create 't1', 'f', {REGION_REPLICATION = 2, REGION_MEMSTORE_REPLICATION =
false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13196) Add info about default number of versions when using the export tool
[
https://issues.apache.org/jira/browse/HBASE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356299#comment-14356299
]
Sean Busbey commented on HBASE-13196:
-
{code}
$ bin/hbase org.apache.hadoop.hbase.mapreduce.Export tablename outputdir
[versions [starttime [endtime]]]
+By default, the `Export` tool only exports the newest version of a given cell,
regardless of the number of versions stored. To export more than one version,
replace *_version_* with the desired number of versions.
{code}
the command help says versions and the text says version
otherwise +1 lgtm, fine to fix on commit.
Add info about default number of versions when using the export tool
Key: HBASE-13196
URL: https://issues.apache.org/jira/browse/HBASE-13196
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Attachments: HBASE-13196.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13191) mvn site fails on jenkins due to permgen
[
https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356318#comment-14356318
]
Hadoop QA commented on HBASE-13191:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703819/HBASE-13191.not-really-a.patch.txt
against master branch at commit 09e9c92d7699b985f45a48242a24af7c848926f0.
ATTACHMENT ID: 12703819
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13170//console
This message is automatically generated.
mvn site fails on jenkins due to permgen
Key: HBASE-13191
URL: https://issues.apache.org/jira/browse/HBASE-13191
Project: HBase
Issue Type: Bug
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 2.0.0
Attachments: HBASE-13191.1.patch.txt,
HBASE-13191.not-really-a.patch.txt, HBASE-13191.not-really-a.patch.txt,
HBASE-13191.not-really-a.patch.txt
patch QA is failing the mvn site goal due to permgen.
looks like we have ~3 places we try to set it. consolidate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-7126) Update website with info on how to report security bugs
[ https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-7126: --- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, pushed. Update website with info on how to report security bugs Key: HBASE-7126 URL: https://issues.apache.org/jira/browse/HBASE-7126 Project: HBase Issue Type: Task Components: documentation Reporter: Eli Collins Assignee: Misty Stanley-Jones Priority: Critical Labels: website Fix For: 2.0.0 Attachments: HBASE-7126-v1.patch, HBASE-7126.patch The HBase website should be updated with information on how to report potential security vulnerabilities. In Hadoop land we have a private security list that anyone case post to that we point to on our list page: Hadoop example http://hadoop.apache.org/general_lists.html#Security. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13142: -- Attachment: 13142v5.addendum.txt Addendum pushed to branch-1+ [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356334#comment-14356334 ] stack commented on HBASE-13142: --- [~ram_krish] You probably need the addendum boss. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356348#comment-14356348 ] ramkrishna.s.vasudevan commented on HBASE-13142: bq.You probably need the addendum Yes +1 for addendum. Thanks Stack. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.addendum.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the reviews, Nick and Andrew. Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.0.1, 1.1.0, 0.98.13 Attachments: 13168-0.98.patch, 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13165) Fix docs and scripts for default max heaps size after HBASE-11804
[ https://issues.apache.org/jira/browse/HBASE-13165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356169#comment-14356169 ] Misty Stanley-Jones commented on HBASE-13165: - [~saint@gmail.com] I can't get the patch to apply. If you can apply it, can you commit it? Fix docs and scripts for default max heaps size after HBASE-11804 - Key: HBASE-13165 URL: https://issues.apache.org/jira/browse/HBASE-13165 Project: HBase Issue Type: Bug Components: documentation, scripts Affects Versions: 1.0.0 Reporter: Lars George Assignee: Lars George Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0 Attachments: HBASE-13165.patch We changed the behavior of JAVA_HEAP_MAX in hbase-env.sh to be unset and default to the JVM default value. But in quite a few places we still say 1GB is default. Fix the scripts and docs accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13196) Add info about default number of versions when using the export tool
[ https://issues.apache.org/jira/browse/HBASE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-13196: Status: Patch Available (was: Open) Add info about default number of versions when using the export tool Key: HBASE-13196 URL: https://issues.apache.org/jira/browse/HBASE-13196 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-13196.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356324#comment-14356324 ] stack commented on HBASE-13142: --- Linking HBASE-13188, a bug found by Ram [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13195) IntegrationTestImportTsv doesn't support Chaos Monkey options
Dima Spivak created HBASE-13195: --- Summary: IntegrationTestImportTsv doesn't support Chaos Monkey options Key: HBASE-13195 URL: https://issues.apache.org/jira/browse/HBASE-13195 Project: HBase Issue Type: Bug Components: integration tests Reporter: Dima Spivak Assignee: Dima Spivak IntegrationTestImportTsv comes from a pre-monkey era and needs some simple changes to support primative behaviors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356322#comment-14356322 ] stack commented on HBASE-13142: --- Addendum fixes possible leak discovered trying to backport this to 0.98. 0.98 is different -- has MetaEditor etc. -- and I was able to fix one issue where we were leaking ByteBuffers and just ran out of memory but now tests won't start a minicluster master stuck trying to assign a region, RS stuck trying to report for duty. [PERF] Reuse the IPCUtil#buildCellBlock buffer -- Key: HBASE-13142 URL: https://issues.apache.org/jira/browse/HBASE-13142 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: Liang Xie Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, net.png, traces.2.svg, traces.svg Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing. I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg. I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k). Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached). Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Open (was: Patch Available)
Task 6 7 from HBASE-9117, delete all public HTable constructors and delete
ConnectionManager#{delete,get}Connection
--
Key: HBASE-12586
URL: https://issues.apache.org/jira/browse/HBASE-12586
Project: HBase
Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: beginner, beginners
Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch,
HBASE-12586.patch
Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
and 7 from the list here:
https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
To be done in master branch only.
Marked as beginner task. The idea is straight-forward. It is just a lot of
work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355276#comment-14355276 ] Ted Yu commented on HBASE-13168: bq. Did you want me to do the 0.98 patch? I can do it after [~apurtell] gives green light. Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.1.0, 0.98.13 Attachments: 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13181:
Resolution: Fixed
Fix Version/s: 1.1.0
2.0.0
Status: Resolved (was: Patch Available)
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13181-v0.patch, HBASE-13181-v1.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355289#comment-14355289 ] stack commented on HBASE-13191: --- +1 Go for it. mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Attachments: HBASE-13191.1.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13191: Status: Patch Available (was: Reopened) mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 Attachments: HBASE-13191.1.patch.txt, HBASE-13191.not-really-a.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13188) java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
[
https://issues.apache.org/jira/browse/HBASE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355272#comment-14355272
]
Hudson commented on HBASE-13188:
FAILURE: Integrated in HBase-TRUNK #6235 (See
[https://builds.apache.org/job/HBase-TRUNK/6235/])
HBASE-13188 java.lang.ArithmeticException issue in
BoundedByteBufferPool.putBuffer (stack: rev
29104da0139cf05357c001e5ef52190ff04ee561)
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java
java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
--
Key: HBASE-13188
URL: https://issues.apache.org/jira/browse/HBASE-13188
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0, 1.1.0, 0.98.12
Attachments: HBASE-13188.patch
Running a range scan with PE tool with 25 threads getting this error
{code}
java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.hbase.io.BoundedByteBufferPool.putBuffer(BoundedByteBufferPool.java:104)
at org.apache.hadoop.hbase.ipc.RpcServer$Call.done(RpcServer.java:325)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.processResponse(RpcServer.java:1078)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.processAllResponses(RpcServer.java:1103)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doAsyncWrite(RpcServer.java:1036)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:956)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:891)
{code}
I checked in the trunk code also. I think the comment in the code suggests
that the size will not be exact so there is a chance that it could be even 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-13191 started by Sean Busbey. --- mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13191: Attachment: HBASE-13191.1.patch.txt mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Attachments: HBASE-13191.1.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355304#comment-14355304 ] Andrew Purtell commented on HBASE-13168: Sure, +1, thanks guys Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.1.0, 0.98.13 Attachments: 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13167) The check for balancerCutoffTime is buggy
[
https://issues.apache.org/jira/browse/HBASE-13167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355318#comment-14355318
]
Andrew Purtell commented on HBASE-13167:
lgtm if we want to stay with the current behavior. (Sounds reasonable to me.)
The check for balancerCutoffTime is buggy
-
Key: HBASE-13167
URL: https://issues.apache.org/jira/browse/HBASE-13167
Project: HBase
Issue Type: Bug
Components: Balancer
Affects Versions: 0.98.8
Reporter: Tianyin Xu
Attachments: HBASE-13167.patch
In HMaster, the checking logic for balancerCutoffTime in buggy.
See the following code:
{code:title=HMaster.java|borderStyle=solid}
1419 private int getBalancerCutoffTime() {
1420 int balancerCutoffTime =
1421 getConfiguration().getInt(hbase.balancer.max.balancing, -1);
1422 if (balancerCutoffTime == -1) {
1423 // No time period set so create one
1424 int balancerPeriod =
1425 getConfiguration().getInt(hbase.balancer.period, 30);
1426 balancerCutoffTime = balancerPeriod;
1427 // If nonsense period, set it to balancerPeriod
1428 if (balancerCutoffTime = 0) balancerCutoffTime = balancerPeriod;
1429 }
1430 return balancerCutoffTime;
1431 }
{code}
Basically, the check for nonsense period is no effect, because
balancerCutoffTime is always assigned to be balancerPeriod whatever the value
is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13191: Attachment: HBASE-13191.not-really-a.patch.txt dummy patch for checking QA mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 Attachments: HBASE-13191.1.patch.txt, HBASE-13191.not-really-a.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13192) IntegrationTestBulkLoad doesn't wait for table modification sometimes leading to spurious test failures
Devaraj Das created HBASE-13192: --- Summary: IntegrationTestBulkLoad doesn't wait for table modification sometimes leading to spurious test failures Key: HBASE-13192 URL: https://issues.apache.org/jira/browse/HBASE-13192 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Noticed this issue in our internal test runs. It shows up when the test is run with replicas turned on. After the bulk load is complete, the test tries to modify the table with an intention of having the replicas load the new region files. But due to a bug in the way the test was waiting for the modifyTable operation to complete, it would not actually wait. Sometimes a replica wouldn't have loaded the new files yet and it might happen that the 'check' step of the test would go to that replica asking for data. It would give empty results back, and the test would fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13181:
Assignee: Matteo Bertozzi (was: Sean Busbey)
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13181-v0.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355240#comment-14355240
]
Sean Busbey commented on HBASE-13181:
-
+1 lgtm.
mvn site is failing due to HBASE-13191. AFAICT, this patch shouldn't impact
site so it's fine while we're blind.
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13181-v0.patch, HBASE-13181-v1.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355288#comment-14355288
]
Nick Dimiduk commented on HBASE-13103:
--
[~mantonov] yeah, RegionEqualizer sounds like the right idea. Maybe
normalizer? But yes, a background chore invokes something that analyzes
current state and makes efforts to move toward an ideal state.
UI improvements across the board are always welcome and very much encouraged.
We don't get enough love in that department. A (graphical?) representation of
region sizes and distribution would be awesome.
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Brainstorming
Components: Usability
Reporter: Nick Dimiduk
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13191: Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 Attachments: HBASE-13191.1.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355360#comment-14355360 ] Vladimir Rodionov commented on HBASE-11425: --- I am quite skeptical to all this idea ... Here is why: Off heap cache can store blocks in a compressed form. It means that you won't be able to back HFileBlock by a such compressed block - you have to decompress it first. From performance point of view it does not matter whether you do this into direct BB (new approach) or into a byte array-backed BB (existing). Am I missing anything? Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn
[
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355362#comment-14355362
]
Enis Soztutar commented on HBASE-13149:
---
bq. Our compatibility guide currently says we will only cause a breaking
dependency change on major version. The comparison Jerry He did above shows
that Jackson 1.8 - Jackson 1.9 is a breaking dependency change, unless we're
missing something in the jackson library that scopes which packages / classes
should be relied on by downstream clients.
I think the changes are in the internals of Jackson from what I see in the
report for jackson-core. I do not realistically expect us to do a full sweep of
every dependency compat story, so we should rely on what our dependencies use
for versioning. Assuming that jackson (or other deps) uses semver or something
similar going from 1.8 to 1.9 should be fine.
I do not think we should corner ourselves into cannot upgrade dependencies
situation (I think [~ndimiduk] started a thread on this some time ago). From
what I see, inconveniencing our users with not upgrading this lib in 1.1 is far
worse than upgrading it, but possibly causing some minor (if any) inconvenience
in case the user is relying on that lib explicitly.
HBase MR is broken on Hadoop 2.5+ Yarn
--
Key: HBASE-13149
URL: https://issues.apache.org/jira/browse/HBASE-13149
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
Reporter: Jerry He
Priority: Critical
Attachments: HBASE-13149-0.98.patch, HBASE-13149-master.patch,
jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html,
jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html
Running the server MR tools is not working on Yarn version 2.5+.
Running org.apache.hadoop.hbase.mapreduce.Export:
{noformat}
Exception in thread main java.lang.NoSuchMethodError:
org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
at
org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
at
org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
at
org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
{noformat}
The problem seems to be the jackson jar version. HADOOP-10104 updated
jackson version to 1.9.13. YARN-2092 reported a problem as well.
HBase is using jackson 1.8.8. This version of the jar in the classpath seem
to cause the problem.
Should we upgrade to jackson 1.9.13?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13192) IntegrationTestBulkLoad doesn't wait for table modification sometimes leading to spurious test failures
[ https://issues.apache.org/jira/browse/HBASE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-13192: Status: Patch Available (was: Open) IntegrationTestBulkLoad doesn't wait for table modification sometimes leading to spurious test failures --- Key: HBASE-13192 URL: https://issues.apache.org/jira/browse/HBASE-13192 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 13192-1.txt Noticed this issue in our internal test runs. It shows up when the test is run with replicas turned on. After the bulk load is complete, the test tries to modify the table with an intention of having the replicas load the new region files. But due to a bug in the way the test was waiting for the modifyTable operation to complete, it would not actually wait. Sometimes a replica wouldn't have loaded the new files yet and it might happen that the 'check' step of the test would go to that replica asking for data. It would give empty results back, and the test would fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
[
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-13181:
Attachment: HBASE-13181-v1.patch
TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
-
Key: HBASE-13181
URL: https://issues.apache.org/jira/browse/HBASE-13181
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13181-v0.patch, HBASE-13181-v1.patch
Noticing the test failures frequently on internal test rig:
{code}
FAILED:
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Patch Available (was: Open)
Task 6 7 from HBASE-9117, delete all public HTable constructors and delete
ConnectionManager#{delete,get}Connection
--
Key: HBASE-12586
URL: https://issues.apache.org/jira/browse/HBASE-12586
Project: HBase
Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: beginner, beginners
Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch,
HBASE-12586.patch
Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
and 7 from the list here:
https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
To be done in master branch only.
Marked as beginner task. The idea is straight-forward. It is just a lot of
work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-13191: - mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 Attachments: HBASE-13191.1.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13183) Make ZK tickTime configurable in standalone HBase
[ https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355299#comment-14355299 ] Andrew Purtell commented on HBASE-13183: That's correct, the 0.98 build has issues related to HBASE-13143 and the 0.98-on-hadoop1.1 test VM crashed or was killed externally. Make ZK tickTime configurable in standalone HBase - Key: HBASE-13183 URL: https://issues.apache.org/jira/browse/HBASE-13183 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.98.10.1 Reporter: Alex Araujo Assignee: Alex Araujo Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-13183-0.98.patch, HBASE-13183.patch Standalone HBase does not allow the ZooKeeper tickTime to be configured for the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 2000 ms, which is too low for some use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355297#comment-14355297 ] stack commented on HBASE-11425: --- Thanks for the writeup. Makes it easier discussing this new dev. Typical used value for max heap size is 32-48 GB. This ain't right, is it? Usually we have folks hover just below 32G so can do compressed pointers. Each bucket’s size is fixed to 4KB. Should bucket size be same as the hfile block size? Can MBB be developed in isolation with tests and refcounting tests apart from main code base? Is that being done? High-level, general question: So eviction was easy before. When memory pressure just evict until needed memory is made available. The eviction is now made more complicated because have to check for non-zero refcount? And what if can't find necessary memory? What happens? Note that the LRU Cache does not have this block reference counting happening as that does not deal with BBs and deals with the HFileblock objects directly. Why not? We copy from the LRU blocks to Cell arrays? Couldn't Cells go against the LRU blocks directly too? Or I have it wrong? I don't see a downside listing that we'll be doubling the objects made when offheap reading. Is that right? Please note that the Cells in the memstore are still KV based (byte [] backed) ... this is because you are only doing read-path in this JIRA, right? Then again, reading, we have to read from the MemStore so this means that read path can be a mix of onheap and offheap results? On adding new methods to Cell, are there 'holes'? We talked about this in the past and it seemed like there could be strange areas in the Cell API if you did certain calls. If you don't know what I am on about, I'll dig up the old discussion (I think it was on mailing list... Ram you asked for input). ... or maybe the holes have been plugged by 'Using getXXXArray() would throw UnSupportedOperationException. '? And This will make so many short living objects creation also. That is why we decided to go with usage of getXXXOffset() and getXXXLength() API usage also along with buffer based APIs So, you might want to underline this point. Its BB but WE are managing the position and length to save on object creation and to bypass BB range checking, etc. What does that mean for the 'client'? When you give out a BB, its position, etc., is not to be relied upon. That will be disorientating. Pity you couldn't throw unsupportedexception if they tried use position, etc. So you need BB AND the Cell to get at content. BB for the array and then Cell for the offset and length... So, this API is for users on client-side? It is going to confuse them when they have a BB but the position and limit are duds. In client, when would they be doing BB? Never? Client won't be offheaping? If so, could the BB APIs be mixed in to Cell on the server only? So, why have the switch at all? The hasArray switch? Why not BB it all the time? It would simplify the read path. Disadvantage would be it'd be extra objects? When you say this: Note that even if the HFileBlock is on heap BB we do not support getXXXArray() APIs. This is only if hasArray returns false, right? Yeah, looks like 2.0. Tell us more about the unsafe manipulation of BBs? How's that work? Nice writeup. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12931) The existing KeyValues in memstore are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355325#comment-14355325
]
Andrew Purtell commented on HBASE-12931:
Sorry, I missed this, yes this bugfix seems fine to backport to 0.98.
The existing KeyValues in memstore are not removed completely after inserting
cell into memStore
-
Key: HBASE-12931
URL: https://issues.apache.org/jira/browse/HBASE-12931
Project: HBase
Issue Type: Bug
Reporter: ChiaPing Tsai
Assignee: ChiaPing Tsai
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12931.patch
If I'm not wrong, the UPSERT method of memStore should remove all existing
KeyValues except the newer version.
In memStore,
{code:title=DefaultMemStore.java|borderStyle=solid}
int versIionsVisible = 0;
...
if (cur.getTypeByte() == KeyValue.Type.Put.getCode()
cur.getSequenceId() = readpoint) {
if (versionsVisible 1) {
// if we get here we have seen at least one version visible to
the oldest scanner,
// which means we can prove that no scanner will see this version
// false means there was a change, so give us the size.
long delta = heapSizeChange(cur, true);
addedSize -= delta;
this.size.addAndGet(-delta);
it.remove();
setOldestEditTimeToNow();
} else {
versionsVisible++;
}
{code}
Does versionsVisible 1 should be changed to versionsVisible = 1 ?
thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13190) REST RegexStringComparator should support setCharset
Serhiy Bilousov created HBASE-13190:
---
Summary: REST RegexStringComparator should support setCharset
Key: HBASE-13190
URL: https://issues.apache.org/jira/browse/HBASE-13190
Project: HBase
Issue Type: Improvement
Components: REST
Reporter: Serhiy Bilousov
There is no way to set charset (ISO-8859-1 as recommended) to
RegexStringComparator through the REST call what make RegexString filter
unusable for the row key made of arbitrary bytes .
This is major blocker in our business case.
{code:title=ScannerModel.java|borderStyle=solid}
case RegexStringComparator:
comparator = new RegexStringComparator(value);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13191) mvn site fails on jenkins due to permgen
Sean Busbey created HBASE-13191: --- Summary: mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355265#comment-14355265 ] Nick Dimiduk commented on HBASE-13168: -- Thanks [~tedyu] +1 Did you want me to do the 0.98 patch? Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.1.0, 0.98.13 Attachments: 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355353#comment-14355353 ] stack commented on HBASE-12405: --- @zhangduo (and [~octo47]) I made you administrator in JIRA and added you as admin on jenkins. WAL accounting by Store --- Key: HBASE-12405 URL: https://issues.apache.org/jira/browse/HBASE-12405 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 2.0.0, 1.1.0 Reporter: zhangduo Assignee: zhangduo Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12405-addendum.patch, HBASE-12405.patch, HBASE-12405_1.patch, HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch, HBASE-12405_5.patch HBASE-10201 has made flush decisions per Store, but has not done enough work on HLog, so there are two problems: 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. 2. There maybe holes in WAL accounting. For example, assume family A with sequence id 1 and 3, family B with seqId 2. If we flush family A, we can only record that WAL before sequence id 1 can be removed safely. If we do a replay at this point, sequence id 3 will also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355376#comment-14355376 ] Anoop Sam John commented on HBASE-11425: Having block data in compressed form in the BC is optional thing. In such a case, yes, have to decompress first and at that time, it can be to a byte array backed BB. We are not trying to change that. The change is when the data is cached in the non compressed form (But can be in the DBE form). Then avoiding need for copy. The block can be backed by N offheap buckets. Cells are made out of that. And cells are backed by buffers rather than byte[] then Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11425: --- Comment: was deleted (was: Having block data in compressed form in the BC is optional thing. In such a case, yes, have to decompress first and at that time, it can be to a byte array backed BB. We are not trying to change that. The change is when the data is cached in the non compressed form (But can be in the DBE form). Then avoiding need for copy. The block can be backed by N offheap buckets. Cells are made out of that. And cells are backed by buffers rather than byte[] then) Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355372#comment-14355372 ] Anoop Sam John commented on HBASE-11425: Having block data in compressed form in the BC is optional thing. In such a case, yes, have to decompress first and at that time, it can be to a byte array backed BB. We are not trying to change that. The change is when the data is cached in the non compressed form (But can be in the DBE form). Then avoiding need for copy. The block can be backed by N offheap buckets. Cells are made out of that. And cells are backed by buffers rather than byte[] then Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355374#comment-14355374 ] Anoop Sam John commented on HBASE-11425: Having block data in compressed form in the BC is optional thing. In such a case, yes, have to decompress first and at that time, it can be to a byte array backed BB. We are not trying to change that. The change is when the data is cached in the non compressed form (But can be in the DBE form). Then avoiding need for copy. The block can be backed by N offheap buckets. Cells are made out of that. And cells are backed by buffers rather than byte[] then Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Fix Version/s: 1.0.1 Hadoop Flags: Reviewed Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.0.1, 1.1.0, 0.98.13 Attachments: 13168-0.98.patch, 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356138#comment-14356138 ] Jingcheng Du commented on HBASE-11339: -- Hi [~Wilm], if you want to enlarge the KeyValue size in your put, you could try to change the conf(hbase.client.keyvalue.maxsize) used by the HTable, which allows you have a larger KeyValue. As Jon mentioned, LOBs in the current API will cause large memory allocations that will stress the memory systems of both servers and the clients. You need to pay attentions on this. HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user guide_v2.docx, MOB user guide_v3.docx, MOB user guide_v4.docx, MOB user guide_v5.docx, hbase-11339-in-dev.patch, merge-150212.patch, merge.150212b.patch, merge.150212c.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356235#comment-14356235 ] Hudson commented on HBASE-13168: FAILURE: Integrated in HBase-1.0 #795 (See [https://builds.apache.org/job/HBase-1.0/795/]) HBASE-13168 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job (tedyu: rev 89112e84957558f31c161256aa2d7054f165ca02) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.0.1, 1.1.0, 0.98.13 Attachments: 13168-0.98.patch, 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356236#comment-14356236 ] Hudson commented on HBASE-12590: FAILURE: Integrated in HBase-1.0 #795 (See [https://builds.apache.org/job/HBase-1.0/795/]) HBASE-13168 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job (tedyu: rev 89112e84957558f31c161256aa2d7054f165ca02) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java A solution for data skew in HBase-Mapreduce Job --- Key: HBASE-12590 URL: https://issues.apache.org/jira/browse/HBASE-12590 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Weichen Ye Assignee: Weichen Ye Fix For: 2.0.0 Attachments: A Solution for Data Skew in HBase-MapReduce Job (Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job (Version3).pdf, HBASE-12590-v3.patch, HBASE-12590-v4.patch, HBase-12590-v1.patch, HBase-12590-v2.patch 1, Motivation In production environment, data skew is a very common case. A HBase table may contains a lot of small regions and several large regions. Small regions waste a lot of computing resources. If we use a job to scan a table with 3000 small regions, we need a job with 3000 mappers. Large regions always block the job. If in a 100-region table, one region is far large then the other 99 regions. When we run a job with the table as input, 99 mappers will be completed very quickly, and then we need to wait for the last mapper for a long time. 2, Configuration Add three new configuration hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in HBase-MapReduce jobs. The default value is false. hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region size is larger than 3x average region size, treat the region as “proportionately too large”. hbase.table.row.textkey = true means the row key is text. False means binary row key. It is used to find the mid row key in large region. The default value is true. If (region size = average size*ratio) : cut the region into two MR input splits If (average size = region size average size*ratio) : one region as one MR input split If (sum of several continuous regions size average size): combine these regions into one MR input split. Example: In attachment Welcome to the Review Board. https://reviews.apache.org/r/28494/diff/# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13196) Add info about default number of versions when using the export tool
[ https://issues.apache.org/jira/browse/HBASE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-13196: Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Add info about default number of versions when using the export tool Key: HBASE-13196 URL: https://issues.apache.org/jira/browse/HBASE-13196 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 Attachments: HBASE-13196.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13196) Add info about default number of versions when using the export tool
[ https://issues.apache.org/jira/browse/HBASE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-13196: Attachment: HBASE-13196.patch Very small change, needs review. Add info about default number of versions when using the export tool Key: HBASE-13196 URL: https://issues.apache.org/jira/browse/HBASE-13196 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-13196.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356182#comment-14356182 ] ramkrishna.s.vasudevan commented on HBASE-11425: bq.Should bucket size be same as the hfile block size? Yes. that would be better in many cases how ever the odd blocks may go beyond the hfile block size. bq.Can MBB be developed in isolation with tests and refcounting tests apart from main code base? Is that being done? We need some tests for the refcounting part. Apart from that they can be individual tasks as Anoop says. Reg the BB and comparators having two paths, that would be the ideal way as per the profiler reports. That is because for all the KVs that is coming from the HHFiles we have Buffer backed cells. But for the cells in memstore is byte[]. So as mentioned in the doc, if we try to create only BB based rows, families and qualifiers, we may have to do wrapping of these byte[]. That is a costlier operation. Also in cases of creating fake keys it is always better to create fake keys in byte[] rather than in BB because for BB's we have to do some allocation and then copy the contents. All these are costlier. Hence when we create a fake key and compare it against a key from HFile we have two version of cells. One backed by byte[] and another by BB. So it would be better if/else based comparisons. Reg the Unsafe comparators, They are just the same as in byte[] array now. bq.So, you might want to underline this point. Its BB but WE are managing the position and length to save on object creation and to bypass BB range checking, etc. Yes. That is the important decision that we had to make. One objective is to reduce the objects creation and another is to use the same APIs for offset and length. bq.Client won't be offheaping? If so, could the BB APIs be mixed in to Cell on the server only? We discussed on that ServerCell concepts. But I would argue not do that because then the user would have two types of Cells - one on the write path and the other cell on the read path. I would say that would make things more complex and not much ease of use too. I would try to make a trunk based patch and upload for reference. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356251#comment-14356251 ] Hudson commented on HBASE-12590: SUCCESS: Integrated in HBase-1.1 #276 (See [https://builds.apache.org/job/HBase-1.1/276/]) HBASE-13168 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job (tedyu: rev 05aef46d942a0196c6c655ab19a160cd7dc56789) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java A solution for data skew in HBase-Mapreduce Job --- Key: HBASE-12590 URL: https://issues.apache.org/jira/browse/HBASE-12590 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Weichen Ye Assignee: Weichen Ye Fix For: 2.0.0 Attachments: A Solution for Data Skew in HBase-MapReduce Job (Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job (Version3).pdf, HBASE-12590-v3.patch, HBASE-12590-v4.patch, HBase-12590-v1.patch, HBase-12590-v2.patch 1, Motivation In production environment, data skew is a very common case. A HBase table may contains a lot of small regions and several large regions. Small regions waste a lot of computing resources. If we use a job to scan a table with 3000 small regions, we need a job with 3000 mappers. Large regions always block the job. If in a 100-region table, one region is far large then the other 99 regions. When we run a job with the table as input, 99 mappers will be completed very quickly, and then we need to wait for the last mapper for a long time. 2, Configuration Add three new configuration hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in HBase-MapReduce jobs. The default value is false. hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region size is larger than 3x average region size, treat the region as “proportionately too large”. hbase.table.row.textkey = true means the row key is text. False means binary row key. It is used to find the mid row key in large region. The default value is true. If (region size = average size*ratio) : cut the region into two MR input splits If (average size = region size average size*ratio) : one region as one MR input split If (sum of several continuous regions size average size): combine these regions into one MR input split. Example: In attachment Welcome to the Review Board. https://reviews.apache.org/r/28494/diff/# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356250#comment-14356250 ] Hudson commented on HBASE-13168: SUCCESS: Integrated in HBase-1.1 #276 (See [https://builds.apache.org/job/HBase-1.1/276/]) HBASE-13168 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job (tedyu: rev 05aef46d942a0196c6c655ab19a160cd7dc56789) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job -- Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk Assignee: Ted Yu Fix For: 1.0.1, 1.1.0, 0.98.13 Attachments: 13168-0.98.patch, 13168-branch-1.0.patch, 13168-branch-1.patch, 13168-branch-1.patch We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11425: --- Comment: was deleted (was: Adding to what Anoop says, we are not trying to focus on where we do compression. That part remains the same. It is only after we start using an Hfileblock we tend to use it in a offheap mode and particularly avoid the copy that happens in the BucketCache every time a block needs to be used. (In this case it is going to be a decompressed block only). But one point to note here is that in the case of DBE it is an encoded block - and we still go on with the encoded block only and the existing logic of decoding the block still works the same way. In the existing code there are two copies that happen here - one from the BucketCache and other in the DBE algo. Now we try to avoid the first one. ) Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356281#comment-14356281 ] Anoop Sam John commented on HBASE-11425: bq.We discussed on that ServerCell concepts. But I would argue not do that because then the user would have two types of Cells - one on the write path and the other cell on the read path. I would say that would make things more complex and not much ease of use too. No here what I mean is the Cell interface wont change and at client side, the user will still interact with Cell. ServerCell is an extension for Cell which is only at server side. Both in read and write paths. Only thing is that the CP and Filter will get a new type then. ServerCell instead of Cell. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356290#comment-14356290 ] stack commented on HBASE-11425: --- [~ram_krish] On this But I would argue not do that because then the user would have two types of Cells - one on the write path and the other cell on the read path., expecting clients to use BB in a wonky way is not on (smile). I think [~anoop.hbase] cleaned up what is meant. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Open (was: Patch Available)
Task 6 7 from HBASE-9117, delete all public HTable constructors and delete
ConnectionManager#{delete,get}Connection
--
Key: HBASE-12586
URL: https://issues.apache.org/jira/browse/HBASE-12586
Project: HBase
Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: beginner, beginners
Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch,
HBASE-12586.patch
Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
and 7 from the list here:
https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
To be done in master branch only.
Marked as beginner task. The idea is straight-forward. It is just a lot of
work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Patch Available (was: Open)
Task 6 7 from HBASE-9117, delete all public HTable constructors and delete
ConnectionManager#{delete,get}Connection
--
Key: HBASE-12586
URL: https://issues.apache.org/jira/browse/HBASE-12586
Project: HBase
Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: beginner, beginners
Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch,
HBASE-12586.patch
Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
and 7 from the list here:
https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
To be done in master branch only.
Marked as beginner task. The idea is straight-forward. It is just a lot of
work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13191) mvn site fails on jenkins due to permgen
[ https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13191: Attachment: HBASE-13191.not-really-a.patch.txt mvn site fails on jenkins due to permgen Key: HBASE-13191 URL: https://issues.apache.org/jira/browse/HBASE-13191 Project: HBase Issue Type: Bug Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 Attachments: HBASE-13191.1.patch.txt, HBASE-13191.not-really-a.patch.txt, HBASE-13191.not-really-a.patch.txt, HBASE-13191.not-really-a.patch.txt patch QA is failing the mvn site goal due to permgen. looks like we have ~3 places we try to set it. consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356188#comment-14356188
]
Enis Soztutar commented on HBASE-13063:
---
The test succeeds run 20 times.
Allow to turn off memstore replication for region replicas
--
Key: HBASE-13063
URL: https://issues.apache.org/jira/browse/HBASE-13063
Project: HBase
Issue Type: New Feature
Components: regionserver, Replication
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
HBASE-13063-v2.patch, hbase-13063-addendum.patch
HBASE-11568 allows to use replication to send wal edits from the primary to
the replicas.
sometimes the memstore replication is not required, so it will be nice have a
flag to disable it.
the result will be more or less the same to what we have in phase-1, but with
the row-level consistency provided by the flush edit transfered via
replication to refresh the hfiles.
create 't1', 'f', {REGION_REPLICATION = 2, REGION_MEMSTORE_REPLICATION =
false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13191) mvn site fails on jenkins due to permgen
[
https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356316#comment-14356316
]
Hadoop QA commented on HBASE-13191:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703819/HBASE-13191.not-really-a.patch.txt
against master branch at commit 09e9c92d7699b985f45a48242a24af7c848926f0.
ATTACHMENT ID: 12703819
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13169//console
This message is automatically generated.
mvn site fails on jenkins due to permgen
Key: HBASE-13191
URL: https://issues.apache.org/jira/browse/HBASE-13191
Project: HBase
Issue Type: Bug
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 2.0.0
Attachments: HBASE-13191.1.patch.txt,
HBASE-13191.not-really-a.patch.txt, HBASE-13191.not-really-a.patch.txt,
HBASE-13191.not-really-a.patch.txt
patch QA is failing the mvn site goal due to permgen.
looks like we have ~3 places we try to set it. consolidate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13196) Add info about default number of versions when using the export tool
[ https://issues.apache.org/jira/browse/HBASE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356317#comment-14356317 ] Misty Stanley-Jones commented on HBASE-13196: - Good catch, fixed and pushed. Add info about default number of versions when using the export tool Key: HBASE-13196 URL: https://issues.apache.org/jira/browse/HBASE-13196 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-13196.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7126) Update website with info on how to report security bugs
[ https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356314#comment-14356314 ] Sean Busbey commented on HBASE-7126: +1 LGTM Update website with info on how to report security bugs Key: HBASE-7126 URL: https://issues.apache.org/jira/browse/HBASE-7126 Project: HBase Issue Type: Task Components: documentation Reporter: Eli Collins Assignee: Misty Stanley-Jones Priority: Critical Labels: website Attachments: HBASE-7126-v1.patch, HBASE-7126.patch The HBase website should be updated with information on how to report potential security vulnerabilities. In Hadoop land we have a private security list that anyone case post to that we point to on our list page: Hadoop example http://hadoop.apache.org/general_lists.html#Security. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13196) Add info about default number of versions when using the export tool
Misty Stanley-Jones created HBASE-13196: --- Summary: Add info about default number of versions when using the export tool Key: HBASE-13196 URL: https://issues.apache.org/jira/browse/HBASE-13196 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7126) Update website with info on how to report security bugs
[ https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-7126: --- Status: Patch Available (was: Open) Update website with info on how to report security bugs Key: HBASE-7126 URL: https://issues.apache.org/jira/browse/HBASE-7126 Project: HBase Issue Type: Task Components: documentation Reporter: Eli Collins Assignee: Misty Stanley-Jones Priority: Critical Labels: website Attachments: HBASE-7126-v1.patch, HBASE-7126.patch The HBase website should be updated with information on how to report potential security vulnerabilities. In Hadoop land we have a private security list that anyone case post to that we point to on our list page: Hadoop example http://hadoop.apache.org/general_lists.html#Security. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7126) Update website with info on how to report security bugs
[
https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356338#comment-14356338
]
Hadoop QA commented on HBASE-7126:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703823/HBASE-7126-v1.patch
against master branch at commit 09e9c92d7699b985f45a48242a24af7c848926f0.
ATTACHMENT ID: 12703823
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+Please use link:https://issues.apache.org/jira/browse/hbase[JIRA] to
report non-security-related bugs.
+To protect existing HBase installations from new vulnerabilities, please *do
not* use JIRA to report security-related bugs. Instead, send your report to the
mailing list priv...@apache.org, which allows anyone to send messages, but
restricts who can read them. Someone on that list will contact you to follow up
on your report.
+NOTE: To protect existing HBase installations from exploitation, please *do
not* use JIRA to report security-related bugs. Instead, send your report to the
mailing list priv...@apache.org, which allows anyone to send messages, but
restricts who can read them. Someone on that list will contact you to follow up
on your report.
+HBase adheres to the Apache Software Foundation's policy on reported
vulnerabilities, available at http://apache.org/security/.
+If you wish to send an encrypted report, you can use the GPG details provided
for the general ASF security list. This will likely increase the response time
to your report.
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13172//console
This message is automatically generated.
Update website with info on how to report security bugs
Key: HBASE-7126
URL:
[jira] [Created] (HBASE-13194) TableNamespaceManager not ready cause MasterQuotaManager initialization fail
zhangduo created HBASE-13194:
Summary: TableNamespaceManager not ready cause MasterQuotaManager
initialization fail
Key: HBASE-13194
URL: https://issues.apache.org/jira/browse/HBASE-13194
Project: HBase
Issue Type: Bug
Components: master
Reporter: zhangduo
This cause TestNamespaceAuditor to fail.
https://builds.apache.org/job/HBase-TRUNK/6237/testReport/junit/org.apache.hadoop.hbase.namespace/TestNamespaceAuditor/testRegionOperations/
{noformat}
2015-03-10 22:42:01,372 ERROR [hemera:48616.activeMasterManager]
namespace.NamespaceStateManager(204): Error while update namespace state.
java.io.IOException: Table Namespace Manager not ready yet, try again later
at
org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1912)
at
org.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:2131)
at
org.apache.hadoop.hbase.namespace.NamespaceStateManager.initialize(NamespaceStateManager.java:188)
at
org.apache.hadoop.hbase.namespace.NamespaceStateManager.start(NamespaceStateManager.java:63)
at
org.apache.hadoop.hbase.namespace.NamespaceAuditor.start(NamespaceAuditor.java:57)
at
org.apache.hadoop.hbase.quotas.MasterQuotaManager.start(MasterQuotaManager.java:88)
at
org.apache.hadoop.hbase.master.HMaster.initQuotaManager(HMaster.java:902)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:756)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:161)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1455)
at java.lang.Thread.run(Thread.java:744)
{noformat}
The direct reason is that we do not have a retry here, if init fails then it
always fails. But I skimmed the code, seems there is no async init operations
when calling finishActiveMasterInitialization, so it is very strange. Need to
dig more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13196) Add info about default number of versions when using the export tool
[
https://issues.apache.org/jira/browse/HBASE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356342#comment-14356342
]
Hadoop QA commented on HBASE-13196:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703822/HBASE-13196.patch
against master branch at commit 09e9c92d7699b985f45a48242a24af7c848926f0.
ATTACHMENT ID: 12703822
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13171//console
This message is automatically generated.
Add info about default number of versions when using the export tool
Key: HBASE-13196
URL: https://issues.apache.org/jira/browse/HBASE-13196
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Fix For: 2.0.0
Attachments: HBASE-13196.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13190) REST RegexStringComparator should support setCharset
[
https://issues.apache.org/jira/browse/HBASE-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serhiy Bilousov updated HBASE-13190:
Priority: Blocker (was: Major)
REST RegexStringComparator should support setCharset
Key: HBASE-13190
URL: https://issues.apache.org/jira/browse/HBASE-13190
Project: HBase
Issue Type: Improvement
Components: REST
Reporter: Serhiy Bilousov
Priority: Blocker
There is no way to set charset (ISO-8859-1 as recommended) to
RegexStringComparator through the REST call what make RegexString filter
unusable for the row key made of arbitrary bytes .
This is major blocker in our business case.
{code:title=ScannerModel.java|borderStyle=solid}
case RegexStringComparator:
comparator = new RegexStringComparator(value);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13188) java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
[
https://issues.apache.org/jira/browse/HBASE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355098#comment-14355098
]
stack commented on HBASE-13188:
---
Leaving open because trying to get the bigger patch into 0.98 and will need to
include this there too.
java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
--
Key: HBASE-13188
URL: https://issues.apache.org/jira/browse/HBASE-13188
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0, 1.1.0, 0.98.12
Attachments: HBASE-13188.patch
Running a range scan with PE tool with 25 threads getting this error
{code}
java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.hbase.io.BoundedByteBufferPool.putBuffer(BoundedByteBufferPool.java:104)
at org.apache.hadoop.hbase.ipc.RpcServer$Call.done(RpcServer.java:325)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.processResponse(RpcServer.java:1078)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.processAllResponses(RpcServer.java:1103)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doAsyncWrite(RpcServer.java:1036)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:956)
at
org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:891)
{code}
I checked in the trunk code also. I think the comment in the code suggests
that the size will not be exact so there is a chance that it could be even 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13191) mvn site fails on jenkins due to permgen
[
https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355593#comment-14355593
]
Hadoop QA commented on HBASE-13191:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703699/HBASE-13191.not-really-a.patch.txt
against master branch at commit ed93ddd94f6264ca246477bece4bf2c895706a22.
ATTACHMENT ID: 12703699
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13157//console
This message is automatically generated.
mvn site fails on jenkins due to permgen
Key: HBASE-13191
URL: https://issues.apache.org/jira/browse/HBASE-13191
Project: HBase
Issue Type: Bug
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 2.0.0
Attachments: HBASE-13191.1.patch.txt,
HBASE-13191.not-really-a.patch.txt
patch QA is failing the mvn site goal due to permgen.
looks like we have ~3 places we try to set it. consolidate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355648#comment-14355648 ] Eshcar Hillel commented on HBASE-13071: --- I think the best way to test this patch is to use the extended version of YCSB which supports measuring multi-step operations like scans (see the link to the code - I added the code in a separate branch). The attached evaluation file describes the settings I used in my test. Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Attachments: 99.eshcar.png, HBASE-13071_98_1.patch, HBASE-13071_trunk_1.patch, HBASE-13071_trunk_2.patch, HBASE-13071_trunk_3.patch, HBASE-13071_trunk_4.patch, HBASE-13071_trunk_5.patch, HBASE-13071_trunk_6.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, gc.eshcar.png, hits.eshcar.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13192) IntegrationTestBulkLoad doesn't wait for table modification sometimes leading to spurious test failures
[
https://issues.apache.org/jira/browse/HBASE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355649#comment-14355649
]
Hadoop QA commented on HBASE-13192:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703712/13192-2.txt
against master branch at commit ed93ddd94f6264ca246477bece4bf2c895706a22.
ATTACHMENT ID: 12703712
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13161//console
This message is automatically generated.
IntegrationTestBulkLoad doesn't wait for table modification sometimes leading
to spurious test failures
---
Key: HBASE-13192
URL: https://issues.apache.org/jira/browse/HBASE-13192
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Assignee: Devaraj Das
Attachments: 13192-1.txt, 13192-2.txt
Noticed this issue in our internal test runs. It shows up when the test is
run with replicas turned on. After the bulk load is complete, the test tries
to modify the table with an intention of having the replicas load the new
region files. But due to a bug in the way the test was waiting for the
modifyTable operation to complete, it would not actually wait. Sometimes a
replica wouldn't have loaded the new files yet and it might happen that the
'check' step of the test would go to that replica asking for data. It would
give empty results back, and the test would fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13191) mvn site fails on jenkins due to permgen
[
https://issues.apache.org/jira/browse/HBASE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355677#comment-14355677
]
Sean Busbey commented on HBASE-13191:
-
The test run has this curious output:
{code}
Maven options
\tMAVEN_OPTS=-Xmx6100M -XX:-UsePerfData -XX:MaxPermSize=256m
{code}
which shows that we're getting the jenkins job provided variable (the one in
jenkinsEnv uses ~3G of memory), but we're getting it with an extra set of
quotes. I'm going to try tweaking that to remove them.
mvn site fails on jenkins due to permgen
Key: HBASE-13191
URL: https://issues.apache.org/jira/browse/HBASE-13191
Project: HBase
Issue Type: Bug
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 2.0.0
Attachments: HBASE-13191.1.patch.txt,
HBASE-13191.not-really-a.patch.txt
patch QA is failing the mvn site goal due to permgen.
looks like we have ~3 places we try to set it. consolidate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13167) The check for balancerCutoffTime is buggy
[
https://issues.apache.org/jira/browse/HBASE-13167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355715#comment-14355715
]
Mikhail Antonov commented on HBASE-13167:
-
Thanks [~apurtell]. It might be more conservative to limit max cutoff time to
about 0.75 of period time, but current behavior sounds reasonable.
The check for balancerCutoffTime is buggy
-
Key: HBASE-13167
URL: https://issues.apache.org/jira/browse/HBASE-13167
Project: HBase
Issue Type: Bug
Components: Balancer
Affects Versions: 0.98.8
Reporter: Tianyin Xu
Attachments: HBASE-13167.patch
In HMaster, the checking logic for balancerCutoffTime in buggy.
See the following code:
{code:title=HMaster.java|borderStyle=solid}
1419 private int getBalancerCutoffTime() {
1420 int balancerCutoffTime =
1421 getConfiguration().getInt(hbase.balancer.max.balancing, -1);
1422 if (balancerCutoffTime == -1) {
1423 // No time period set so create one
1424 int balancerPeriod =
1425 getConfiguration().getInt(hbase.balancer.period, 30);
1426 balancerCutoffTime = balancerPeriod;
1427 // If nonsense period, set it to balancerPeriod
1428 if (balancerCutoffTime = 0) balancerCutoffTime = balancerPeriod;
1429 }
1430 return balancerCutoffTime;
1431 }
{code}
Basically, the check for nonsense period is no effect, because
balancerCutoffTime is always assigned to be balancerPeriod whatever the value
is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
