[jira] [Assigned] (HBASE-15470) Add a setting for Priority queue length
[ https://issues.apache.org/jira/browse/HBASE-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HBASE-15470: - Assignee: Elliott Clark > Add a setting for Priority queue length > --- > > Key: HBASE-15470 > URL: https://issues.apache.org/jira/browse/HBASE-15470 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > > Meta can have very different request rates than any other region. We > shouldn't use the same call queue length. > For example: > If a normal request takes 100ms ( long scan or large get ) > but a call to meta is all in memory so it takes < 1ms > So a call queue length of 100 represents multiple seconds of work for normal > requests, but less than a second for meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15405) Fix PE logging and wrong defaults in help message
[
https://issues.apache.org/jira/browse/HBASE-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-15405:
-
Attachment: HBASE-15405-master-v4.patch
> Fix PE logging and wrong defaults in help message
> -
>
> Key: HBASE-15405
> URL: https://issues.apache.org/jira/browse/HBASE-15405
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-15405-master-v2.patch,
> HBASE-15405-master-v3.patch, HBASE-15405-master-v4.patch,
> HBASE-15405-master.patch, HBASE-15405-master.patch
>
>
> Corrects wrong default values for few options in the help message.
> Final stats from multiple clients are intermingled making it hard to
> understand. Also the logged stats aren't very machine readable. It can be
> helpful in a daily perf testing rig which scraps logs for results.
> Example of logs before the change.
> {noformat}
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.98, min=359.00, max=324050.00, stdDev=851.82, 95th=1368.00,
> 99th=1625.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.92, min=356.00, max=323394.00, stdDev=817.55, 95th=1370.00,
> 99th=1618.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.98, min=367.00, max=322745.00, stdDev=840.43, 95th=1369.00,
> 99th=1622.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 375.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 363.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.6624126434326
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.4124526977539
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 781.3929776087633
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 742.8027916717297
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1070.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1071.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th =
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th =
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th =
> 1623.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th =
> 1624.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 372.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th =
> 3013.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.2451229095459
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th =
> 3043.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 725.4744472152282
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th =
> 25282.38016755
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 895.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th =
> 25812.76334
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1071.0
> 16/03/05 22:43:06 INFO
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199117#comment-15199117 ] Elliott Clark commented on HBASE-15390: --- Can we get a comment on why that if is needed? +1 with the comment added. > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390-addendum.patch, > HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15472) replication_admin_test creates a table it doesn't use
[ https://issues.apache.org/jira/browse/HBASE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-15472: Attachment: HBASE-15472.patch > replication_admin_test creates a table it doesn't use > - > > Key: HBASE-15472 > URL: https://issues.apache.org/jira/browse/HBASE-15472 > Project: HBase > Issue Type: Bug > Components: Replication, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > Attachments: HBASE-15472.patch > > > I noticed while adding tests to replication_admin_test.rb for HBASE-12940 > that it creates an HBase table "hbase_shell_tests_table" that is never used > in any of the suite's tests. Removing the table creation statements speeds up > the suite locally from 1min 10s to 2s. > Note that until HBASE-14562 is worked, this test suite doesn't run as part of > the automatic test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated HBASE-15334: --- Description: Avro is a popular format for hbase storage. User may want the support natively in the connector. With the support, user can save serialized avro into hbase table, and then query on top of it using spark sql. This is one way of support complex data types automatically supported. Otherwise, user has to define their own customized Serdes to support complex data types. was:Avro is a popular format for hbase storage. User may want the support natively in the connector. > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. > With the support, user can save serialized avro into hbase table, and then > query on top of it using spark sql. This is one way of support complex data > types automatically supported. Otherwise, user has to define their own > customized Serdes to support complex data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202463#comment-15202463 ] Hudson commented on HBASE-15302: SUCCESS: Integrated in HBase-1.3-IT #564 (See [https://builds.apache.org/job/HBase-1.3-IT/564/]) HBASE-15302 Reenable the other tests disabled by HBASE-14678 (Phil Yang) (tedyu: rev d942d4e5a1fe5644a51b5775f3a3e27dd219ad1d) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15481: Attachment: HBASE-15481-v0.patch > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15228) Add the methods to RegionObserver to trigger start/complete restoring WALs
[
https://issues.apache.org/jira/browse/HBASE-15228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202581#comment-15202581
]
Hadoop QA commented on HBASE-15228:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 4s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 6m 22s
{color} | {color:red} hbase-server: patch generated 1 new + 481 unchanged - 1
fixed = 482 total (was 482) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
47m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 189m 13s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 276m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.master.TestTableLockManager |
| | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
|
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12788382/HBASE-15228-v1.patch |
| JIRA Issue | HBASE-15228 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf911.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3a6d683 |
| Default Java | 1.7.0_79 |
| Multi-JDK
[jira] [Commented] (HBASE-15300) Upgrade to zookeeper 3.4.8
[
https://issues.apache.org/jira/browse/HBASE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199412#comment-15199412
]
Hadoop QA commented on HBASE-15300:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 37s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 51s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
0s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 20s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 30s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 13m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patch modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 48s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 19s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 20s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 18s
{color} | {color:red} root in the patch failed. {color} |
|
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198276#comment-15198276
]
Hudson commented on HBASE-15430:
SUCCESS: Integrated in HBase-1.3-IT #556 (See
[https://builds.apache.org/job/HBase-1.3-IT/556/])
HBASE-15430 Failed taking snapshot - Manifest proto-message too large
(matteo.bertozzi: rev 103c313b0d0fe4a40d33bb4a81b75ba66a6c6f3e)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotManifest.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Assignee: JunHo Cho
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18
>
> Attachments: HBASE-15430-v4.patch, HBASE-15430-v5.patch,
> HBASE-15430-v6.patch, hbase-15430-v1.patch, hbase-15430-v2.patch,
> hbase-15430-v3.branch.0.98.patch, hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
>
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200897#comment-15200897 ] Duo Zhang commented on HBASE-15389: --- [~sershe] Any concerns on the modification of stripe compaction related codes? Thanks. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-uc.patch, > HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch, > HBASE-15389-v12.patch, HBASE-15389-v2.patch, HBASE-15389-v3.patch, > HBASE-15389-v4.patch, HBASE-15389-v5.patch, HBASE-15389-v6.patch, > HBASE-15389-v7.patch, HBASE-15389-v8.patch, HBASE-15389-v9.patch, > HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15360: -- Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15464) Flush / Compaction metrics revisited
[ https://issues.apache.org/jira/browse/HBASE-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199127#comment-15199127 ] Elliott Clark commented on HBASE-15464: --- * Need to fix the TestHeapSize * Why the changes around throughput controller? TestStripeStoreEngine etc. * Dead store in find bugs seems legit. * Everything else looks good. > Flush / Compaction metrics revisited > > > Key: HBASE-15464 > URL: https://issues.apache.org/jira/browse/HBASE-15464 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: hbase-15464_v1.patch, hbase-15464_v2.patch > > > We can add a couple of metrics related to flushes and compactions: > - flush memstore and output file size histogram: This will allow seeing > whether we are flushing too early due to memory pressure, too many regions, > etc. Tracking flush memstore size vs output file size is useful in > understanding the block encoding compression benefits. > - total flushed output bytes: This will allow to monitor the IO / throughput > from flushers. You can use this to set num flushers, flush throttle, etc. > - smallCompactionQueueLength / large...: This is tracked, but not emitted > anymore due to a bug. > - compaction time histogram: similar to flush time histogram, how long > compactions are taking. > - compaction input num files / output num files histogram: How many files on > average we are compacting. Stripe compaction / date tiered compaction can use > the num output files metric. > - compaction input / output data sizes histogram: How much data on average > we are compacting. > - compaction input / output total bytes: Measure compaction IO / throughput. > measure write amplification, enables to set compaction throttle. > - Breakdown for above for major compactions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15464) Flush / Compaction metrics revisited
[ https://issues.apache.org/jira/browse/HBASE-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15464: -- Attachment: hbase-15464_v2.patch Rebased. Any interest in a review? > Flush / Compaction metrics revisited > > > Key: HBASE-15464 > URL: https://issues.apache.org/jira/browse/HBASE-15464 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: hbase-15464_v1.patch, hbase-15464_v2.patch > > > We can add a couple of metrics related to flushes and compactions: > - flush memstore and output file size histogram: This will allow seeing > whether we are flushing too early due to memory pressure, too many regions, > etc. Tracking flush memstore size vs output file size is useful in > understanding the block encoding compression benefits. > - total flushed output bytes: This will allow to monitor the IO / throughput > from flushers. You can use this to set num flushers, flush throttle, etc. > - smallCompactionQueueLength / large...: This is tracked, but not emitted > anymore due to a bug. > - compaction time histogram: similar to flush time histogram, how long > compactions are taking. > - compaction input num files / output num files histogram: How many files on > average we are compacting. Stripe compaction / date tiered compaction can use > the num output files metric. > - compaction input / output data sizes histogram: How much data on average > we are compacting. > - compaction input / output total bytes: Measure compaction IO / throughput. > measure write amplification, enables to set compaction throttle. > - Breakdown for above for major compactions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197228#comment-15197228
]
Hadoop QA commented on HBASE-14703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 15s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
46s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 39s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 42s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 33s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 42s {color} |
{color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 33s {color} |
{color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 31s {color} |
{color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 29s {color} |
{color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 31s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 17 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
|
[jira] [Updated] (HBASE-14985) TimeRange constructors should set allTime when appropriate
[ https://issues.apache.org/jira/browse/HBASE-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-14985: Attachment: HBASE-14985-v1.patch Revision of the patch that should apply cleanly this time, and incorporates feedback about where to place the unit test. > TimeRange constructors should set allTime when appropriate > -- > > Key: HBASE-14985 > URL: https://issues.apache.org/jira/browse/HBASE-14985 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.1.2, 0.98.16.1 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > Attachments: HBASE-14985-v1.patch, HBASE-14985.patch > > > The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE > and sets an allTime flag to true. This flag allows some performance > optimizations when comparing or using TimeRanges. > This flag is not set, however, if you call "new TimeRange(0L)" or "new > TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically > equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and > detecting this condition is trivial, we should set the flag automatically in > the explicit constructors when appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198826#comment-15198826 ] Hudson commented on HBASE-15390: SUCCESS: Integrated in HBase-1.3-IT #557 (See [https://builds.apache.org/job/HBase-1.3-IT/557/]) HBASE-15390 Unnecessary MetaCache evictions cause elevated number of (antonov: rev 94d576025fd158dae03dd6a6b50a62483b0f04e0) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndPut() and checkAndDelete()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197849#comment-15197849 ] Esther Kundin commented on HBASE-8458: -- I actually have the same problem and attempted to fix with a coprocessor. An observer coprocessor is unable to do the appropriate locking if writes are happening from different clients. You cannot run a checkAndPut from within the coprocessor as by the time you are in the preBatchMutate, the row is locked for writing and a checkAndPut will be unable to run on the same row. If you implement it as an endpoint observer, then it should work, though I haven't tried it, but it will do a read/write lock on the row, which is a stricter locking than checkAndPut would do, so it would hurt performance more than a batch checkAndPut which will not block reads for as long. So it would seem that a batch checkAndPut would be the best solution. > Support for batch version of checkAndPut() and checkAndDelete() > --- > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: Improvement > Components: Client, regionserver >Affects Versions: 0.95.0 >Reporter: Hari Mankude > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndPut() instead of a put(). However, doing a checkAndPut() rpc for > every key update is non optimal. It would be good to have a batch version of > checkAndPut() similar to batch put(). The client can partition the keys on > region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndPut() > Batch version of checkAndDelete() is a similar requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202132#comment-15202132 ] Anastasia Braginsky commented on HBASE-14921: - [~anoop.hbase], [~stack], can you please clarify what are the use cases where the chunkPool is null in MSLAB? > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14983) Create metrics for per block type hit/miss ratios
[
https://issues.apache.org/jira/browse/HBASE-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198762#comment-15198762
]
Hadoop QA commented on HBASE-14983:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 31s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 5m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 19m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 22s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 56s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 25s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
53m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 14m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 11s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 29s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197742#comment-15197742
]
Francis Liu commented on HBASE-6721:
{quote}
I'd say either of the two you'd prefer to start is good.
{quote}
Thanks [~busbey], I started the discuss thread. Also, I've moved the jiras.
{quote}
BTW, if there were a backport to branch-1 we'd probably try it. (Or if there
were not a backport at the time, I might do it.)
{quote}
Thanks! BTW I think [~enis] might already have a backport as he created one
before.
> RegionServer Group based Assignment
> ---
>
> Key: HBASE-6721
> URL: https://issues.apache.org/jira/browse/HBASE-6721
> Project: HBase
> Issue Type: New Feature
> Components: regionserver
>Reporter: Francis Liu
>Assignee: Francis Liu
> Labels: hbase-6721
> Fix For: 2.0.0
>
> Attachments: 6721-master-webUI.patch, HBASE-6721
> GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf,
> HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf,
> HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch,
> HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch,
> HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch,
> HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch,
> HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch,
> HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch,
> HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch,
> HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch,
> HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch,
> HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg,
> hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch,
> hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch,
> hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch,
> hbase-6721-v25.patch, hbase-6721-v26.patch, hbase-6721-v26_draft1.patch,
> hbase-6721-v27.patch, hbase-6721-v27.patch, hbase-6721-v27.patch.txt,
> hbase-6721-v28.patch, hbase-6721-v28.patch, hbase-6721-v29.patch,
> immediateAssignments Sequence Diagram.svg, randomAssignment Sequence
> Diagram.svg, retainAssignment Sequence Diagram.svg, roundRobinAssignment
> Sequence Diagram.svg
>
>
> In multi-tenant deployments of HBase, it is likely that a RegionServer will
> be serving out regions from a number of different tables owned by various
> client applications. Being able to group a subset of running RegionServers
> and assign specific tables to it, provides a client application a level of
> isolation and resource allocation.
> The proposal essentially is to have an AssignmentManager which is aware of
> RegionServer groups and assigns tables to region servers based on groupings.
> Load balancing will occur on a per group basis as well.
> This is essentially a simplification of the approach taken in HBASE-4120. See
> attached document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15353) Add metric for number of CallQueueTooBigException's
[ https://issues.apache.org/jira/browse/HBASE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15353: -- Priority: Minor (was: Major) > Add metric for number of CallQueueTooBigException's > --- > > Key: HBASE-15353 > URL: https://issues.apache.org/jira/browse/HBASE-15353 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, metrics >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Priority: Minor > Labels: easy, noob > > This exception is being thrown more. We should add a metric for this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15083: --- Attachment: HBASE-15083-0.98.patch > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.18 > > Attachments: HBASE-15083-0.98.patch, HBASE-15083-branch-1.patch, > HBASE-15083.patch, HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198682#comment-15198682
]
Hudson commented on HBASE-15430:
SUCCESS: Integrated in HBase-1.1-JDK7 #1682 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1682/])
HBASE-15430 Failed taking snapshot - Manifest proto-message too large
(matteo.bertozzi: rev 1aafc240257833541a119eccf9067e7d054b7980)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotManifest.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Assignee: JunHo Cho
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18
>
> Attachments: HBASE-15430-v4.patch, HBASE-15430-v5.patch,
> HBASE-15430-v6.patch, hbase-15430-v1.patch, hbase-15430-v2.patch,
> hbase-15430-v3.branch.0.98.patch, hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
>
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199514#comment-15199514
]
Hadoop QA commented on HBASE-14703:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 58s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
10s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
16s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 28s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 28s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 58s
{color} | {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new + 6
unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 58s
{color} | {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new + 6
unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 50s
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6
new + 6 unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 50s
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6
new + 6 unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s {color} | {color:green} hbase-protocol: patch generated 0 new + 0 unchanged
- 83 fixed = 0 total (was 83) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} hbase-protocol: patch generated 0 new + 0 unchanged
- 83 fixed = 0 total (was 83) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} hbase-client: patch generated 0 new + 0 unchanged -
83 fixed = 0 total (was 83) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} hbase-client: patch generated 0 new + 0 unchanged -
83 fixed = 0 total (was 83) {color} |
| {color:green}+1{color} | {color:green} checkstyle
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198858#comment-15198858
]
deepankar commented on HBASE-15064:
---
No I was saying, when you have done mbb1.get() for 4 times and then you check
hasRemaining it will return false;
so I think the following test will fail
{code}
bb1 = ByteBuffer.wrap(b);
bb2 = ByteBuffer.wrap(b1);
bb3 = ByteBuffer.allocate(4);
mbb1 = new MultiByteBuff(bb1, bb2, bb3);
mbb1.limit(12);
for(int i = 0; i < 12; i++) {
assertTrue(mbb1.hasRemaining());
mbb1.get();
}
assertFalse(mbb1.hasRemaining());
{code}
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch, MBB_hasRemaining.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198807#comment-15198807 ] Hudson commented on HBASE-15390: FAILURE: Integrated in HBase-1.4 #24 (See [https://builds.apache.org/job/HBase-1.4/24/]) HBASE-15390 Unnecessary MetaCache evictions cause elevated number of (antonov: rev 94d576025fd158dae03dd6a6b50a62483b0f04e0) * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetricsConnection.java * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199139#comment-15199139 ] Mikhail Antonov commented on HBASE-15390: - Pushed addendum to master, branch-1, branch-1.3. This should fixed the tests broken by this commit like TestCorruptedRegionStoreFile (which is a LargeTest, so got caught late) > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390-addendum.patch, > HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200994#comment-15200994 ] Ted Yu commented on HBASE-15482: Welcome back, Liyin. Yeah. This makes sense. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15441) Fix WAL splitting when region has moved multiple times
[
https://issues.apache.org/jira/browse/HBASE-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198809#comment-15198809
]
Hadoop QA commented on HBASE-15441:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 33s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 17m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 22s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 16m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
75m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 19m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 30s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 42s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 43s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 6s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 5m
11s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 436m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.ipc.TestSimpleRpcScheduler |
| | hadoop.hbase.ipc.TestSimpleRpcScheduler |
| Timed out junit tests |
[jira] [Commented] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197442#comment-15197442 ] stack commented on HBASE-15360: --- bq. I'm already on the bed and using a mobile phone to type these words... Smile. Will do. Thanks [~Apache9] > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15064:
---
Attachment: MBB_hasRemaining.patch
Patch for reference.
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch, MBB_hasRemaining.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15479) No more garbage or beware of autoboxing
[ https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15479: -- Status: Open (was: Patch Available) > No more garbage or beware of autoboxing > --- > > Key: HBASE-15479 > URL: https://issues.apache.org/jira/browse/HBASE-15479 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch > > > Quick journey with JMC in a profile mode revealed very interesting and > unexpected heap polluter on a client side. Patch will shortly follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15456) CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor
[
https://issues.apache.org/jira/browse/HBASE-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198196#comment-15198196
]
huaxiang sun commented on HBASE-15456:
--
For branch-1 patch, the new change is to address the unittest failures listed
as follows, due to open a region without any column family.
org.apache.hadoop.hbase.master.TestOpenedRegionHandler.testShouldNotCompeleteOpenedRegionSuccessfullyIfVersionMismatches
org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler.testRegionServerAbortionDueToFailureTransitioningToOpened
org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler.testYankingRegionFromUnderIt
org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
> CreateTableProcedure/ModifyTableProcedure needs to fail when there is no
> family in table descriptor
> ---
>
> Key: HBASE-15456
> URL: https://issues.apache.org/jira/browse/HBASE-15456
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15456-001_branch-1.patch, HBASE-15456-v001.patch,
> HBASE-15456-v002.patch, HBASE-15456-v002.patch, HBASE-15456-v003.patch
>
>
> If there is only one family in the table, DeleteColumnFamilyProcedure will
> fail.
> Currently, when hbase.table.sanity.checks is set to false, hbase master logs
> a warning and CreateTableProcedure/ModifyTableProcedure will succeed.
> This behavior is not consistent with DeleteColumnFamilyProcedure's.
> Another point, before HBASE-13145, PeriodicMemstoreFlusher will run into the
> following exception. lastStoreFlushTimeMap is populated for families, if
> there is no family in the table, there is no entry in lastStoreFlushTimeMap.
> {code}
> 16/02/01 11:14:26 ERROR regionserver.HRegionServer$PeriodicMemstoreFlusher:
> Caught exception
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentHashMap$HashIterator.nextEntry(ConcurrentHashMap.java:1354)
>
> at
> java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:1384)
>
> at java.util.Collections.min(Collections.java:628)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getEarliestFlushTimeForAllStores(HRegion.java:1572)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.shouldFlush(HRegion.java:1904)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemstoreFlusher.chore(HRegionServer.java:1509)
>
> at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198821#comment-15198821
]
deepankar commented on HBASE-15064:
---
I think the test in your patch should fail Mbb.limit(12); because once your
cross the 4th byte the hasRemaining will start returning false as you are
checking only the limit index. I think just [~anoop.hbase] 's suggestion will
not work, that coupled with the modification of the way we calculate
limitedItemIndex should work I think
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch, MBB_hasRemaining.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15382) Expose regionserver metadata (ie groups, tables, servers) via JMX
[ https://issues.apache.org/jira/browse/HBASE-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-15382: Labels: hbase-6721 (was: ) > Expose regionserver metadata (ie groups, tables, servers) via JMX > - > > Key: HBASE-15382 > URL: https://issues.apache.org/jira/browse/HBASE-15382 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > > This feature was removed from the base patch. So we can come up with a proper > interface for other components to use as well, as directly accessing jmx is > not an option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13506) AES-GCM cipher support where available
[ https://issues.apache.org/jira/browse/HBASE-13506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13506: --- Fix Version/s: (was: 0.98.18) 0.98.19 > AES-GCM cipher support where available > -- > > Key: HBASE-13506 > URL: https://issues.apache.org/jira/browse/HBASE-13506 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.3.0, 0.98.19 > > > The initial encryption drop only had AES-CTR support because authenticated > modes such as GCM are only available in Java 7 and up, and our trunk at the > time was targeted at Java 6. However we can optionally use AES-GCM cipher > support where available. For HBase 1.0 and up, Java 7 is now the minimum so > use of AES-GCM can go in directly. It's probably possible to add support in > 0.98 too using reflection for cipher object initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198751#comment-15198751
]
deepankar commented on HBASE-15064:
---
yeah I think that will work, but what about the other bug in calculating
limitedItemIndex
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15323) Hbase Rest CheckAndDeleteAPi should be able to delete more cells
[ https://issues.apache.org/jira/browse/HBASE-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197221#comment-15197221 ] Hudson commented on HBASE-15323: ABORTED: Integrated in HBase-0.98-matrix #312 (See [https://builds.apache.org/job/HBase-0.98-matrix/312/]) HBASE-15323 Hbase Rest CheckAndDeleteAPi should be able to delete more (apurtell: rev f0a38ba29d03bf935b8cabaf29befb46cdd221f8) * hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestGetAndPutResource.java * hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java * hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/RowResourceBase.java > Hbase Rest CheckAndDeleteAPi should be able to delete more cells > > > Key: HBASE-15323 > URL: https://issues.apache.org/jira/browse/HBASE-15323 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 1.1.1 > Environment: Linux And Windows >Reporter: Ajith > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18, 1.4.0 > > Attachments: HBASE-15323 Rest API CheckAndDelete4.patch, > HBASE-15323-addendum-0.98.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Java CheckAndDelete API accepts Delete object which can be used to delete (a > cell / cell version / multiple cells / column family or a row), but the rest > api only allows to delete the cell (without any version) > Need to add this capability to rest api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15477) Do not save 'next block header' when we cache hfileblocks
[ https://issues.apache.org/jira/browse/HBASE-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15477: -- Status: Patch Available (was: Open) > Do not save 'next block header' when we cache hfileblocks > - > > Key: HBASE-15477 > URL: https://issues.apache.org/jira/browse/HBASE-15477 > Project: HBase > Issue Type: Sub-task > Components: BlockCache, Performance >Reporter: stack >Assignee: stack > Attachments: 15477.patch > > > When we read from HDFS, we overread to pick up the next blocks header. > Doing this saves a seek as we move through the hfile; we save having to > do an explicit seek just to read the block header every time we need to > read the body. We used to read in the next header as part of the > current blocks buffer. This buffer was then what got persisted to > blockcache; so we were over-persisting wrtiting out our block plus the > next blocks' header (overpersisting 33 bytes). Parse of HFileBlock > complicated by this extra tail. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198873#comment-15198873
]
ramkrishna.s.vasudevan commented on HBASE-15064:
Ideally this limitedItemIndex should be used if the curItemIndex is just the
one before it right? Otherwise it makes sense to say hasReminaing is true.
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch, MBB_hasRemaining.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1515#comment-1515
]
Hudson commented on HBASE-15430:
FAILURE: Integrated in HBase-1.0 #1150 (See
[https://builds.apache.org/job/HBase-1.0/1150/])
HBASE-15430 Failed taking snapshot - Manifest proto-message too large
(matteo.bertozzi: rev d8a5820becfb776ba7d9f5597e56563ca67dc87e)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotManifest.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Assignee: JunHo Cho
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18
>
> Attachments: HBASE-15430-v4.patch, HBASE-15430-v5.patch,
> HBASE-15430-v6.patch, hbase-15430-v1.patch, hbase-15430-v2.patch,
> hbase-15430-v3.branch.0.98.patch, hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
>
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201485#comment-15201485
]
Ted Yu commented on HBASE-15406:
But shouldn't the admin rerun hbck if previous hbck run gets aborted ?
At the end of successful hbck run, switch state would be restored anyway.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198848#comment-15198848
]
ramkrishna.s.vasudevan commented on HBASE-15064:
{code}
bb1 = ByteBuffer.wrap(b);
bb2 = ByteBuffer.wrap(b1);
bb3 = ByteBuffer.allocate(4);
mbb1 = new MultiByteBuff(bb1, bb2, bb3);
mbb1.limit(12);
for(int i = 0; i < 12; i++) {
mbb1.get();
}
assertFalse(mbb1.hasRemaining());
{code}
You mean this hasRemaining() will return true in this case? No it will work
correctly. Because the limitedItemIndex will be 2 and once we have reached the
end of 12 bytes the item at the limitedItemIndex will have its limit 0. So
this check 'return this.items[limitedItemIndex].hasRemaining();' will return
false.
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch, MBB_hasRemaining.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15433) SnapshotManager#restoreSnapshot not update table and region count quota correctly when encountering exception
[
https://issues.apache.org/jira/browse/HBASE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201414#comment-15201414
]
Ted Yu edited comment on HBASE-15433 at 3/18/16 12:46 PM:
--
Jianwei:
I looked at the scenario (region counts of 8 and 10 in snapshots) above where
retrieving from region locator may produce inaccurate end result.
We can go with patch v4.
Since branch-1 is among the Fix versions, can you attach patch for branch-1 ?
branch-1 tests are more stable, hopefully we can get higher confidence.
Thanks
was (Author: yuzhih...@gmail.com):
Jianwei:
I looked at the scenario (region counts of 8 and 10 in snapshots) above where
retrieving from region locator may produce inaccurate end result.
We can go with path v4.
Since branch-1 is among the Fix versions, can you attach patch for branch-1 ?
branch-1 tests are more stable, hopefully we can get higher confidence.
Thanks
> SnapshotManager#restoreSnapshot not update table and region count quota
> correctly when encountering exception
> -
>
> Key: HBASE-15433
> URL: https://issues.apache.org/jira/browse/HBASE-15433
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.4.0, 1.1.5
>
> Attachments: HBASE-15433-trunk-v1.patch, HBASE-15433-trunk-v2.patch,
> HBASE-15433-trunk.patch, HBASE-15433-v3.patch, HBASE-15433-v4.patch
>
>
> In SnapshotManager#restoreSnapshot, the table and region quota will be
> checked and updated as:
> {code}
> try {
> // Table already exist. Check and update the region quota for this
> table namespace
> checkAndUpdateNamespaceRegionQuota(manifest, tableName);
> restoreSnapshot(snapshot, snapshotTableDesc);
> } catch (IOException e) {
>
> this.master.getMasterQuotaManager().removeTableFromNamespaceQuota(tableName);
> LOG.error("Exception occurred while restoring the snapshot " +
> snapshot.getName()
> + " as table " + tableName.getNameAsString(), e);
> throw e;
> }
> {code}
> The 'checkAndUpdateNamespaceRegionQuota' will fail if regions in the snapshot
> make the region count quota exceeded, then, the table will be removed in the
> 'catch' block. This will make the current table count and region count
> decrease, following table creation or region split will succeed even if the
> actual quota is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15477) Do not save 'next block header' when we cache hfileblocks
[ https://issues.apache.org/jira/browse/HBASE-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15477: -- Attachment: 15477v2.patch Finished patch > Do not save 'next block header' when we cache hfileblocks > - > > Key: HBASE-15477 > URL: https://issues.apache.org/jira/browse/HBASE-15477 > Project: HBase > Issue Type: Sub-task > Components: BlockCache, Performance >Reporter: stack >Assignee: stack > Attachments: 15477.patch, 15477v2.patch > > > When we read from HDFS, we overread to pick up the next blocks header. > Doing this saves a seek as we move through the hfile; we save having to > do an explicit seek just to read the block header every time we need to > read the body. We used to read in the next header as part of the > current blocks buffer. This buffer was then what got persisted to > blockcache; so we were over-persisting wrtiting out our block plus the > next blocks' header (overpersisting 33 bytes). Parse of HFileBlock > complicated by this extra tail. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13895) DATALOSS: Region assigned before WAL replay when abort
[ https://issues.apache.org/jira/browse/HBASE-13895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13895: -- Release Note: If the master went to assign a region concurrent with a RegionServer abort, the returned RegionServerAbortedException was being handled as though the region had been cleanly offlined so assign was allowed proceed. If the region was opened in its new location before WAL replay completion, the replayed edits were ignored, worst case, or were later played over the top of edits that had come in since open and so susceptible to overwrite. In either case, DATALOSS. > DATALOSS: Region assigned before WAL replay when abort > -- > > Key: HBASE-13895 > URL: https://issues.apache.org/jira/browse/HBASE-13895 > Project: HBase > Issue Type: Bug > Components: Recovery, Region Assignment, wal >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.1.2 > > Attachments: 13895.branch-1.2.txt, 13895.master.addendum2.txt, > 13895.master.patch, hbase-13895_addendum-master.patch, > hbase-13895_addendum.patch, hbase-13895_addendum3-branch-1.1.patch, > hbase-13895_addendum3-branch-1.patch, hbase-13895_addendum3-master.patch, > hbase-13895_v1-branch-1.1.patch > > > Opening a place holder till finish analysis. > I have dataloss running ITBLL at 3B (testing HBASE-13877). Most obvious > culprit is the double-assignment that I can see. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15475) Allow TimestampsFilter to provide a seek hint
[ https://issues.apache.org/jira/browse/HBASE-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15475: -- Attachment: HBASE-15475-v1.patch > Allow TimestampsFilter to provide a seek hint > - > > Key: HBASE-15475 > URL: https://issues.apache.org/jira/browse/HBASE-15475 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15475-v1.patch, HBASE-15475.patch > > > If a user wants to read specific timestamps currently it's a very linear > scan. This is so that all deletes can be respected. However if the user > doesn't care about deletes it can dramatically speed up the request to seek. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198068#comment-15198068 ] Elliott Clark commented on HBASE-15390: --- +1 remove the unused import. > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0, 1.2.1 > > Attachments: HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15480) Bloom Filter array access needs to be more efficient
[ https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Koetke updated HBASE-15480: -- Description: It is currently inefficient to do lots of bloom filter checks. Each check has overhead like going to the block cache to retrieve the block and recording metrics. It would be good to have one bloom filter check api that does a bunch of checks without so much block retrieval and metrics updates. (was: The bloom filter implementation for an array of items is inefficient.) > Bloom Filter array access needs to be more efficient > > > Key: HBASE-15480 > URL: https://issues.apache.org/jira/browse/HBASE-15480 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 1.0.3 > > > It is currently inefficient to do lots of bloom filter checks. Each check has > overhead like going to the block cache to retrieve the block and recording > metrics. It would be good to have one bloom filter check api that does a > bunch of checks without so much block retrieval and metrics updates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15412) Add average region size metric
[ https://issues.apache.org/jira/browse/HBASE-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alicia Ying Shu updated HBASE-15412: Status: Open (was: Patch Available) > Add average region size metric > -- > > Key: HBASE-15412 > URL: https://issues.apache.org/jira/browse/HBASE-15412 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alicia Ying Shu > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15412.patch, HBASE-15412.patch, HBASE-15412.patch > > > We have several metrics related to region store file size, num regions, etc > per regionserver, but we do not have a single metric to track the average > region size per regionserver. > Avg region size is important to look at for deciding on the split policy, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198796#comment-15198796
]
deepankar commented on HBASE-15064:
---
Also the hasRemaining method [~anoop.hbase] suggested should also work
seamlessly similar to the generic API right ?
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15323) Hbase Rest CheckAndDeleteAPi should be able to delete more cells
[ https://issues.apache.org/jira/browse/HBASE-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197913#comment-15197913 ] Hudson commented on HBASE-15323: FAILURE: Integrated in HBase-0.98-matrix #313 (See [https://builds.apache.org/job/HBase-0.98-matrix/313/]) Amend HBASE-15323 Commit changes staged for 0.98 but not committed (apurtell: rev 3dd36585985b950e036e196767dc7794ab1a8c98) * hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java > Hbase Rest CheckAndDeleteAPi should be able to delete more cells > > > Key: HBASE-15323 > URL: https://issues.apache.org/jira/browse/HBASE-15323 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 1.1.1 > Environment: Linux And Windows >Reporter: Ajith >Assignee: Ajith > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18, 1.4.0 > > Attachments: HBASE-15323 Rest API CheckAndDelete4.patch, > HBASE-15323-addendum-0.98.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Java CheckAndDelete API accepts Delete object which can be used to delete (a > cell / cell version / multiple cells / column family or a row), but the rest > api only allows to delete the cell (without any version) > Need to add this capability to rest api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15441) Fix WAL splitting when region has moved multiple times
[ https://issues.apache.org/jira/browse/HBASE-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198420#comment-15198420 ] Elliott Clark commented on HBASE-15441: --- Pushed to master, branch-1, branch-1.3, branch-1.2 > Fix WAL splitting when region has moved multiple times > -- > > Key: HBASE-15441 > URL: https://issues.apache.org/jira/browse/HBASE-15441 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.4.0 > > Attachments: HBASE-15441-v1.patch, HBASE-15441-v2.patch, > HBASE-15441-v3.patch, HBASE-15441-v4.patch, HBASE-15441-v5.patch, > HBASE-15441.patch > > > Currently WAL splitting is broken when a region has been opened multiple > times in recent minutes. > Region open and region close write event markers to the wal. These markers > should have the sequence id in them. However it is currently getting 1. That > means that if a region has moved multiple times in the last few mins then > multiple split log workers will try and create the recovered edits file for > sequence id 1. One of the workers will fail and on failing they will delete > the recovered edits. Causing all split wal attempts to fail. > We need to: > # make sure that close get the correct sequence id for open. > # Filter all region events from recovered edits > It appears that the close event with a sequence id of one is coming from > region warm up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14546) Backport stub DNS re-resolution options to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14546: --- Fix Version/s: (was: 0.98.18) 0.98.19 > Backport stub DNS re-resolution options to 0.98 > --- > > Key: HBASE-14546 > URL: https://issues.apache.org/jira/browse/HBASE-14546 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.98.19 > > > HBASE-12943 and HBASE-13067 addresses infinite caching preventing servers > from rejoining a cluster using the same hostname but a different IP address. > HBASE-14544 modifies this to be optional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15390: Fix Version/s: (was: 1.2.1) 2.0.0 > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15334) Add avro support for spark hbase connector
[ https://issues.apache.org/jira/browse/HBASE-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated HBASE-15334: --- Description: Avro is a popular format for hbase storage. User may want the support natively in the connector. With the support, user can save serialized avro into hbase table, and then query on top of it using spark sql. The conversion between avro and catalyst datatype will be handled automatically. This is one way of support complex data types. Otherwise, user has to define their own customized Serdes to support complex data types. was: Avro is a popular format for hbase storage. User may want the support natively in the connector. With the support, user can save serialized avro into hbase table, and then query on top of it using spark sql. This is one way of support complex data types automatically supported. Otherwise, user has to define their own customized Serdes to support complex data types. > Add avro support for spark hbase connector > -- > > Key: HBASE-15334 > URL: https://issues.apache.org/jira/browse/HBASE-15334 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15334-1.patch, HBASE-15334-2.patch, > HBASE-15334-3.patch, HBASE-15334-4.patch, HBASE-15334-5.patch > > > Avro is a popular format for hbase storage. User may want the support > natively in the connector. > With the support, user can save serialized avro into hbase table, and then > query on top of it using spark sql. The conversion between avro and catalyst > datatype will be handled automatically. This is one way of support complex > data types. Otherwise, user has to define their own customized Serdes to > support complex data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199603#comment-15199603 ] Hudson commented on HBASE-15390: SUCCESS: Integrated in HBase-1.3 #607 (See [https://builds.apache.org/job/HBase-1.3/607/]) HBASE-15390 Unnecessary MetaCache evictions cause elevated number of (antonov: rev f81348894b60921b2974c1a5712d98588c14c2a5) * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390-addendum.patch, > HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198473#comment-15198473
]
deepankar commented on HBASE-15064:
---
I am still seeing this exception on our servers, I think I found something,
what I observe is that a couple of things, in the normal byte buffers
(java.nio) the hasRemaining function uses the current position and limit {code}
/**
* Tells whether there are any elements between the current position and
* the limit.
*
* @return true if, and only if, there is at least one element
* remaining in this buffer
*/
public final boolean hasRemaining() {
return position < limit;
}
{code}
But in the MultiByteBuff we have the hasRemaining is not taking care of limit
{code}
/**
* Returns true if there are elements between the current position and the
limit
* @return true if there are elements, false otherwise
*/
@Override
public final boolean hasRemaining() {
return this.curItem.hasRemaining() || this.curItemIndex < this.items.length
- 1;
}
{code}
Also the items array is not changed in the limit(int) function, this means
there could be a scenario where the user has asked to limit at the end of first
buffer, but the hasRemaining() will still return true, Is there any flaw in my
logic here ?
Also in the limit(int) function in the MultiByteBuff function we are doing
{code}
// Normally the limit will try to limit within the last BB item
int limitedIndexBegin = this.itemBeginPos[this.limitedItemIndex];
if (limit >= limitedIndexBegin && limit <
this.itemBeginPos[this.limitedItemIndex + 1]) {
this.items[this.limitedItemIndex].limit(limit - limitedIndexBegin);
return this;
}
{code}
here I think in the if statement isn't the logic be just {noformat} if (limit
> limitedIndexBegin && limit < this.itemBeginPos[this.limitedItemIndex + 1])
{noformat} because if somebody is trying to limit at the place which is exactly
at the boundary of the limitIndexBuffer then we are also including the last
item which does not have any data as you are limiting at 0 (as limit ==
limitedIndexBegin, which is at the boundary), But then once you have read
everything in the previous buffer if the client consults hasRemaining function
this will return again true (as curIterm < no_of_items in array) but when you
actually try to read anything we will throw BufferUnderFlowException because
again the last element has no data. There is similar issue with
{{getItemIndexfunction}} when again the {{elemIndex}} matches with the boundary
> BufferUnderflowException after last Cell fetched from an HFile Block served
> from L2 offheap cache
> -
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15064.patch
>
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[
https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199409#comment-15199409
]
Hadoop QA commented on HBASE-15325:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 48s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 59s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 23s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 28s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 28s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 23s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 45s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 58s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
38s {color} | {color:green} Patch does not generate ASF License warnings.
[jira] [Updated] (HBASE-15478) add comments to FSHLog explaining why syncRunnerIndex won't overflow
[ https://issues.apache.org/jira/browse/HBASE-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15478: Attachment: HBASE-15478.1.patch -01 - moves the "no exceptions here" comment closer to where it's talking about - adds a comment about why syncRunnerIndex works > add comments to FSHLog explaining why syncRunnerIndex won't overflow > > > Key: HBASE-15478 > URL: https://issues.apache.org/jira/browse/HBASE-15478 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15478.1.patch > > > A comment near the fix for HBASE-14759 will make the code easier for folks to > follow later down the road. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15470) Add a setting for Priority queue length
[ https://issues.apache.org/jira/browse/HBASE-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198432#comment-15198432 ] Mikhail Antonov commented on HBASE-15470: - +1 > Add a setting for Priority queue length > --- > > Key: HBASE-15470 > URL: https://issues.apache.org/jira/browse/HBASE-15470 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15470.patch > > > Meta can have very different request rates than any other region. We > shouldn't use the same call queue length. > For example: > If a normal request takes 100ms ( long scan or large get ) > but a call to meta is all in memory so it takes < 1ms > So a call queue length of 100 represents multiple seconds of work for normal > requests, but less than a second for meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Koetke updated HBASE-12148: -- Attachment: HBASE-12148-V3.patch > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.18 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148-V3.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot > 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15479) No more garbage or beware of autoboxing
[ https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202302#comment-15202302 ] Enis Soztutar commented on HBASE-15479: --- lgtm. > No more garbage or beware of autoboxing > --- > > Key: HBASE-15479 > URL: https://issues.apache.org/jira/browse/HBASE-15479 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch > > > Quick journey with JMC in a profile mode revealed very interesting and > unexpected heap polluter on a client side. Patch will shortly follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15487) Deletions done via BulkDeleteEndpoint make past data re-appear
[
https://issues.apache.org/jira/browse/HBASE-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mathias Herberts updated HBASE-15487:
-
Attachment: HBaseTest.java
This test reproduces the problem against a standalone 1.0.3 install.
> Deletions done via BulkDeleteEndpoint make past data re-appear
> --
>
> Key: HBASE-15487
> URL: https://issues.apache.org/jira/browse/HBASE-15487
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.3
>Reporter: Mathias Herberts
> Attachments: HBaseTest.java
>
>
> The Warp10 (www.warp10.io) time series database uses HBase as its underlying
> data store. The deletion of ranges of cells is performed using the
> BulkDeleteEndpoint.
> In the following scenario the deletion does not appear to be working properly:
> The table 't' is created with a single version using:
> create 't', {NAME => 'v', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER =>
> 'NONE', REPLICATION_SCOPE => '0', VERSIONS=> '1', MIN_VERSIONS => '0', TTL =>
> '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY
> =>'false', BLOCKCACHE => 'true'}
> We write a cell at row '0x00', colfam 'v', colq '', value 0x0
> We write the same cell again with value 0x1
> A scan will return a single value 0x1
> We then perform a delete using the BulkDeleteEndpoint and a Scan with a
> DeleteType of 'VERSION'
> The reported number of deleted versions is 1 (which is coherent given the
> table was created with MAX_VERSIONS=1)
> The same scan as the one performed before the delete returns a single value
> 0x0.
> This seems to happen when all operations are performed against the memstore.
> A regular delete will remove the cell and a later scan won't show it.
> I'll attach a test which demonstrates the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202558#comment-15202558 ] Matteo Bertozzi commented on HBASE-15481: - I think everyone is one the same page that exposing Path is something we don't want to do in the future. In 2.0 with the RedoFS work (HBASE-14090) we will go in a place where we will have an object e.g. StoreFile or WalFile that will have an incrementReference() and an open() that return a stream that we pass around. without leaking the detail implementation. Unfortunately for branch-1 there is no easy way out. WALActionListener is not an alternative. first of all is not pluggable. You can Implement a coprocessor just (that you don't use) just to have a way to get your class loaded inside hbase. then you have to try to get the WAL objects. there is no easy way to get all the wals. so you probably have to use the postRegionOpen() coprocessor event to get the WAL object from the factory walFactory.getWAL(regionInfo) and at that point you can register the listener... at that point you'll say... if only we had an Observer on the wal... and we have. the WALObserver is there for that reason. also, you can try to Implement your own WAL without Files. unfortunately if you do that and don't call the listener roll passing the path. Replication will not work since it relies on the file. so there is nothing you can do to abstract from the Path now. coprocessor are also known to break almost every release. and this one will not change for sure for all the 1.x release series. given all the places we are locked in relying on the fs-layout. For 2.0 the story will be different and since it is a major we are allowed to change things around, and that will be the place to get rid of Path and the fs-layout exposure (HBASE-14090). > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch, > HBASE-15481-v1.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202506#comment-15202506 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.4 #31 (See [https://builds.apache.org/job/HBase-1.4/31/]) HBASE-15302 Reenable the other tests disabled by HBASE-14678 (Phil Yang) (tedyu: rev d942d4e5a1fe5644a51b5775f3a3e27dd219ad1d) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15488) Add ACL for setting split merge switch
Ted Yu created HBASE-15488: -- Summary: Add ACL for setting split merge switch Key: HBASE-15488 URL: https://issues.apache.org/jira/browse/HBASE-15488 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Currently there is no access control for the split merge switch setter in MasterRpcServices. This JIRA adds necessary coprocessor hook along with enforcing permission check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202556#comment-15202556 ] Sean Busbey commented on HBASE-15481: - I'd also love to kill WALActionListener. The problem with it here is that you can't say "I want to listen to all wals". You'd effectively have to first make a coprocessor that was notified of every wal somehow (I guess WALObserver and track what comes in on writes?) and then register yourself as a listener on each. It's terrible and I don't really have a better solution for now, which is why I'm -0 rather than -1. > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch, > HBASE-15481-v1.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389-v13.patch Addressing the comments on RB. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-uc.patch, > HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch, > HBASE-15389-v12.patch, HBASE-15389-v13.patch, HBASE-15389-v2.patch, > HBASE-15389-v3.patch, HBASE-15389-v4.patch, HBASE-15389-v5.patch, > HBASE-15389-v6.patch, HBASE-15389-v7.patch, HBASE-15389-v8.patch, > HBASE-15389-v9.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202553#comment-15202553
]
Duo Zhang commented on HBASE-15389:
---
{quote}
So HBASE-15400 sits on top of this, and without it, this jira is not useful
from my understanding, right?
{quote}
Yes, it is the first part of implementing a DateTieredCompaction which could
write multiple output files.
{quote}
Previously there was some mention of cleanup on the store engine / compaction
policy interfaces. Are we doing that or no?
{quote}
We could do it in HBASE-15400. Let's discuss there.
The new patch will be ready soon. Thanks.
> Write out multiple files when compaction
>
>
> Key: HBASE-15389
> URL: https://issues.apache.org/jira/browse/HBASE-15389
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.19
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19
>
> Attachments: HBASE-15389-0.98.patch, HBASE-15389-uc.patch,
> HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch,
> HBASE-15389-v12.patch, HBASE-15389-v2.patch, HBASE-15389-v3.patch,
> HBASE-15389-v4.patch, HBASE-15389-v5.patch, HBASE-15389-v6.patch,
> HBASE-15389-v7.patch, HBASE-15389-v8.patch, HBASE-15389-v9.patch,
> HBASE-15389.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15483) After disabling Authorization, user should not be allowed to modify ACL record
[ https://issues.apache.org/jira/browse/HBASE-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-15483. Resolution: Not A Bug This is expected behavior and was described in the release notes when this setting was introduced. > After disabling Authorization, user should not be allowed to modify ACL > record > --- > > Key: HBASE-15483 > URL: https://issues.apache.org/jira/browse/HBASE-15483 > Project: HBase > Issue Type: Bug > Components: security >Reporter: meiwen li > > After setting hbase.security.authorization to be false, hbase does NOT do > authority check for any operations by any users. Thus, any user, including > read only user, has the authority to grant . The > change to ACL record is lasted and will take effective after next > authorization enabling. > The conseqence is, > A readonly user can change an admin user to be a "readonly" user after a > round of "disable authorization" and "enable authorization" > Also, > A readonly user can change a "readonly" user to be an Admin after such a > round of disable/enable. > It is expected that > after authorization is disabled, the authorization related file, the ACL > record, should not be open to users and not be changed. Otherwise, after the > authorization next enablement, the changed ACL takes action and users get > unexpected authority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15396) Enhance mapreduce.TableSplit to add encoded region name
[
https://issues.apache.org/jira/browse/HBASE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202274#comment-15202274
]
Hadoop QA commented on HBASE-15396:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
18s {color} | {color:green} hbase-server: patch generated 0 new + 23 unchanged
- 9 fixed = 23 total (was 32) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 32s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 162m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.util.TestHBaseFsckOneRS |
| | org.apache.hadoop.hbase.mapred.TestTableInputFormat |
| |
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202494#comment-15202494 ] Enis Soztutar commented on HBASE-15389: --- bq. I will open new jira to address the duplicated code in the Compactor implementations That is acceptable as long as we are addressing that. We can do a parent jira subtask. Just left some comments in RB, but small stuff. Looks good to go. So HBASE-15400 sits on top of this, and without it, this jira is not useful from my understanding, right? Previously there was some mention of cleanup on the store engine / compaction policy interfaces. Are we doing that or no? > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-uc.patch, > HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch, > HBASE-15389-v12.patch, HBASE-15389-v2.patch, HBASE-15389-v3.patch, > HBASE-15389-v4.patch, HBASE-15389-v5.patch, HBASE-15389-v6.patch, > HBASE-15389-v7.patch, HBASE-15389-v8.patch, HBASE-15389-v9.patch, > HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15481) Add pre/post roll to WALObserver
[
https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202547#comment-15202547
]
Duo Zhang commented on HBASE-15481:
---
I agree with [~busbey] that we may have a WAL which does not rely a FileSystem
in the future so it is not a good idea to expose a 'Path' parameter in
coprocessor because it could be used outside out control.
I think you could use {{WALActionsListener}} for now since it is controlled by
us although [~stack] has said many times he want to kill that stuff :)
> Add pre/post roll to WALObserver
>
>
> Key: HBASE-15481
> URL: https://issues.apache.org/jira/browse/HBASE-15481
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 1.3.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch,
> HBASE-15481-v1.patch
>
>
> currently the WALObserver has only a pre/post Write. It will be useful to
> have a pre/post Roll too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15392:
--
Attachment: 15392v7 (1).patch
Retry. See if timeouts have gone away
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch, 15392v5.patch,
> 15392v6.patch, 15392v7 (1).patch, 15392v7.patch, 15392v7.patch,
> 15392v7.patch, 15392v7.patch, 15392v7.patch, HBASE-15392_suggest.patch,
> gc.png, gc.png, io.png, no_optimize.patch, no_optimize.patch, reads.png,
> reads.png, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:279)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:806)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:795)
>
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202381#comment-15202381
]
Duo Zhang commented on HBASE-15389:
---
{quote}
Why was TestStripeCompactor test removed?
{quote}
I just moved it to the rs.compactions package.
[~enis] I will open new jira to address the duplicated code in the Compactor
implementations. Is there any other problems? Thanks.
> Write out multiple files when compaction
>
>
> Key: HBASE-15389
> URL: https://issues.apache.org/jira/browse/HBASE-15389
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.19
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19
>
> Attachments: HBASE-15389-0.98.patch, HBASE-15389-uc.patch,
> HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch,
> HBASE-15389-v12.patch, HBASE-15389-v2.patch, HBASE-15389-v3.patch,
> HBASE-15389-v4.patch, HBASE-15389-v5.patch, HBASE-15389-v6.patch,
> HBASE-15389-v7.patch, HBASE-15389-v8.patch, HBASE-15389-v9.patch,
> HBASE-15389.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15488) Add ACL for setting split merge switch
[ https://issues.apache.org/jira/browse/HBASE-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15488: --- Attachment: (was: HBASE-15488.v1.patch) > Add ACL for setting split merge switch > -- > > Key: HBASE-15488 > URL: https://issues.apache.org/jira/browse/HBASE-15488 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > > Currently there is no access control for the split merge switch setter in > MasterRpcServices. > This JIRA adds necessary coprocessor hook along with enforcing permission > check in AccessController through the new hook. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202071#comment-15202071 ] Liyin Tang commented on HBASE-15482: Yeah, that's right. Ideally, if SnapshotInputFormat can read directly from snapshot instead of restore, that will be awesome ! Restoring a millions of storefiles will also take a long time. But that will be out of the scope of this jira. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202489#comment-15202489
]
Hadoop QA commented on HBASE-15400:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 42s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 38s
{color} | {color:red} hbase-server: patch generated 1 new + 96 unchanged - 28
fixed = 97 total (was 124) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 55s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 50s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 44s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 42s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 38s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 38s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 39s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 35s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 30s
{color} | {color:red} Patch causes 27 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 27s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 15s
{color} | {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 3 new + 1
unchanged - 0 fixed = 4 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} |
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202543#comment-15202543 ] Dave Latham commented on HBASE-15482: - Yes, I quite agree - we also skip it. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15477) Do not save 'next block header' when we cache hfileblocks
[ https://issues.apache.org/jira/browse/HBASE-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15477: -- Attachment: 15366v4.patch Address [~ram_krish] comments up on rb and fix failing unit tests > Do not save 'next block header' when we cache hfileblocks > - > > Key: HBASE-15477 > URL: https://issues.apache.org/jira/browse/HBASE-15477 > Project: HBase > Issue Type: Sub-task > Components: BlockCache, Performance >Reporter: stack >Assignee: stack > Attachments: 15366v4.patch, 15477.patch, 15477v2.patch, 15477v3.patch > > > When we read from HDFS, we overread to pick up the next blocks header. > Doing this saves a seek as we move through the hfile; we save having to > do an explicit seek just to read the block header every time we need to > read the body. We used to read in the next header as part of the > current blocks buffer. This buffer was then what got persisted to > blockcache; so we were over-persisting wrtiting out our block plus the > next blocks' header (overpersisting 33 bytes). Parse of HFileBlock > complicated by this extra tail. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202167#comment-15202167 ] Sean Busbey commented on HBASE-15481: - What use case are we trying to solve? Is including the Path necessary to solve that? e.g. are we just trying to give some indication that the WAL system has reached some kind of a "checkpoint" state? > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch, > HBASE-15481-v1.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[
https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202540#comment-15202540
]
Ted Yu commented on HBASE-15302:
{code}
Fetching
https://builds.apache.org/job/HBase-1.4/31/jdk=latest1.7,label=yahoo-not-h2/console
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers
Hanging test :
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
Hanging test : org.apache.hadoop.hbase.wal.TestWALSplit
Hanging test : org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
...
Fetching
https://builds.apache.org/job/HBase-1.4/31/jdk=latest1.8,label=yahoo-not-h2/consoleFull
Building remotely on H9
(Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs yahoo-not-h2) in workspace
/home/jenkins/jenkins-slave/workspace/HBase-1.4/jdk/latest1.8/label/yahoo-not-h2
Printing hanging tests
Hanging test :
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
Printing Failing tests
Failing test :
org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController
Failing test : org.apache.hadoop.hbase.client.TestAsyncProcess
{code}
Reverted TestMasterFailoverWithProcedures.java, TestWALSplit.java and
TestWALSplitCompressed
> Reenable the other tests disabled by HBASE-14678
>
>
> Key: HBASE-15302
> URL: https://issues.apache.org/jira/browse/HBASE-15302
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt,
> HBASE-15302-v1.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15479) No more garbage or beware of autoboxing
[
https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202107#comment-15202107
]
Ted Yu edited comment on HBASE-15479 at 3/18/16 9:39 PM:
-
Not running hbase-server has hidden cost:
{code}
Author: Mikhail Antonov
Date: Thu Mar 17 01:06:42 2016 -0700
HBASE-15390 Unnecessary MetaCache evictions cause elevated number of
requests to meta (ADDENDUM)
{code}
Currently Yetus only runs unit tests in modules which are touched by the patch.
This may lead to regression some cases where bug was detected when running
hbase-server module tests.
was (Author: yuzhih...@gmail.com):
Not running hbase-server has hidden cost:
{code}
Author: Mikhail Antonov
Date: Thu Mar 17 01:06:42 2016 -0700
HBASE-15390 Unnecessary MetaCache evictions cause elevated number of
requests to meta (ADDENDUM)
{code}
> No more garbage or beware of autoboxing
> ---
>
> Key: HBASE-15479
> URL: https://issues.apache.org/jira/browse/HBASE-15479
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch
>
>
> Quick journey with JMC in a profile mode revealed very interesting and
> unexpected heap polluter on a client side. Patch will shortly follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15405) Fix PE logging and wrong defaults in help message
[
https://issues.apache.org/jira/browse/HBASE-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15405:
--
Attachment: HBASE-15405-master-v4.patch
Retry
> Fix PE logging and wrong defaults in help message
> -
>
> Key: HBASE-15405
> URL: https://issues.apache.org/jira/browse/HBASE-15405
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-15405-master-v2.patch,
> HBASE-15405-master-v3.patch, HBASE-15405-master-v4.patch,
> HBASE-15405-master-v4.patch, HBASE-15405-master.patch,
> HBASE-15405-master.patch
>
>
> Corrects wrong default values for few options in the help message.
> Final stats from multiple clients are intermingled making it hard to
> understand. Also the logged stats aren't very machine readable. It can be
> helpful in a daily perf testing rig which scraps logs for results.
> Example of logs before the change.
> {noformat}
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.98, min=359.00, max=324050.00, stdDev=851.82, 95th=1368.00,
> 99th=1625.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.92, min=356.00, max=323394.00, stdDev=817.55, 95th=1370.00,
> 99th=1618.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.98, min=367.00, max=322745.00, stdDev=840.43, 95th=1369.00,
> 99th=1622.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 375.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 363.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.6624126434326
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.4124526977539
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 781.3929776087633
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 742.8027916717297
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1070.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1071.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th =
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th =
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th =
> 1623.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th =
> 1624.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 372.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th =
> 3013.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.2451229095459
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th =
> 3043.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 725.4744472152282
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th =
> 25282.38016755
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 895.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th =
> 25812.76334
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th
[jira] [Commented] (HBASE-15460) Fix infer issues in hbase-common
[ https://issues.apache.org/jira/browse/HBASE-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202536#comment-15202536 ] stack commented on HBASE-15460: --- Skimmed. +1 > Fix infer issues in hbase-common > > > Key: HBASE-15460 > URL: https://issues.apache.org/jira/browse/HBASE-15460 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15460-v1.patch, HBASE-15460.patch, > hbase-common.infer > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202539#comment-15202539 ] Liyin Tang commented on HBASE-15482: Dave, thanks for the response. Even we use HDFS snapshots, it will be great to have an option to skip calculating block locations. To decouple computing with storage , it is possible to set up computing layer for query engine like Spark/Hive/Presto in a different cluster. In these cases, the locality doesn't matter for both HBase and HDFS snapshots. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-13938:
--
Release Note: Use the master's timestamp when sending hbase:meta edits on
region merge to ensure proper ordering of new region addition and old region
deletes.
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15441) Fix WAL splitting when region has moved multiple times
[ https://issues.apache.org/jira/browse/HBASE-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15441: -- Attachment: HBASE-15441-v3.patch The tests all seem to be passing on my machine. Since there were so many of them, I suspect a test environment issue. > Fix WAL splitting when region has moved multiple times > -- > > Key: HBASE-15441 > URL: https://issues.apache.org/jira/browse/HBASE-15441 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15441-v1.patch, HBASE-15441-v2.patch, > HBASE-15441-v3.patch, HBASE-15441.patch > > > Currently WAL splitting is broken when a region has been opened multiple > times in recent minutes. > Region open and region close write event markers to the wal. These markers > should have the sequence id in them. However it is currently getting 1. That > means that if a region has moved multiple times in the last few mins then > multiple split log workers will try and create the recovered edits file for > sequence id 1. One of the workers will fail and on failing they will delete > the recovered edits. Causing all split wal attempts to fail. > We need to: > # make sure that close get the correct sequence id for open. > # Filter all region events from recovered edits > It appears that the close event with a sequence id of one is coming from > region warm up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14703: Component/s: Client > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-start.patch, HBASE-14703-v4.1.patch, > HBASE-14703-v4.patch, HBASE-14703-v6_with-check-and-mutate.patch, > HBASE-14703.patch, HBASE-14703_v1.patch, HBASE-14703_v10.patch, > HBASE-14703_v10.patch, HBASE-14703_v11.patch, HBASE-14703_v12.patch, > HBASE-14703_v13.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch, > HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, > HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch, HBASE-14703_v7.patch, > HBASE-14703_v8.patch, HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201435#comment-15201435 ] Phil Yang commented on HBASE-15302: --- There are many warnings and they seems not caused by this patch. The edits of WALSplitter has been pushed to master. Strange... Maybe we can wait for the results of retrying. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198277#comment-15198277 ] Hudson commented on HBASE-15360: SUCCESS: Integrated in HBase-1.3-IT #556 (See [https://builds.apache.org/job/HBase-1.3-IT/556/]) HBASE-15360 Fix flaky TestSimpleRpcScheduler (Duo Zhang) (stack: rev 3f9ce9785a647ff2a9e64baad1e0398ace59060d) * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0 >Reporter: Mikhail Antonov >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14963) Remove use of Guava Stopwatch from HBase client code
[ https://issues.apache.org/jira/browse/HBASE-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-14963: - Attachment: (was: HBASE-14963-branch-1.patch) > Remove use of Guava Stopwatch from HBase client code > > > Key: HBASE-14963 > URL: https://issues.apache.org/jira/browse/HBASE-14963 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Devaraj Das >Assignee: Devaraj Das > Labels: needs_releasenote > Fix For: 2.0.0 > > Attachments: no-stopwatch.txt > > > We ran into an issue where an application bundled its own Guava (and that > happened to be in the classpath first) and HBase's MetaTableLocator threw an > exception due to the fact that Stopwatch's constructor wasn't compatible... > Might be better to not depend on Stopwatch at all in MetaTableLocator since > the functionality is easily doable without. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15460) Fix infer issues in hbase-common
[
https://issues.apache.org/jira/browse/HBASE-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198545#comment-15198545
]
Hadoop QA commented on HBASE-15460:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 55s
{color} | {color:red} hbase-common: patch generated 7 new + 72 unchanged - 32
fixed = 79 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
78m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 44s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 43s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12793777/HBASE-15460-v1.patch |
| JIRA Issue | HBASE-15460 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux priapus.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3adcc75 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HBASE-15390) Unnecessary MetaCache evictions cause elevated number of requests to meta
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199425#comment-15199425 ] Hudson commented on HBASE-15390: SUCCESS: Integrated in HBase-1.3-IT #559 (See [https://builds.apache.org/job/HBase-1.3-IT/559/]) HBASE-15390 Unnecessary MetaCache evictions cause elevated number of (antonov: rev a317a0f89c3d3ea10814b176ca230d719a984d9b) * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java > Unnecessary MetaCache evictions cause elevated number of requests to meta > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15390-addendum.patch, > HBASE-15390.branch-1.3.v1.patch, HBASE-15390.v1.patch > > > - In ClientExceptionsUtil#findException() we don't unwrap correctly all > remote exceptions but a few selected ones; >more specifically, we don't unwrap correctly all remote exceptions where > getCause() exception doesn't have a constructor (String message) > - In AsyncProcess#receiveGlobalFailure() we don't check for > #isMetaClearingException() if tableName is null > - We don't have metric on client side to track number of cache drops -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198627#comment-15198627
]
Hudson commented on HBASE-15430:
SUCCESS: Integrated in HBase-1.1-JDK8 #1768 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1768/])
HBASE-15430 Failed taking snapshot - Manifest proto-message too large
(matteo.bertozzi: rev 1aafc240257833541a119eccf9067e7d054b7980)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotManifest.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Assignee: JunHo Cho
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18
>
> Attachments: HBASE-15430-v4.patch, HBASE-15430-v5.patch,
> HBASE-15430-v6.patch, hbase-15430-v1.patch, hbase-15430-v2.patch,
> hbase-15430-v3.branch.0.98.patch, hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
>
[jira] [Updated] (HBASE-15475) Allow TimestampsFilter to provide a seek hint
[ https://issues.apache.org/jira/browse/HBASE-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15475: -- Component/s: regionserver Filters Client > Allow TimestampsFilter to provide a seek hint > - > > Key: HBASE-15475 > URL: https://issues.apache.org/jira/browse/HBASE-15475 > Project: HBase > Issue Type: Improvement > Components: Client, Filters, regionserver >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15475-v1.patch, HBASE-15475.patch > > > If a user wants to read specific timestamps currently it's a very linear > scan. This is so that all deletes can be respected. However if the user > doesn't care about deletes it can dramatically speed up the request to seek. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15464) Flush / Compaction metrics revisited
[
https://issues.apache.org/jira/browse/HBASE-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198759#comment-15198759
]
Hadoop QA commented on HBASE-15464:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 52s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 23s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 23s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 29s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 45s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
40m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 41s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 42s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 55s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red}
[jira] [Updated] (HBASE-15353) Add metric for number of CallQueueTooBigException's
[ https://issues.apache.org/jira/browse/HBASE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15353: -- Assignee: (was: Elliott Clark) > Add metric for number of CallQueueTooBigException's > --- > > Key: HBASE-15353 > URL: https://issues.apache.org/jira/browse/HBASE-15353 > Project: HBase > Issue Type: Bug > Components: IPC/RPC, metrics >Reporter: Elliott Clark > Labels: easy, noob > > This exception is being thrown more. We should add a metric for this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
