[jira] [Updated] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19639:
--
Attachment: hbase-stack-regionserver-ve0528.log.gz
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient]
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities
> for 6 ms, close active connection. Will reconnect next time when there
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=12/11
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405829#comment-16405829
]
stack commented on HBASE-19639:
---
[~eshcar] Here is a log from one of the cluster servers. Over the life of the
job, regions went from zero to 15 regions.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient]
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities
> for 6 ms, close active connection. Will reconnect next time when there
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6]
>
[jira] [Commented] (HBASE-20116) Optimize the region last pushed sequence id layout on zk
[
https://issues.apache.org/jira/browse/HBASE-20116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405792#comment-16405792
]
Hadoop QA commented on HBASE-20116:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 16s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20116 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915245/HBASE-20116.v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 64299735e691 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12027/testReport/ |
| Max. process+thread count | 220 (vs. ulimit of 1) |
| modules | C: hbase-replication U: hbase-replication |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12027/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Optimize the region
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405790#comment-16405790
]
Hadoop QA commented on HBASE-19441:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
13s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} hbase-backup generated 2 new + 62 unchanged - 0 fixed = 64 total
(was 62) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 17 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 17s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 33s{color}
| {color:red} hbase-backup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
10s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.backup.TestBackupManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19441 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915241/HBASE-19441-v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 65336e2d69ce 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/12026/artifact/patchprocess/diff-compile-javac-hbase-backup.txt
|
| whitespace |
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405775#comment-16405775
]
Hadoop QA commented on HBASE-19852:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s{color}
| {color:red} hbase-thrift generated 1 new + 16 unchanged - 0 fixed = 17 total
(was 16) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915239/HBASE-19852.master.004.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux 4ae97c213bb6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/12025/artifact/patchprocess/diff-compile-javac-hbase-thrift.txt
|
|
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405769#comment-16405769 ] Anoop Sam John commented on HBASE-17819: It is now 1.64 GB. This patch as such reduces only 16 bytes per entry. So not so much of difference. May be like 0.15 GB only.. The other item is the skipList in BC which eats 40 bytes per entry. If that is also done, total can be reduced to 1.1 GB. But that needs some more discussion.. I wanted the simpler change to go in first. I had measured the object sizes using java.lang.instrument.Instrumentation way also and the theory looks correct as per that. > Reduce the heap overhead for BucketCache > > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch, > HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = align(12 (Object) + 17 )= 32 > BucketEntry > > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = align(12 (Object) + 51) = 64 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 32 + 64 + 80 = 176 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405751#comment-16405751
]
Hadoop QA commented on HBASE-19852:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s{color}
| {color:red} hbase-thrift generated 1 new + 16 unchanged - 0 fixed = 17 total
(was 16) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} hbase-thrift: The patch generated 16 new + 0 unchanged
- 0 fixed = 16 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 19s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
57s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915232/HBASE-19852.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux de43600cca79 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| javac |
[jira] [Updated] (HBASE-20116) Optimize the region last pushed sequence id layout on zk
[ https://issues.apache.org/jira/browse/HBASE-20116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20116: - Fix Version/s: 2.1.0 Status: Patch Available (was: Open) > Optimize the region last pushed sequence id layout on zk > > > Key: HBASE-20116 > URL: https://issues.apache.org/jira/browse/HBASE-20116 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20116.v1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20116) Optimize the region last pushed sequence id layout on zk
[ https://issues.apache.org/jira/browse/HBASE-20116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20116: - Attachment: HBASE-20116.v1.patch > Optimize the region last pushed sequence id layout on zk > > > Key: HBASE-20116 > URL: https://issues.apache.org/jira/browse/HBASE-20116 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20116.v1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405736#comment-16405736
]
Hadoop QA commented on HBASE-19852:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 43s{color}
| {color:red} hbase-thrift generated 1 new + 16 unchanged - 0 fixed = 17 total
(was 16) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} hbase-thrift: The patch generated 15 new + 0 unchanged
- 0 fixed = 15 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
40s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 12s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915229/HBASE-19852.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux 0fb1bf2c5667 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| javac |
[jira] [Updated] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-19441:
--
Status: Patch Available (was: Open)
Patch v2.
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov reassigned HBASE-19441:
-
Assignee: Vladimir Rodionov
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-19441:
--
Attachment: HBASE-19441-v2.patch
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20220) [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group
[ https://issues.apache.org/jira/browse/HBASE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405733#comment-16405733 ] Guangxu Cheng commented on HBASE-20220: --- Thanks [~yuzhih...@gmail.com]. Do we need push it to branch-2 and branch-1? > [RSGroup] Check if table exists in the cluster before moving it to the > specified regionserver group > --- > > Key: HBASE-20220 > URL: https://issues.apache.org/jira/browse/HBASE-20220 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20220.master.001.patch, > HBASE-20220.master.002.patch > > > move_tables_rsgroup/move_servers_tables_rsgroup can move non-existent tables > in the cluster to the specified regionserver group successfully.It's not > reasonable.We should check whether the table has already existed in the > cluster before moving the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Attachment: HBASE-19852.master.004.patch
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Attachment: HBASE-19852.master.003.patch
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405709#comment-16405709
]
Kevin Risden commented on HBASE-19852:
--
So this apparently created a second review board link not 100% sure why.
I'll close the first one and keep the autogenerated one. Making sure I address
all the initial review comments.
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Attachment: HBASE-19852.master.002.patch
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20095) Redesign single instance pool in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405693#comment-16405693 ] Reid Chan commented on HBASE-20095: --- ping [~mdrob] > Redesign single instance pool in CleanerChore > - > > Key: HBASE-20095 > URL: https://issues.apache.org/jira/browse/HBASE-20095 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Critical > Attachments: HBASE-20095.master.001.patch, > HBASE-20095.master.002.patch, HBASE-20095.master.003.patch, > HBASE-20095.master.004.patch, HBASE-20095.master.005.patch, > HBASE-20095.master.006.patch, HBASE-20095.master.007.patch, > HBASE-20095.master.008.patch, HBASE-20095.master.009.patch, > HBASE-20095.master.010.patch, HBASE-20095.master.011.patch, > HBASE-20095.master.012.patch, HBASE-20095.master.013.patch, > HBASE-20095.master.014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20226) Performance Improvement Taking Large Snapshots In Remote Filesystems
Saad Mufti created HBASE-20226: -- Summary: Performance Improvement Taking Large Snapshots In Remote Filesystems Key: HBASE-20226 URL: https://issues.apache.org/jira/browse/HBASE-20226 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 1.4.0 Environment: HBase 1.4.0 running on an AWS EMR cluster with the hbase.rootdir set to point to a folder in S3 Reporter: Saad Mufti When taking a snapshot of any table, one of the last steps is to delete the region manifests, which have already been rolled up into a larger overall manifest and thus have redundant information. This proposal is to do the deletion in a thread pool bounded by hbase.snapshot.thread.pool.max . For large tables with a lot of regions, the current single threaded deletion is taking longer than all the rest of the snapshot tasks when the Hbase data and the snapshot folder are both in a remote filesystem like S3. I have a patch for this proposal almost ready and will submit it tomorrow for feedback, although I haven't had a chance to write any tests yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405685#comment-16405685
]
Kevin Risden commented on HBASE-19852:
--
Ugh got distracted from this again. I hope to have something soon. Sorry for
the delay.
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20225) [RPC] Server does not say what version it is
stack created HBASE-20225: - Summary: [RPC] Server does not say what version it is Key: HBASE-20225 URL: https://issues.apache.org/jira/browse/HBASE-20225 Project: HBase Issue Type: Bug Components: rpc Affects Versions: 2.0.0 Reporter: stack Strange. Server does not tell clients what version it is. It is explicitly this way. See [1] from refguide appendix on rpc protocol where client says what it wants and server is silent unless it is unable to satisfy the client request. I suppose it made sense at the time trying to squeeze in protobuf handling into a pre-existing RPC but in hindsight, it seems a little silly we don't answer the Connection setup with a Connection setup response that has stuff like server version and capabilities. Its not so much a problem for our clients currently but I'm in here because asynchbase is broke against hbase2 [2]; hbase2 removes support for getClosestRowOrBefore doing meta lookups; clients are supposed to do a reverse scan instead. [~manolamancha] has just made a fix but you have to specify you are connecting to hbase2 which is not how asynchbase does it; in the past, asynchbase would just figure out what to do going off hints and exceptions thrown by our server. Not sure there is anything to do here. I tried reto-fitting a connection response but it will break hbase1 clients which we need to avoid. Here is yet another reason for our throwing away this home-grown RPC. New Project: put up an alternate port on which we'd provide a modern RPC, one that does streaming, etc., and basics like return server version. 1. http://hbase.apache.org/book.html#_server 2. https://github.com/OpenTSDB/asynchbase/issues/150#issuecomment-373949082 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20214) Review of RegionLocationFinder Class
[ https://issues.apache.org/jira/browse/HBASE-20214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20214: --- Fix Version/s: (was: 3.0.0) > Review of RegionLocationFinder Class > > > Key: HBASE-20214 > URL: https://issues.apache.org/jira/browse/HBASE-20214 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HBASE-20214.1.patch > > > # Use SLF4J parameter logging > # Remove superfluous code > # Replace code with re-usable libraries where possible > # Use different data structure > # Small perf improvements > # Fix some checkstyle -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20214) Review of RegionLocationFinder Class
[
https://issues.apache.org/jira/browse/HBASE-20214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405608#comment-16405608
]
Ted Yu commented on HBASE-20214:
Getting the following error when building with patch from HBASE-20223 where
commons-collections4 dependency is dropped:
{code}
[ERROR]
/a/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java:[251,5]
cannot find symbol
[ERROR] symbol: class MultiValuedMap
[ERROR] location: class
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder
{code}
> Review of RegionLocationFinder Class
>
>
> Key: HBASE-20214
> URL: https://issues.apache.org/jira/browse/HBASE-20214
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, master
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20214.1.patch
>
>
> # Use SLF4J parameter logging
> # Remove superfluous code
> # Replace code with re-usable libraries where possible
> # Use different data structure
> # Small perf improvements
> # Fix some checkstyle
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20214) Review of RegionLocationFinder Class
[ https://issues.apache.org/jira/browse/HBASE-20214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-20214: > Review of RegionLocationFinder Class > > > Key: HBASE-20214 > URL: https://issues.apache.org/jira/browse/HBASE-20214 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20214.1.patch > > > # Use SLF4J parameter logging > # Remove superfluous code > # Replace code with re-usable libraries where possible > # Use different data structure > # Small perf improvements > # Fix some checkstyle -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405571#comment-16405571
]
Hadoop QA commented on HBASE-19994:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} hbase-client: The patch generated 4 new + 2 unchanged
- 0 fixed = 6 total (was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
57s{color} | {color:red} hbase-server: The patch generated 2 new + 36 unchanged
- 1 fixed = 38 total (was 37) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
1s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}202m
19s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}248m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19994 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915173/HBASE-19994-master-v03.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cf316d551f61 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64
[jira] [Resolved] (HBASE-19584) hbase-thirdparty L refer to items not actually in the src release
[
https://issues.apache.org/jira/browse/HBASE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved HBASE-19584.
Resolution: Fixed
Assignee: Josh Elser
Hadoop Flags: Reviewed
Thanks, Mike!
> hbase-thirdparty L refer to items not actually in the src release
> ---
>
> Key: HBASE-19584
> URL: https://issues.apache.org/jira/browse/HBASE-19584
> Project: HBase
> Issue Type: Bug
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19584.001.patch
>
>
> From [~elserj]'s vote on 2.0-RC0:
> {quote}
> * L not entirely accurate, IMO. They state that things are included in the
> src release which are not. I think it would be more appropriate to push the
> relevant information down into src/main/apppended-resources for each module
> (e.g. hbase-shaded-protobuf would have
> src/main/appended-resources/\{LICENSE,NOTICE}) which have the relevant L
> content for the products being bundled. Thus, we'd have nothing in the
> 3rdparty L which reflects the src release.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved HBASE-19560.
Resolution: Fixed
Hadoop Flags: Reviewed
Pushed. Thanks for the review, Mike!
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch, HBASE-19560.002.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405545#comment-16405545
]
Mike Drob commented on HBASE-19560:
---
{code}
+# To finish, check what was build. If good copy to people.apache.org and
{code}
Missed this on the first pass. Should be dist/dev. Fine to fix on commit.
+1
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch, HBASE-19560.002.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-13160) SplitLogWorker does not pick up the task immediately
[
https://issues.apache.org/jira/browse/HBASE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-13160:
---
Fix Version/s: (was: 1.4.3)
1.4.4
> SplitLogWorker does not pick up the task immediately
>
>
> Key: HBASE-13160
> URL: https://issues.apache.org/jira/browse/HBASE-13160
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.4
>
> Attachments: hbase-13160_v1.patch
>
>
> We were reading some code with Jeffrey, and we realized that the
> SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
> sleeps, but have another mechanism to learn about new tasks, but does not
> make affective use of the zk notification.
> I have a simple patch which might improve this area.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12943: --- Fix Version/s: (was: 1.4.3) 1.4.4 > Set sun.net.inetaddr.ttl in HBase > - > > Key: HBASE-12943 > URL: https://issues.apache.org/jira/browse/HBASE-12943 > Project: HBase > Issue Type: Bug >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4 > > Attachments: 12943-1-master.txt > > > The default value of config: sun.net.inetaddr.ttl is -1 and the java > processes will cache the mapping of hostname to ip address forever, See: > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > But things go wrong when a regionserver with same hostname and different ip > address rejoins the hbase cluster. The HMaster will get wrong ip address of > the regionserver from this cache and every region assignment to this > regionserver will be blocked for a time because the HMaster can't communicate > with the regionserver. > A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong > cache expired. > Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17885: --- Fix Version/s: (was: 1.4.3) 1.4.4 1.5.0 > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.4 > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16071:
---
Fix Version/s: (was: 1.4.3)
1.4.4
> The VisibilityLabelFilter and AccessControlFilter should not count the
> "delete cell"
>
>
> Key: HBASE-16071
> URL: https://issues.apache.org/jira/browse/HBASE-16071
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.5.0, 1.3.3, 1.4.4
>
> Attachments: HBASE-16071-v1.patch, HBASE-16071-v2.patch,
> HBASE-16071-v3.patch
>
>
> The VisibilityLabelFilter will see and count the "delete cell" if the
> scan.isRaw() returns true, so the (put) cell will be skipped if it has lower
> version than "delete cell"
> The critical code is shown below:
> {code:title=VisibilityLabelFilter.java|borderStyle=solid}
> public ReturnCode filterKeyValue(Cell cell) throws IOException {
> if (curFamily.getBytes() == null
> || !(CellUtil.matchingFamily(cell, curFamily.getBytes(),
> curFamily.getOffset(),
> curFamily.getLength( {
> curFamily.set(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength());
> // For this family, all the columns can have max of
> curFamilyMaxVersions versions. No need to
> // consider the older versions for visibility label check.
> // Ideally this should have been done at a lower layer by HBase (?)
> curFamilyMaxVersions = cfVsMaxVersions.get(curFamily);
> // Family is changed. Just unset curQualifier.
> curQualifier.unset();
> }
> if (curQualifier.getBytes() == null
> || !(CellUtil.matchingQualifier(cell, curQualifier.getBytes(),
> curQualifier.getOffset(),
> curQualifier.getLength( {
> curQualifier.set(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength());
> curQualMetVersions = 0;
> }
> curQualMetVersions++;
> if (curQualMetVersions > curFamilyMaxVersions) {
> return ReturnCode.SKIP;
> }
> return this.expEvaluator.evaluate(cell) ? ReturnCode.INCLUDE :
> ReturnCode.SKIP;
> }
> {code}
> [VisibilityLabelFilter.java|https://github.com/apache/hbase/blob/d7a4499dfc8b3936a0eca867589fc2b23b597866/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelFilter.java]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14223:
---
Fix Version/s: (was: 1.4.3)
1.4.4
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 2.0.0, 1.5.0, 1.3.3, 1.2.8, 1.4.4
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:50,497
>
[jira] [Updated] (HBASE-17890) FuzzyRowFilter fail if unaligned support is false
[
https://issues.apache.org/jira/browse/HBASE-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17890:
---
Fix Version/s: (was: 1.4.3)
1.4.4
> FuzzyRowFilter fail if unaligned support is false
> -
>
> Key: HBASE-17890
> URL: https://issues.apache.org/jira/browse/HBASE-17890
> Project: HBase
> Issue Type: Sub-task
> Components: util
>Affects Versions: 2.0.0, 1.2.5
>Reporter: Jerry He
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.3.3, 1.2.8, 1.4.4
>
> Attachments: HBASE-17890.v0.branch-1.patch, HBASE-17890.v0.patch,
> HBASE-17890.v1.branch-1.patch, HBASE-17890.v1.patch, HBASE-17890.v2.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch, HBASE-17890.v3.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch
>
>
> When unaligned support is false, FuzzyRow tests fail:
> {noformat}
> Failed tests:
> TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186
> expected:<10> but was:<0>
> TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
> TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but
> was:
> TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343
> expected:<6250> but was:<0>
> TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445
> expected:<5> but was:<0>
> TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
> {noformat}
> This can be reproduced in the case described in HBASE-17869. Or on a platform
> really without unaligned support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18228) HBCK improvements
[ https://issues.apache.org/jira/browse/HBASE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18228: --- Fix Version/s: (was: 1.4.3) 1.4.4 > HBCK improvements > - > > Key: HBASE-18228 > URL: https://issues.apache.org/jira/browse/HBASE-18228 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Lars Hofhansl >Assignee: Karan Mehta >Priority: Critical > Fix For: 1.5.0, 1.4.4 > > Attachments: HBASE-18228.branch-1.3.patch > > > We just had a prod issue and running HBCK the way we did actually causes more > problems. > In part HBCK did stuff we did not expect, in part we had little visibility > into what HBCK was doing, and in part the logging was confusing. > I'm proposing 2 improvements: > 1. A dry-run mode. Run, and just list what would have been done. > 2. An interactive mode. Run, and for each action request Y/N user input. So > that a user can opt-out of stuff. > [~jmhsieh], FYI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18549) Unclaimed replication queues can go undetected
[ https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18549: --- Fix Version/s: (was: 1.4.3) 1.4.4 > Unclaimed replication queues can go undetected > -- > > Key: HBASE-18549 > URL: https://issues.apache.org/jira/browse/HBASE-18549 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Ashu Pachauri >Priority: Critical > Fix For: 1.5.0, 1.3.3, 1.4.4 > > > We have come across this situation multiple times where a zookeeper issues > can cause NodeFailoverWorker to fail picking up replication queue for a dead > region server silently. One example is when the znode size for a particular > queue exceed jute.maxBuffer value. > There can be other situations that may lead to this and just go undetected. > We need to have a metric for number of unclaimed replication queues. This > will help in mitigating the problem through alerting on the metric and > identifying underlying issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19364) Truncate_preserve fails with table when replica region > 1
[
https://issues.apache.org/jira/browse/HBASE-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405522#comment-16405522
]
Andrew Purtell commented on HBASE-19364:
{quote}For branch-2 and master, it needs another patch for HBASE-17319 , keep
it open until patch is pushed for branch-2 and master.
{quote}
Please stop doing this practice. When something is committed the issue needs to
be resolved. I am resolving this because it has been committed to branch-1,
branch-1.4, etc. and I am going to release from branch-1.4 with this change in
place.
> Truncate_preserve fails with table when replica region > 1
> --
>
> Key: HBASE-19364
> URL: https://issues.apache.org/jira/browse/HBASE-19364
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 1.4.3
>
> Attachments: HBASE-19364-branch-1-v2.patch,
> HBASE-19364-branch-1-v3.patch, HBASE-19364-branch-1.patch,
> HBASE-19364-branch-1.patch
>
>
> Root cause is same as HBASE-17319, here we need to exclude secondary regions
> while reading meta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19364) Truncate_preserve fails with table when replica region > 1
[ https://issues.apache.org/jira/browse/HBASE-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19364: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Truncate_preserve fails with table when replica region > 1 > -- > > Key: HBASE-19364 > URL: https://issues.apache.org/jira/browse/HBASE-19364 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > Fix For: 1.5.0, 1.4.3 > > Attachments: HBASE-19364-branch-1-v2.patch, > HBASE-19364-branch-1-v3.patch, HBASE-19364-branch-1.patch, > HBASE-19364-branch-1.patch > > > Root cause is same as HBASE-17319, here we need to exclude secondary regions > while reading meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18415) The local timeout may cause Admin to submit duplicate request
[ https://issues.apache.org/jira/browse/HBASE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18415: --- Fix Version/s: (was: 1.4.3) 1.4.4 > The local timeout may cause Admin to submit duplicate request > - > > Key: HBASE-18415 > URL: https://issues.apache.org/jira/browse/HBASE-18415 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4 > > Attachments: HBASE-18415.branch-1.ut.patch, > HBASE-18415.branch-1.v0.patch, HBASE-18415.branch-1.v1.patch, > HBASE-18415.branch-1.v2.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v3.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v4.patch, HBASE-18415.branch-1.v4.patch, > HBASE-18415.branch-1.v4.patch > > > After a timeout occurs on first request, client will retry the request with > distinct group/nonce. The second request may bring the TableXXXException back > if the first request have changed the table state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18639) nightly jobs that need to do jdk7 + jdk8 need more than 6 hours
[ https://issues.apache.org/jira/browse/HBASE-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18639: --- Fix Version/s: (was: 1.4.3) 1.4.4 > nightly jobs that need to do jdk7 + jdk8 need more than 6 hours > --- > > Key: HBASE-18639 > URL: https://issues.apache.org/jira/browse/HBASE-18639 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Priority: Major > Labels: beginner > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.4 > > > looks like the branches that need to do 2 jdks will need more than 6 hours, > at least until we can parallelize things. > e.g. > * > [branch-1.2|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/buildTimeTrend] > * > [branch-1.3|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.3/buildTimeTrend] > branch-1.4 and branch-1 are failing in jdk7 checks, so they don't go that > long yet, but once we get those tests fixed presumably they'll need the time > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20026) Add 1.4 release line to the JDK and Hadoop expectation tables
[ https://issues.apache.org/jira/browse/HBASE-20026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20026: --- Fix Version/s: (was: 1.4.3) 1.4.4 > Add 1.4 release line to the JDK and Hadoop expectation tables > - > > Key: HBASE-20026 > URL: https://issues.apache.org/jira/browse/HBASE-20026 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.4.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.4.4 > > > the ref guide currently doesn't have any expectations listed for branch-1.4 > releases around JDK and Hadoop versions. > either add it, or maybe update the existing entries so we have "1.2, 1.3, > 1.4" in a single entry. unless we're ready to include something different > among them. (Maybe note the default Hadoop we ship with? Or Hadoop 2.8.2+ > moving to S maybe? if we've actually done any of the legwork.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19444) RSGroups test units cannot be concurrently executed
[ https://issues.apache.org/jira/browse/HBASE-19444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19444: --- Fix Version/s: (was: 1.4.3) 1.4.4 > RSGroups test units cannot be concurrently executed > --- > > Key: HBASE-19444 > URL: https://issues.apache.org/jira/browse/HBASE-19444 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.5.0, 1.4.4 > > > TestRSGroups and friends cannot be concurrently executed or they are very > likely to flake, failing with constraint exceptions. If executed serially all > units pass. Fix for concurrent execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20063) Port HBASE-19799 (Add web UI to rsgroup) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-20063. Resolution: Later Assignee: (was: Andrew Purtell) Fix Version/s: (was: 1.4.3) (was: 1.5.0) There are some java 8-isms and a difficult problem with how to handle jsp pages meant for hbase-server that must be separated out into hbase-rsgroup on branch-1 and dynamically loaded. Not worth it at this time I think, so resolving as Later (probably Never) > Port HBASE-19799 (Add web UI to rsgroup) to branch-1 > > > Key: HBASE-20063 > URL: https://issues.apache.org/jira/browse/HBASE-20063 > Project: HBase > Issue Type: Task > Components: rsgroup, UI >Reporter: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405508#comment-16405508 ] Mike Drob commented on HBASE-17819: --- I haven't been following the technical details of this issue very closely, so do we have a high level overview of how much heap this will save total? We start off at 10 million blocks in 1.6G of heap, is there a measured amount after the patch is in? I ask for measured in case our theoretical math is off by something unexpected. > Reduce the heap overhead for BucketCache > > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch, > HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = align(12 (Object) + 17 )= 32 > BucketEntry > > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = align(12 (Object) + 51) = 64 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 32 + 64 + 80 = 176 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20224) Web UI is broken in standalone mode
Umesh Agashe created HBASE-20224: Summary: Web UI is broken in standalone mode Key: HBASE-20224 URL: https://issues.apache.org/jira/browse/HBASE-20224 Project: HBase Issue Type: Bug Components: UI Affects Versions: 2.0.0-beta-2 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0 Web UI doesn't show up in standalone mode on default port. This can be seen on master and branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HBASE-20224) Web UI is broken in standalone mode
[ https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-20224 started by Umesh Agashe. > Web UI is broken in standalone mode > --- > > Key: HBASE-20224 > URL: https://issues.apache.org/jira/browse/HBASE-20224 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Web UI doesn't show up in standalone mode on default port. This can be seen > on master and branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19924) hbase rpc throttling does not work for multi() with request count rater.
[
https://issues.apache.org/jira/browse/HBASE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405499#comment-16405499
]
huaxiang sun commented on HBASE-19924:
--
For Q1 and Q2, I will upload a new patch.
For Q3, the current design is that it needs to compensate the difference for
estimated size and actual size. So it only needs size (not count, as the count
is accurate).
For Q4, it is not an issue as all these size are the estimated one, so even for
check-exist Get, the estimated size is non-0, though at the other hand, the
check is not necessary.
> hbase rpc throttling does not work for multi() with request count rater.
>
>
> Key: HBASE-19924
> URL: https://issues.apache.org/jira/browse/HBASE-19924
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 0.16.0, 1.2.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Attachments: HBASE-19924-master-v001.patch
>
>
> Basically, rpc throttling does not work for request count based rater for
> multi. for the following code, when it calls limiter's checkQuota(),
> numWrites/numReads is lost.
> {code:java}
> @Override
> public void checkQuota(int numWrites, int numReads, int numScans) throws
> ThrottlingException {
> writeConsumed = estimateConsume(OperationType.MUTATE, numWrites, 100);
> readConsumed = estimateConsume(OperationType.GET, numReads, 100);
> readConsumed += estimateConsume(OperationType.SCAN, numScans, 1000);
> writeAvailable = Long.MAX_VALUE;
> readAvailable = Long.MAX_VALUE;
> for (final QuotaLimiter limiter : limiters) {
> if (limiter.isBypass()) continue;
> limiter.checkQuota(writeConsumed, readConsumed);
> readAvailable = Math.min(readAvailable, limiter.getReadAvailable());
> writeAvailable = Math.min(writeAvailable, limiter.getWriteAvailable());
> }
> for (final QuotaLimiter limiter : limiters) {
> limiter.grabQuota(writeConsumed, readConsumed);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19560:
---
Attachment: HBASE-19560.002.patch
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch, HBASE-19560.002.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405491#comment-16405491
]
Josh Elser commented on HBASE-19560:
.002 fixes up the suggestions Mike made. Largely not adding any more
automation, just fixing some of the stale recommendation/guidance.
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch, HBASE-19560.002.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20179) [TESTING] Scale
[ https://issues.apache.org/jira/browse/HBASE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405464#comment-16405464 ] stack edited comment on HBASE-20179 at 3/19/18 9:16 PM: Just had a successful 25B ITBLL run w/ monkeys on 7 node cluster. Took about just over a day. Made ~100 regions per server. 22hours to generate and 5 hours to verify. Needs to go faster so can do more cycles! (Don't recall doing 25B on branch-1... maybe someone else did). was (Author: stack): Just had a successful 25B ITBLL run w/ monkeys on 7 node cluster. Took about two days. Need to make it go faster so can do more test cycles. > [TESTING] Scale > --- > > Key: HBASE-20179 > URL: https://issues.apache.org/jira/browse/HBASE-20179 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > Umbrella issue for scale testing for 2.0. > At least keep account of what testing has been done in here. > TODO as subtasks > * Long-running test > * ITBLL w/o killing Master > * Big ITBLL (1B works, 10B needs more verification, 100B, and 1T). > * Many Regions (43k regions takes about 12 minutes on a 6node cluster to > deploy and 3 1/2 minutes to go down -- needs tuning). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405469#comment-16405469
]
Josh Elser commented on HBASE-19560:
{quote}
There is already dev-support/make_rc.sh
How about putting the new script under dev-support and naming it
make_3rdparty_rc.sh ?
{quote}
This is for the thirdparty repo.
bq. New policy is that we don't do md5 any more
Ah, good catch. Let me rip those out, happily.
bq. is this right? I thought it was an svn push? Also, no md5 again
I was wondering about this, actually. Was a straight copy-paste from the core
make_rc.sh. I guess it's just wrong :)
Thanks Mike.
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405465#comment-16405465
]
Mike Drob commented on HBASE-19560:
---
bq. +# Presumes that dev-support/generate-hadoopX-poms.sh has already been run.
I don't htink we need this, right?
bq. +# Get mvn protject version
nit: typo
{code}
+git clean -f -x -d
+$\{mvn} clean
{code}
mvn clean is likely redundant, but can leave it in since I don't feel like
trying to prove it
bq. +# Now do the two builds, one for hadoop1, then hadoop2
Don't need?
bq. +for i in *.tar.gz; do echo $i; gpg --print-md MD5 $i > $i.md5 ; done
New policy is that we don't do md5 any more
bq. +echo ' rsync -av ${output_dir}/*.gz ${output_dir}/*.md5
${output_dir}/*.sha512 ${output_dir}/*.asc
${APACHE_HBASE_DIST_DEV_DIR}/${hbase_name}/'
is this right? I thought it was an svn push? Also, no md5 again
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20179) [TESTING] Scale
[ https://issues.apache.org/jira/browse/HBASE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405464#comment-16405464 ] stack commented on HBASE-20179: --- Just had a successful 25B ITBLL run w/ monkeys on 7 node cluster. Took about two days. Need to make it go faster so can do more test cycles. > [TESTING] Scale > --- > > Key: HBASE-20179 > URL: https://issues.apache.org/jira/browse/HBASE-20179 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > Umbrella issue for scale testing for 2.0. > At least keep account of what testing has been done in here. > TODO as subtasks > * Long-running test > * ITBLL w/o killing Master > * Big ITBLL (1B works, 10B needs more verification, 100B, and 1T). > * Many Regions (43k regions takes about 12 minutes on a 6node cluster to > deploy and 3 1/2 minutes to go down -- needs tuning). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19584) hbase-thirdparty L refer to items not actually in the src release
[
https://issues.apache.org/jira/browse/HBASE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405446#comment-16405446
]
Mike Drob commented on HBASE-19584:
---
+1
> hbase-thirdparty L refer to items not actually in the src release
> ---
>
> Key: HBASE-19584
> URL: https://issues.apache.org/jira/browse/HBASE-19584
> Project: HBase
> Issue Type: Bug
>Reporter: Mike Drob
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19584.001.patch
>
>
> From [~elserj]'s vote on 2.0-RC0:
> {quote}
> * L not entirely accurate, IMO. They state that things are included in the
> src release which are not. I think it would be more appropriate to push the
> relevant information down into src/main/apppended-resources for each module
> (e.g. hbase-shaded-protobuf would have
> src/main/appended-resources/\{LICENSE,NOTICE}) which have the relevant L
> content for the products being bundled. Thus, we'd have nothing in the
> 3rdparty L which reflects the src release.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[
https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405428#comment-16405428
]
Ted Yu commented on HBASE-17819:
Looks good overall.
{code}
+ protected int incrementRefCount() {
+return unsafe.getAndAddInt(this, refCountOffset, 1) + 1;
{code}
Since the {{incrementRefCount}} method returns the new count, probably naming
it {{incrementRefCountAndGet}} is better.
> Reduce the heap overhead for BucketCache
>
>
> Key: HBASE-17819
> URL: https://issues.apache.org/jira/browse/HBASE-17819
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch,
> HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch
>
>
> We keep Bucket entry map in BucketCache. Below is the math for heapSize for
> the key , value into this map.
> BlockCacheKey
> ---
> String hfileName - Ref - 4
> long offset - 8
> BlockType blockType - Ref - 4
> boolean isPrimaryReplicaBlock - 1
> Total = align(12 (Object) + 17 )= 32
> BucketEntry
>
> int offsetBase - 4
> int length - 4
> byte offset1 - 1
> byte deserialiserIndex - 1
> long accessCounter - 8
> BlockPriority priority - Ref - 4
> volatile boolean markedForEvict - 1
> AtomicInteger refCount - 16 + 4
> long cachedTime - 8
> Total = align(12 (Object) + 51) = 64
> ConcurrentHashMap Map.Entry - 40
> blocksByHFile ConcurrentSkipListSet Entry - 40
> Total = 32 + 64 + 80 = 176
> For 10 million blocks we will end up having 1.6GB of heap size.
> This jira aims to reduce this as much as possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20218) Proposed Performance Enhancements For TableSnapshotInputFomat
[
https://issues.apache.org/jira/browse/HBASE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405408#comment-16405408
]
Hadoop QA commented on HBASE-20218:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 52s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}107m
1s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
1s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20218 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915171/HBASE-20218.01.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 70a8e660f026 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / df5de33a02 |
| maven | version:
[jira] [Commented] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405406#comment-16405406
]
Ted Yu commented on HBASE-19560:
There is already dev-support/make_rc.sh
How about putting the new script under dev-support and naming it
make_3rdparty_rc.sh ?
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20222) Bundle commons-collections
[ https://issues.apache.org/jira/browse/HBASE-20222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20222: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Umesh! > Bundle commons-collections > -- > > Key: HBASE-20222 > URL: https://issues.apache.org/jira/browse/HBASE-20222 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.1.0 > > Attachments: HBASE-20222.001.patch > > > commons-collections recently had some incompatibilities in their packaging > (between 3 and 4). Let's include it in our shaded-misc jar to avoid any > potential problems in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20221) Upgrade protobuf to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20221: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Upgrade protobuf to 3.5.1 > - > > Key: HBASE-20221 > URL: https://issues.apache.org/jira/browse/HBASE-20221 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.1.0 > > Attachments: HBASE-20221.001.patch > > > Get hbase2 onto the latest protobuf because it's there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20222) Bundle commons-collections
[ https://issues.apache.org/jira/browse/HBASE-20222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405364#comment-16405364 ] Umesh Agashe commented on HBASE-20222: -- +1 > Bundle commons-collections > -- > > Key: HBASE-20222 > URL: https://issues.apache.org/jira/browse/HBASE-20222 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.1.0 > > Attachments: HBASE-20222.001.patch > > > commons-collections recently had some incompatibilities in their packaging > (between 3 and 4). Let's include it in our shaded-misc jar to avoid any > potential problems in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405354#comment-16405354
]
Josh Elser commented on HBASE-19560:
.001 has a trimmed down version of the core make_rc.sh. Takes a "commit-ish"
(getting a branch or tag is on the user) and builds, deploys, and creates
src-release for us.
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19560:
---
Attachment: HBASE-19560.001.patch
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19560.001.patch
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[
https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405333#comment-16405333
]
Hadoop QA commented on HBASE-17819:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
10s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
9s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
14s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
21s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}102m
50s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-17819 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915168/HBASE-17819_new_V1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3422cabda365
[jira] [Assigned] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser reassigned HBASE-19560:
--
Assignee: Josh Elser
> create make-rc.sh for hbase-thirdparty
> --
>
> Key: HBASE-19560
> URL: https://issues.apache.org/jira/browse/HBASE-19560
> Project: HBase
> Issue Type: Task
> Components: hbase-thirdparty
>Reporter: Mike Drob
>Assignee: Josh Elser
>Priority: Major
> Fix For: thirdparty-2.1.0
>
>
> Make it easier to automate the process...
> Some notes, may not correspond to exact commands I ran...
> {noformat}
> mvn clean deploy -Papache-release
> git tag -s HEAD
> git archive HEAD -o release.tar.gz
> gpg --sign release.tar.gz
> gpg --print-mds
> gpg --print-md MD5
> gpg --print-md SHA512
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19584) hbase-thirdparty L refer to items not actually in the src release
[
https://issues.apache.org/jira/browse/HBASE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19584:
---
Attachment: HBASE-19584.001.patch
> hbase-thirdparty L refer to items not actually in the src release
> ---
>
> Key: HBASE-19584
> URL: https://issues.apache.org/jira/browse/HBASE-19584
> Project: HBase
> Issue Type: Bug
>Reporter: Mike Drob
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19584.001.patch
>
>
> From [~elserj]'s vote on 2.0-RC0:
> {quote}
> * L not entirely accurate, IMO. They state that things are included in the
> src release which are not. I think it would be more appropriate to push the
> relevant information down into src/main/apppended-resources for each module
> (e.g. hbase-shaded-protobuf would have
> src/main/appended-resources/\{LICENSE,NOTICE}) which have the relevant L
> content for the products being bundled. Thus, we'd have nothing in the
> 3rdparty L which reflects the src release.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19584) hbase-thirdparty L refer to items not actually in the src release
[
https://issues.apache.org/jira/browse/HBASE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405294#comment-16405294
]
Josh Elser commented on HBASE-19584:
.001 Removes content from top-level L, moves into hbase-shaded-protobuf's L
(to land in the jar) and fixes the top-level copyright year in NOTICE.txt
fyi [~mdrob]
> hbase-thirdparty L refer to items not actually in the src release
> ---
>
> Key: HBASE-19584
> URL: https://issues.apache.org/jira/browse/HBASE-19584
> Project: HBase
> Issue Type: Bug
>Reporter: Mike Drob
>Priority: Major
> Fix For: thirdparty-2.1.0
>
> Attachments: HBASE-19584.001.patch
>
>
> From [~elserj]'s vote on 2.0-RC0:
> {quote}
> * L not entirely accurate, IMO. They state that things are included in the
> src release which are not. I think it would be more appropriate to push the
> relevant information down into src/main/apppended-resources for each module
> (e.g. hbase-shaded-protobuf would have
> src/main/appended-resources/\{LICENSE,NOTICE}) which have the relevant L
> content for the products being bundled. Thus, we'd have nothing in the
> 3rdparty L which reflects the src release.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19924) hbase rpc throttling does not work for multi() with request count rater.
[
https://issues.apache.org/jira/browse/HBASE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405291#comment-16405291
]
huaxiang sun commented on HBASE-19924:
--
Thanks [~chia7712] for the comments and reviewing these now, will post update
later.
> hbase rpc throttling does not work for multi() with request count rater.
>
>
> Key: HBASE-19924
> URL: https://issues.apache.org/jira/browse/HBASE-19924
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 0.16.0, 1.2.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Attachments: HBASE-19924-master-v001.patch
>
>
> Basically, rpc throttling does not work for request count based rater for
> multi. for the following code, when it calls limiter's checkQuota(),
> numWrites/numReads is lost.
> {code:java}
> @Override
> public void checkQuota(int numWrites, int numReads, int numScans) throws
> ThrottlingException {
> writeConsumed = estimateConsume(OperationType.MUTATE, numWrites, 100);
> readConsumed = estimateConsume(OperationType.GET, numReads, 100);
> readConsumed += estimateConsume(OperationType.SCAN, numScans, 1000);
> writeAvailable = Long.MAX_VALUE;
> readAvailable = Long.MAX_VALUE;
> for (final QuotaLimiter limiter : limiters) {
> if (limiter.isBypass()) continue;
> limiter.checkQuota(writeConsumed, readConsumed);
> readAvailable = Math.min(readAvailable, limiter.getReadAvailable());
> writeAvailable = Math.min(writeAvailable, limiter.getWriteAvailable());
> }
> for (final QuotaLimiter limiter : limiters) {
> limiter.grabQuota(writeConsumed, readConsumed);
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20221) Upgrade protobuf to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405288#comment-16405288 ] Josh Elser commented on HBASE-20221: Thanks [~uagashe]. This is against the thirdparty repository, not the core repository. I don't think anyone has made QA smart enough to pick the right repo (or I haven't named the patch appropriately to tell QA to do that). > Upgrade protobuf to 3.5.1 > - > > Key: HBASE-20221 > URL: https://issues.apache.org/jira/browse/HBASE-20221 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.1.0 > > Attachments: HBASE-20221.001.patch > > > Get hbase2 onto the latest protobuf because it's there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20223) Use hbase-thirdparty 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20223: --- Priority: Blocker (was: Critical) > Use hbase-thirdparty 2.1.0 > -- > > Key: HBASE-20223 > URL: https://issues.apache.org/jira/browse/HBASE-20223 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-20223.001.patch > > > Update hbase to account for the changes to hbase-thirdparty: > * new (internal) protobuf version > * shaded commons-cli > * shaded commons-collections4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20223) Use hbase-thirdparty 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405281#comment-16405281 ] Josh Elser commented on HBASE-20223: .001 has the changes necessary for 3rdparty 2.1.0, but avoiding "submit patch" as this release isn't yet published. > Use hbase-thirdparty 2.1.0 > -- > > Key: HBASE-20223 > URL: https://issues.apache.org/jira/browse/HBASE-20223 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20223.001.patch > > > Update hbase to account for the changes to hbase-thirdparty: > * new (internal) protobuf version > * shaded commons-cli > * shaded commons-collections4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20223) Use hbase-thirdparty 2.1.0
[ https://issues.apache.org/jira/browse/HBASE-20223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20223: --- Attachment: HBASE-20223.001.patch > Use hbase-thirdparty 2.1.0 > -- > > Key: HBASE-20223 > URL: https://issues.apache.org/jira/browse/HBASE-20223 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20223.001.patch > > > Update hbase to account for the changes to hbase-thirdparty: > * new (internal) protobuf version > * shaded commons-cli > * shaded commons-collections4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20201) HBase must provide commons-cli-1.4 for mapreduce jobs with H3
[
https://issues.apache.org/jira/browse/HBASE-20201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved HBASE-20201.
Resolution: Done
Marking as "Done" – the actual fix will come with the upgrade to
hbase-thirdparty 2.1.0.
No specific code changes to be done to HBase as a part of this specific problem.
> HBase must provide commons-cli-1.4 for mapreduce jobs with H3
> -
>
> Key: HBASE-20201
> URL: https://issues.apache.org/jira/browse/HBASE-20201
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-20201.001.patch
>
>
> Been trying to get some pre-existing mapreduce tests working against HBase2.
> There's an inherent problem right now that hadoop-common depends on
> commons-cli-1.2 and HBase depends on commons-cli-1.4. This means that if you
> use {{$(hbase mapredcp)}} to submit a mapreduce job via {{hadoop jar}},
> you'll get an error like:
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/cli/DefaultParser
> Â Â Â at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.isHelpCommand(AbstractHBaseTool.java:165)
> Â Â Â at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:133)
> Â Â Â at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> Â Â Â at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:270)
> Â Â Â at hbase_it.App.main(App.java:85)
> Â Â Â at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Â Â Â at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Â Â Â at java.lang.reflect.Method.invoke(Method.java:498)
> Â Â Â at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> Â Â Â at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.cli.DefaultParser
> Â Â Â at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> Â Â Â at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> Â Â Â at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> Â Â Â at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> Â Â Â ... 11 more{noformat}
> My guess is that in previous versions, we didn't have this conflict with
> Hadoop (we were on the same version). Now, we're not.
> I see two routes:
> # We just alter the mapredcp to include our "correct" commons-cli-1.4 on the
> classpath and remind users to make use of the {{HADOOP_USER_CLASSPATH_FIRST}}
> environment variable
> # We put commons-cli into our hbase-thirdparty and stop using it directly.
> The former is definitely quicker, but I'm guessing the latter would insulate
> us more nicely.
> Thoughts, [~stack], [~busbey], [~mdrob] (and others who have done H3 work?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20223) Use hbase-thirdparty 2.1.0
Josh Elser created HBASE-20223: -- Summary: Use hbase-thirdparty 2.1.0 Key: HBASE-20223 URL: https://issues.apache.org/jira/browse/HBASE-20223 Project: HBase Issue Type: Task Components: dependencies Reporter: Josh Elser Assignee: Josh Elser Fix For: 2.0.0 Update hbase to account for the changes to hbase-thirdparty: * new (internal) protobuf version * shaded commons-cli * shaded commons-collections4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20221) Upgrade protobuf to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405267#comment-16405267 ] Josh Elser commented on HBASE-20221: Had to fix one of our PB patches (constant was renamed), but I didn't do an exhaustive review to make sure all of the patches semantically applied (release notes didn't call out anything scary). Local PE runs don't show any real difference in execution times. > Upgrade protobuf to 3.5.1 > - > > Key: HBASE-20221 > URL: https://issues.apache.org/jira/browse/HBASE-20221 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.0.1 > > Attachments: HBASE-20221.001.patch > > > Get hbase2 onto the latest protobuf because it's there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20221) Upgrade protobuf to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405266#comment-16405266 ] Umesh Agashe commented on HBASE-20221: -- +1. build failed as patch doesn't apply to master, looks like rebase is needed. > Upgrade protobuf to 3.5.1 > - > > Key: HBASE-20221 > URL: https://issues.apache.org/jira/browse/HBASE-20221 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.0.1 > > Attachments: HBASE-20221.001.patch > > > Get hbase2 onto the latest protobuf because it's there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405264#comment-16405264
]
huaxiang sun commented on HBASE-19994:
--
Upload a new rebased patch.
> Create a new class for RPC throttling exception, make it retryable.
>
>
> Key: HBASE-19994
> URL: https://issues.apache.org/jira/browse/HBASE-19994
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Attachments: HBASE-19994-master-v01.patch,
> HBASE-19994-master-v02.patch, HBASE-19994-master-v03.patch
>
>
> Based on a discussion at dev mailing list.
> Â
> {code:java}
> Thanks Andrew.
> +1 for the second option, I will create a jira for this change.
> Huaxiang
> On Feb 9, 2018, at 1:09 PM, Andrew Purtell wrote:
> We have
> public class ThrottlingException extends QuotaExceededException
> public class QuotaExceededException extends DoNotRetryIOException
> Let the storage quota limits throw QuotaExceededException directly (based
> on DNRIOE). That seems fine.
> However, ThrottlingException is thrown as a result of a temporal quota,
> so it is inappropriate for this to inherit from DNRIOE, it should inherit
> IOException instead so the client is allowed to retry until successful, or
> until the retry policy is exhausted.
> We are in a bit of a pickle because we've released with this inheritance
> hierarchy, so to change it we will need a new minor, or we will want to
> deprecate ThrottlingException and use a new exception class instead, one
> which does not inherit from DNRIOE.
> On Feb 7, 2018, at 9:25 AM, Huaxiang Sun wrote:
> Hi Mike,
> Â Â You are right. For rpc throttling, definitely it is retryable. For storage
> quota, I think it will be fail faster (non-retryable).
> Â Â We probably need to separate these two types of exceptions, I will do some
> more research and follow up.
> Â Â Thanks,
> Â Â Huaxiang
> On Feb 7, 2018, at 9:16 AM, Mike Drob wrote:
> I think, philosophically, there can be two kinds of QEE -
> For throttling, we can retry. The quota is a temporal quota - you have done
> too many operations this minute, please try again next minute and
> everything will work.
> For storage, we shouldn't retry. The quota is a fixed quote - you have
> exceeded your allotted disk space, please do not try again until you have
> remedied the situation.
> Our current usage conflates the two, sometimes it is correct, sometimes not.
> On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote:
> Hi Stack,
> Â I run into a case that a mapreduce job in hive cannot finish because
> it runs into a QEE.
> I need to look into the hive mr task to see if QEE is not handled
> correctly in hbase code or in hive code.
> I am thinking that if  QEE is a retryable exception, then it should be
> taken care of by the hbase code.
> I will check more and report back.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 8:23 AM, Stack wrote:
> QEE being a DNRIOE seems right on the face of it.
> But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a
> throttling scenario Huaxiang?
> Thanks,
> S
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun updated HBASE-19994:
-
Attachment: HBASE-19994-master-v03.patch
> Create a new class for RPC throttling exception, make it retryable.
>
>
> Key: HBASE-19994
> URL: https://issues.apache.org/jira/browse/HBASE-19994
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Attachments: HBASE-19994-master-v01.patch,
> HBASE-19994-master-v02.patch, HBASE-19994-master-v03.patch
>
>
> Based on a discussion at dev mailing list.
> Â
> {code:java}
> Thanks Andrew.
> +1 for the second option, I will create a jira for this change.
> Huaxiang
> On Feb 9, 2018, at 1:09 PM, Andrew Purtell wrote:
> We have
> public class ThrottlingException extends QuotaExceededException
> public class QuotaExceededException extends DoNotRetryIOException
> Let the storage quota limits throw QuotaExceededException directly (based
> on DNRIOE). That seems fine.
> However, ThrottlingException is thrown as a result of a temporal quota,
> so it is inappropriate for this to inherit from DNRIOE, it should inherit
> IOException instead so the client is allowed to retry until successful, or
> until the retry policy is exhausted.
> We are in a bit of a pickle because we've released with this inheritance
> hierarchy, so to change it we will need a new minor, or we will want to
> deprecate ThrottlingException and use a new exception class instead, one
> which does not inherit from DNRIOE.
> On Feb 7, 2018, at 9:25 AM, Huaxiang Sun wrote:
> Hi Mike,
> Â Â You are right. For rpc throttling, definitely it is retryable. For storage
> quota, I think it will be fail faster (non-retryable).
> Â Â We probably need to separate these two types of exceptions, I will do some
> more research and follow up.
> Â Â Thanks,
> Â Â Huaxiang
> On Feb 7, 2018, at 9:16 AM, Mike Drob wrote:
> I think, philosophically, there can be two kinds of QEE -
> For throttling, we can retry. The quota is a temporal quota - you have done
> too many operations this minute, please try again next minute and
> everything will work.
> For storage, we shouldn't retry. The quota is a fixed quote - you have
> exceeded your allotted disk space, please do not try again until you have
> remedied the situation.
> Our current usage conflates the two, sometimes it is correct, sometimes not.
> On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote:
> Hi Stack,
> Â I run into a case that a mapreduce job in hive cannot finish because
> it runs into a QEE.
> I need to look into the hive mr task to see if QEE is not handled
> correctly in hbase code or in hive code.
> I am thinking that if  QEE is a retryable exception, then it should be
> taken care of by the hbase code.
> I will check more and report back.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 8:23 AM, Stack wrote:
> QEE being a DNRIOE seems right on the face of it.
> But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a
> throttling scenario Huaxiang?
> Thanks,
> S
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20220) [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group
[ https://issues.apache.org/jira/browse/HBASE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20220: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the patch, Guangxu > [RSGroup] Check if table exists in the cluster before moving it to the > specified regionserver group > --- > > Key: HBASE-20220 > URL: https://issues.apache.org/jira/browse/HBASE-20220 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20220.master.001.patch, > HBASE-20220.master.002.patch > > > move_tables_rsgroup/move_servers_tables_rsgroup can move non-existent tables > in the cluster to the specified regionserver group successfully.It's not > reasonable.We should check whether the table has already existed in the > cluster before moving the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20220) [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group
[
https://issues.apache.org/jira/browse/HBASE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405206#comment-16405206
]
Hadoop QA commented on HBASE-20220:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
6s{color} | {color:red} The patch generated 4 new + 25 unchanged - 4 fixed = 29
total (was 29) {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
1s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
23s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
37s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20220 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915160/HBASE-20220.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux ed3cdb4e4b8c 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC
[jira] [Updated] (HBASE-20218) Proposed Performance Enhancements For TableSnapshotInputFomat
[ https://issues.apache.org/jira/browse/HBASE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saad Mufti updated HBASE-20218: --- Attachment: HBASE-20218.01.patch > Proposed Performance Enhancements For TableSnapshotInputFomat > - > > Key: HBASE-20218 > URL: https://issues.apache.org/jira/browse/HBASE-20218 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 > Environment: HBase 1.4.0 running in AWS EMR 5.12.0 with the HBase > rootdir set to a folder in S3 >  >Reporter: Saad Mufti >Priority: Minor > Attachments: HBASE-20218.01.patch, HBASE-20218.patch > > > I have been testing a few Spark jobs we have at my company which work off of > TableSnapshotInputFormat to read directly from the filesystem snapshots > created on another EMR/Hbase cluster and stored in S3. During performance > testing I found various small changes which would greatly enhance peformance. > Right now we are running our jobs linked with a patched version of HBase > 1.4.0 in which I made these changes, and I am hoping to submit my patch for > review and eventual acceptance into the main codebase. >  > The list of changes are : >  > 1. a flag to control whether the snapshot restore uses a UUID based random > temp dir in the specified restore directory. We use the flag to turn this off > so that we can benefit from a AWS S3 specific bucket partitioning scheme we > have provisioned. The way S3 partitioning works, you have to give a fixed > path prefix and a pattern of files after that, and AWS can then partition on > the paths after the fixed prefix into different resources to get more > parallelization. We were advised by AWS that we could only get this good > partitioning behavior if we didn't have that rancom directory in there. >  > 2. a flag to turn off the code that tries to compute locality information > for the splits. This is useless when dealing with S3 since the files are not > on the cluster so there is no use in computing locality; and worse yet, it > uses a single thread in the driver to iterate over all the files in the > restored snapshot. For a very large table this was taking hours and hours > iterating through S3 objects just to list them (about 360,000 of them for the > our specific table). >  > 3. a flag to override the column family schema setting to prefetch regions on > open. This was causing the main executor thread on which a Spark task was > running, which was trying to read through HFile's for its scan, compete for a > lock on the underlying EMRFS stream object with prefetch threads trying to > read the same file, so most tasks in the Spark stage would finish but the > last few would linger half an hour or more competing with the prefetch > threads alternately for a lock on an EMRFS stream object. This is the only > change that had to be outside the mapreduce package as it directly affects > the prefetch behavior in CacheConfig.java >  > 4. a flag to turn off maintenance of Scan metrics. this was also causing a > major slowdown, getting rid of this sped things up 4-5 times. What I observed > in the thread dumps was that every call to update scan metrics was trying to > get some HBase counter object and deep underneath was trying to access some > Java resource bundle, and throwing an exception that it wasn't found. The > exception was never visible at the application level and was swallowed > underneath but whatever it was doing was causing a major slowdown. So we use > this flag to avoid collecting those metrics because we never used them >  > I am polishing my patch a bit more and hopefully will attach it tomorrow. One > caveat, I tried but struggled with how to write any useful unit/component > tests for these as these are invisible behaviors that do not affect the final > result at all. And I am not that familiar with the HBase testing standards, > so for now I am looking for guidance on what to tests. >  > Would appreciate any feedback plus guidance on writing tests, provided of > course there is interest in incorporating my patch into the main codebase. >  > Cheers. >  > Saad >  -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20218) Proposed Performance Enhancements For TableSnapshotInputFomat
[ https://issues.apache.org/jira/browse/HBASE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405170#comment-16405170 ] Saad Mufti commented on HBASE-20218: I attached a new patch after fixing the checkstyle errors > Proposed Performance Enhancements For TableSnapshotInputFomat > - > > Key: HBASE-20218 > URL: https://issues.apache.org/jira/browse/HBASE-20218 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 > Environment: HBase 1.4.0 running in AWS EMR 5.12.0 with the HBase > rootdir set to a folder in S3 >  >Reporter: Saad Mufti >Priority: Minor > Attachments: HBASE-20218.01.patch, HBASE-20218.patch > > > I have been testing a few Spark jobs we have at my company which work off of > TableSnapshotInputFormat to read directly from the filesystem snapshots > created on another EMR/Hbase cluster and stored in S3. During performance > testing I found various small changes which would greatly enhance peformance. > Right now we are running our jobs linked with a patched version of HBase > 1.4.0 in which I made these changes, and I am hoping to submit my patch for > review and eventual acceptance into the main codebase. >  > The list of changes are : >  > 1. a flag to control whether the snapshot restore uses a UUID based random > temp dir in the specified restore directory. We use the flag to turn this off > so that we can benefit from a AWS S3 specific bucket partitioning scheme we > have provisioned. The way S3 partitioning works, you have to give a fixed > path prefix and a pattern of files after that, and AWS can then partition on > the paths after the fixed prefix into different resources to get more > parallelization. We were advised by AWS that we could only get this good > partitioning behavior if we didn't have that rancom directory in there. >  > 2. a flag to turn off the code that tries to compute locality information > for the splits. This is useless when dealing with S3 since the files are not > on the cluster so there is no use in computing locality; and worse yet, it > uses a single thread in the driver to iterate over all the files in the > restored snapshot. For a very large table this was taking hours and hours > iterating through S3 objects just to list them (about 360,000 of them for the > our specific table). >  > 3. a flag to override the column family schema setting to prefetch regions on > open. This was causing the main executor thread on which a Spark task was > running, which was trying to read through HFile's for its scan, compete for a > lock on the underlying EMRFS stream object with prefetch threads trying to > read the same file, so most tasks in the Spark stage would finish but the > last few would linger half an hour or more competing with the prefetch > threads alternately for a lock on an EMRFS stream object. This is the only > change that had to be outside the mapreduce package as it directly affects > the prefetch behavior in CacheConfig.java >  > 4. a flag to turn off maintenance of Scan metrics. this was also causing a > major slowdown, getting rid of this sped things up 4-5 times. What I observed > in the thread dumps was that every call to update scan metrics was trying to > get some HBase counter object and deep underneath was trying to access some > Java resource bundle, and throwing an exception that it wasn't found. The > exception was never visible at the application level and was swallowed > underneath but whatever it was doing was causing a major slowdown. So we use > this flag to avoid collecting those metrics because we never used them >  > I am polishing my patch a bit more and hopefully will attach it tomorrow. One > caveat, I tried but struggled with how to write any useful unit/component > tests for these as these are invisible behaviors that do not affect the final > result at all. And I am not that familiar with the HBase testing standards, > so for now I am looking for guidance on what to tests. >  > Would appreciate any feedback plus guidance on writing tests, provided of > course there is interest in incorporating my patch into the main codebase. >  > Cheers. >  > Saad >  -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20090:
---
Release Note:
When there is concurrent region split, MemStoreFlusher may not find flushable
region if the only candidate region left hasn't received writes.
After this JIRA, such scenario wouldn't trigger Precondition assertion.
Instead, a DEBUG log would appear in region server log.
was:
When there is concurrent region split, MemStoreFlusher may not find flushable
region if the only candidate left doesn't receive writes.
After this JIRA, such scenario wouldn't trigger Precondition assertion.
Instead, a DEBUG log would appear in region server log.
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing memstore for region " + this
> + ", flushing=" + writestate.flushing + ", writesEnabled="
> + writestate.writesEnabled);
> }
> {code}
> Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving
> memory pressure at high level.
> When MemStoreFlusher ran to the following call, the region was no longer a
> flush candidate:
> {code}
> HRegion bestFlushableRegion =
> getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
> {code}
> So the other region,
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
> next. Since the region was not receiving write, the (current) Precondition
> check failed.
> The proposed fix is to convert the Precondition to normal return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-17819) Reduce the heap overhead for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17819: --- Attachment: HBASE-17819_new_V1.patch > Reduce the heap overhead for BucketCache > > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch, > HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = align(12 (Object) + 17 )= 32 > BucketEntry > > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = align(12 (Object) + 51) = 64 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 32 + 64 + 80 = 176 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405145#comment-16405145
]
Ted Yu commented on HBASE-20090:
bq. Why would a region not receive writes?
Please see description above:
Region 0453f29030757eedb6e6a1c57e88c085, belonging to TestTable, was being
split.
The other region, atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae
(of table atlas_janus), was not being written to - I used PerformanceEvaluation
which wrote to TestTable only.
Region fbcb5e495344542daf8b499e4bac03ae's data size was 0, triggering the
Precondition assertion (before the fix).
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing memstore for region " + this
> + ", flushing=" + writestate.flushing + ", writesEnabled="
> + writestate.writesEnabled);
> }
> {code}
> Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving
> memory pressure at high level.
> When MemStoreFlusher ran to the following call, the region was no longer a
> flush candidate:
> {code}
> HRegion bestFlushableRegion =
> getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
> {code}
> So the other region,
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
> next. Since the region was not receiving write, the (current) Precondition
> check failed.
> The proposed fix is to convert the Precondition to normal return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405136#comment-16405136
]
stack commented on HBASE-20090:
---
How does this happen "... if the only candidate left doesn't receive writes. "
Why would a region not receive writes?
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing memstore for region " + this
> + ", flushing=" + writestate.flushing + ", writesEnabled="
> + writestate.writesEnabled);
> }
> {code}
> Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving
> memory pressure at high level.
> When MemStoreFlusher ran to the following call, the region was no longer a
> flush candidate:
> {code}
> HRegion bestFlushableRegion =
> getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
> {code}
> So the other region,
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
> next. Since the region was not receiving write, the (current) Precondition
> check failed.
> The proposed fix is to convert the Precondition to normal return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20220) [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group
[ https://issues.apache.org/jira/browse/HBASE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405133#comment-16405133 ] Guangxu Cheng commented on HBASE-20220: --- v2 takes the suggestion from [~yuzhih...@gmail.com] > [RSGroup] Check if table exists in the cluster before moving it to the > specified regionserver group > --- > > Key: HBASE-20220 > URL: https://issues.apache.org/jira/browse/HBASE-20220 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: HBASE-20220.master.001.patch, > HBASE-20220.master.002.patch > > > move_tables_rsgroup/move_servers_tables_rsgroup can move non-existent tables > in the cluster to the specified regionserver group successfully.It's not > reasonable.We should check whether the table has already existed in the > cluster before moving the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20220) [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group
[ https://issues.apache.org/jira/browse/HBASE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20220: -- Attachment: HBASE-20220.master.002.patch > [RSGroup] Check if table exists in the cluster before moving it to the > specified regionserver group > --- > > Key: HBASE-20220 > URL: https://issues.apache.org/jira/browse/HBASE-20220 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: HBASE-20220.master.001.patch, > HBASE-20220.master.002.patch > > > move_tables_rsgroup/move_servers_tables_rsgroup can move non-existent tables > in the cluster to the specified regionserver group successfully.It's not > reasonable.We should check whether the table has already existed in the > cluster before moving the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405125#comment-16405125
]
stack commented on HBASE-19639:
---
These are 31G heaps.
bq. That looks to be too high boss. IMO it should be less than 4x.. ...
Yeah. I was playing around trying to get rid of the RegionTooBusyException.
Forgot to set it back down. Will do.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient]
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities
> for 6 ms, close active connection. Will reconnect next time when there
> are new requests.
> 2017-12-26 19:24:39,122 WARN
[jira] [Commented] (HBASE-20220) [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group
[
https://issues.apache.org/jira/browse/HBASE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405108#comment-16405108
]
Guangxu Cheng commented on HBASE-20220:
---
{quote}What if the exception changes in the future ? Would "Unknown table"
always apply ?
{quote}
{code:java}
/** @param s message */
public TableNotFoundException(String s) {
super(s);
}
public TableNotFoundException(byte[] tableName) {
super(Bytes.toString(tableName));
}
{code}
The message of *TableNotFoundException* are of two types, the name of the table
and common message.The common message contains multiple words, so when there is
only one word in the message, the word is the name of the table.
The message is the name of the table in this issue. If there are multiple
words, I will keep the old processing logic.
> [RSGroup] Check if table exists in the cluster before moving it to the
> specified regionserver group
> ---
>
> Key: HBASE-20220
> URL: https://issues.apache.org/jira/browse/HBASE-20220
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 3.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-20220.master.001.patch
>
>
> move_tables_rsgroup/move_servers_tables_rsgroup can move non-existent tables
> in the cluster to the specified regionserver group successfully.It's not
> reasonable.We should check whether the table has already existed in the
> cluster before moving the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405105#comment-16405105
]
Anoop Sam John commented on HBASE-19639:
bq.I don't follow. 4x is the default. You suggesting we do 2x?
No boss. Yes it is 4x by default. I was just saying the fact that you are
seeing the Exception because flush seems to be slower compared to the writes
speed. 4x seems good enough
bq.we should be able to test in isolation?
Yes we can I believe.
bq.This is default. I'm trying to run w/ all defaults. I will change default if
we come up w/ something better ("...95% of
hbase.regionserver.global.memstore.size..."). We don't block writes now.
Instead we throw the RegionTooBusyException. We don't put anything in the logs.
Let me fix this. Would be good to note when the Region is struggling... or at
least let me see if a metric I could use.
I mean to ask what is the size in GBs for the global memstore size. This is
default 40% of Xmx. We throw RegionTooBusyException from region when we try
write on a memstore with size >= 4 x flush size.But when the global
memstore size (Sum of all memstores) above this barrier we will block writes. I
can not see throwing Exception in such case.
bq.Hmm.. This last run of mine had it at 8 *. Let me set it do the default (4 *
).
That looks to be too high boss. IMO it should be less than 4x.. Because at
each region level, we have 4x as max size and above which we throw Exception.
The global memstore size barrier being 8x of (regions count * per region flush
size) means theoretically we allow each region to grow upto 8x of flush size.
To be clear I mean this. For my tests
regions count * per region flush size * 2 = 40% xmx
Ya when writes are becoming fast, we badly need faster flushes.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
>
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405099#comment-16405099 ] stack commented on HBASE-20188: --- Yeah, sorry if messed up the focus of this issue. This issue is about addressing the rumors that 2.0 is slower than 1.x (as [~carp84] reminds us). I dumped in here some questions around CMS (Compacting MemStore) because I'm running some scale tests and want them to finish faster... ; we used to block writing when we hit global or store limits; now we instead throw a RegionTooBusyExceptions back to the client -- sometimes the MR tasks spend a long time hanging out cycling through RegionTooBusyExceptions (We don't seem to be blocking because we've hit the global limit which is good...). Let me answer some comments in the above. [~eshcar] Do you pre-split the regions before the test? No. I'm running the ITBLL test. It starts with clean table of one region. We want to include testing we do not lose data across splits. FYI. bq. Can you estimate the rate at which the test is writing, what is the throughput? what is the data size written at each operation? Smile. Good questions. Let me do a compare to 1.4. bq. To check if this is indeed the problem you can decrease hbase.hregion.compacting.pipeline.segments.limit. The default is currently 4, you can try it with 2. bq. To further reduce these compares you can decrease the size of the CSLM by decreasing hbase.memstore.inmemoryflush.threshold.factor . The current default is 0.1 but you can try 0.02 it worked well in the past. Let me try this. [~anastas] bq. I saw the flushes not fast enough in all the tests I have done. As you suggest, NOT flushing fast enough is not a new problem. It has been with us always. I'm just trying to figure if we are doing better (or worse) with our new defaults. bq. Is the Eshcar's fix to HBASE-18294 included? It is there, yes. Let me add logging. In old days we'd have a log saying we'd hit the memstore limit. Now we just throw region busy exception (What [~anoop.hbase] says above ..."The issue Stack mentioned is he is getting so many RegionTooBusyExceptions at client side. "). Thanks all. Will be back w/ more info. If anyone manages to run other tests or compares to branch-1, dump your findings in there. Thanks. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20090:
---
Release Note:
When there is concurrent region split, MemStoreFlusher may not find flushable
region if the only candidate left doesn't receive writes.
After this JIRA, such scenario wouldn't trigger Precondition assertion.
Instead, a DEBUG log would appear in region server log.
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing memstore for region " + this
> + ", flushing=" + writestate.flushing + ", writesEnabled="
> + writestate.writesEnabled);
> }
> {code}
> Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving
> memory pressure at high level.
> When MemStoreFlusher ran to the following call, the region was no longer a
> flush candidate:
> {code}
> HRegion bestFlushableRegion =
> getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
> {code}
> So the other region,
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
> next. Since the region was not receiving write, the (current) Precondition
> check failed.
> The proposed fix is to convert the Precondition to normal return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405066#comment-16405066
]
stack commented on HBASE-20090:
---
Can we get a summary of what the problem is in release note and implications
should the condition arise?
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing memstore for region " + this
> + ", flushing=" + writestate.flushing + ", writesEnabled="
> + writestate.writesEnabled);
> }
> {code}
> Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving
> memory pressure at high level.
> When MemStoreFlusher ran to the following call, the region was no longer a
> flush candidate:
> {code}
> HRegion bestFlushableRegion =
> getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
> {code}
> So the other region,
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
> next. Since the region was not receiving write, the (current) Precondition
> check failed.
> The proposed fix is to convert the Precondition to normal return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405040#comment-16405040
]
stack commented on HBASE-19639:
---
[~eshcar] any chance you can attach here the entire log file of some RS? If
possible from start-hbase point and then throughout the test. Thanks.
Will do. Let me get you a good one (Cluster doing another test at mo will
be back).
[~anoop.hbase]
bq. In tests, we try flush to SDD or HDD boss?
HDD
bq. Now I can see the issue you face is the exception because memstore size is
4x larger than flush size.
I don't follow. 4x is the default. You suggesting we do 2x?
bq. But with compacting memstore, it will become fresh CSLM again and so very
fast writes.
Before in-memory compaction, we used to snapshot; move aside current CSLM
taking no more writes to it and replace it with a new one so new writes
would go fast again.
For your #1 and #2, we should be able to test in isolation? No? A compare of an
old-school flush against new in-memory compaction flush?
Wonder if a difference between read form CSLM and heap read from a bunch of
segments.
bq. How about your global memstore size limit.
This is default. I'm trying to run w/ all defaults. I will change default if we
come up w/ something better ("...95% of
hbase.regionserver.global.memstore.size..."). We don't block writes now.
Instead we throw the RegionTooBusyException. We don't put anything in the logs.
Let me fix this. Would be good to note when the Region is struggling... or at
least let me see if a metric I could use.
bq. When I did tests normally will select this barrier as 2 * regions# * flush
size.
Hmm.. This last run of mine had it at 8 *. Let me set it do the default (4 * ).
bq. If not on SSD, any chance for an SSD based tests?
Nope. No SSD in these chassis.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
>
[jira] [Commented] (HBASE-20222) Bundle commons-collections
[
https://issues.apache.org/jira/browse/HBASE-20222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405031#comment-16405031
]
Hadoop QA commented on HBASE-20222:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-20222 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20222 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915147/HBASE-20222.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12017/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Bundle commons-collections
> --
>
> Key: HBASE-20222
> URL: https://issues.apache.org/jira/browse/HBASE-20222
> Project: HBase
> Issue Type: Task
> Components: thirdparty
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: thirdparty-2.0.1
>
> Attachments: HBASE-20222.001.patch
>
>
> commons-collections recently had some incompatibilities in their packaging
> (between 3 and 4). Let's include it in our shaded-misc jar to avoid any
> potential problems in the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20221) Upgrade protobuf to 3.5.1
[
https://issues.apache.org/jira/browse/HBASE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405032#comment-16405032
]
Hadoop QA commented on HBASE-20221:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-20221 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20221 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915146/HBASE-20221.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12018/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Upgrade protobuf to 3.5.1
> -
>
> Key: HBASE-20221
> URL: https://issues.apache.org/jira/browse/HBASE-20221
> Project: HBase
> Issue Type: Task
> Components: thirdparty
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: thirdparty-2.0.1
>
> Attachments: HBASE-20221.001.patch
>
>
> Get hbase2 onto the latest protobuf because it's there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20222) Bundle commons-collections
[ https://issues.apache.org/jira/browse/HBASE-20222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20222: --- Attachment: HBASE-20222.001.patch > Bundle commons-collections > -- > > Key: HBASE-20222 > URL: https://issues.apache.org/jira/browse/HBASE-20222 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.0.1 > > Attachments: HBASE-20222.001.patch > > > commons-collections recently had some incompatibilities in their packaging > (between 3 and 4). Let's include it in our shaded-misc jar to avoid any > potential problems in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20222) Bundle commons-collections
[ https://issues.apache.org/jira/browse/HBASE-20222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20222: --- Status: Patch Available (was: Open) > Bundle commons-collections > -- > > Key: HBASE-20222 > URL: https://issues.apache.org/jira/browse/HBASE-20222 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.0.1 > > Attachments: HBASE-20222.001.patch > > > commons-collections recently had some incompatibilities in their packaging > (between 3 and 4). Let's include it in our shaded-misc jar to avoid any > potential problems in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20221) Upgrade protobuf to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20221: --- Attachment: HBASE-20221.001.patch > Upgrade protobuf to 3.5.1 > - > > Key: HBASE-20221 > URL: https://issues.apache.org/jira/browse/HBASE-20221 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: thirdparty-2.0.1 > > Attachments: HBASE-20221.001.patch > > > Get hbase2 onto the latest protobuf because it's there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
