[jira] [Commented] (HBASE-20579) Improve copy snapshot manifest in ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480175#comment-16480175 ] Jingyun Tian commented on HBASE-20579: -- [~yuzhih...@gmail.com] Thanks for your review. Can you help commit this patch? > Improve copy snapshot manifest in ExportSnapshot > > > Key: HBASE-20579 > URL: https://issues.apache.org/jira/browse/HBASE-20579 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0, 1.5.0, 2.0.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Minor > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20579.master.001.patch, > HBASE-20579.master.002.patch > > > ExportSnapshot need to copy snapshot manifest to destination cluster first, > then setOwner and setPermission for those paths. But it's done with one > thread, which lead to a long time to submit the job if your snapshot is big. > I tried to make them processing in parallel, which can reduce the total time > of submitting dramatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480113#comment-16480113 ] Ted Yu commented on HBASE-20565: bq. so the x is unstable You mean the number of cells is unpredictable. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20579) Improve copy snapshot manifest in ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480109#comment-16480109 ] Ted Yu commented on HBASE-20579: lgtm > Improve copy snapshot manifest in ExportSnapshot > > > Key: HBASE-20579 > URL: https://issues.apache.org/jira/browse/HBASE-20579 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0, 1.5.0, 2.0.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Minor > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20579.master.001.patch, > HBASE-20579.master.002.patch > > > ExportSnapshot need to copy snapshot manifest to destination cluster first, > then setOwner and setPermission for those paths. But it's done with one > thread, which lead to a long time to submit the job if your snapshot is big. > I tried to make them processing in parallel, which can reduce the total time > of submitting dramatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20579) Improve copy snapshot manifest in ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480107#comment-16480107 ] Jingyun Tian commented on HBASE-20579: -- [~yuzhih...@gmail.com] Of course. But I test it using version 0.98. For a snapshot which has 115328 reference files, , it cost 3122 seconds to submit the job without the patch. And it cost 775 seconds with the patch. > Improve copy snapshot manifest in ExportSnapshot > > > Key: HBASE-20579 > URL: https://issues.apache.org/jira/browse/HBASE-20579 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0, 1.5.0, 2.0.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Minor > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20579.master.001.patch, > HBASE-20579.master.002.patch > > > ExportSnapshot need to copy snapshot manifest to destination cluster first, > then setOwner and setPermission for those paths. But it's done with one > thread, which lead to a long time to submit the job if your snapshot is big. > I tried to make them processing in parallel, which can reduce the total time > of submitting dramatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480099#comment-16480099 ] Zheng Hu commented on HBASE-20565: -- bq. Is the scenario limited to FilterListWithAND only ? No , FilterListWithOR has the similar problem too ... because currently the offset is neither an absolute offset nor relative offset, if we passed x cells to FilterList (WithAND or WithOR), then then the first offset cell will be abandoned, and start to include cell from the next , for ColumnPaginationFilter(offset, limit) but some times sub-filters return SEEK_NEXT_USING_HINT or some other RC, so the x is unstable, which means the offset is unstable (see [~jinghe]'s testing. ) > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20561) The way we stop a ReplicationSource may cause the RS down
[
https://issues.apache.org/jira/browse/HBASE-20561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480097#comment-16480097
]
Duo Zhang commented on HBASE-20561:
---
OK, checked the code, the SystemErrorException is thrown by us in ZKWatcher.
Then let's call initCause to save the InterruptedException?
> The way we stop a ReplicationSource may cause the RS down
> -
>
> Key: HBASE-20561
> URL: https://issues.apache.org/jira/browse/HBASE-20561
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-20561.master.001.patch,
> HBASE-20561.master.002.patch
>
>
> See this:
> https://builds.apache.org/job/HBASE-Flaky-Tests/31125/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL-output.txt
> {noformat}
> 2018-05-09 15:07:00,887 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.RefreshPeerCallable(52): Received a peer change event, peerId=2,
> type=REMOVE_PEER
> 2018-05-09 15:07:00,890 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.ReplicationSource(485): Closing source
> 2-asf916.gq1.ygridcore.net,36287,1525878368395 because: Replication stream
> was removed by a user
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-0,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:871)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.removeWAL(ZKReplicationQueueStorage.java:198)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$cleanOldLogs$8(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:454)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:506)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:489)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:231)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:133)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:103)
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-1,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:990)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:663)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1690)
> at
>
[jira] [Updated] (HBASE-20424) Allow writing WAL to local and remote cluster concurrently
[ https://issues.apache.org/jira/browse/HBASE-20424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20424: -- Parent Issue: HBASE-19064 (was: HBASE-20422) > Allow writing WAL to local and remote cluster concurrently > -- > > Key: HBASE-20424 > URL: https://issues.apache.org/jira/browse/HBASE-20424 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20424-HBASE-19064.patch > > > For better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20556) Backport HBASE-16490 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480088#comment-16480088
]
Hadoop QA commented on HBASE-20556:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} hbase-server in branch-1 failed with JDK v1.7.0_181.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_181.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_181. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} hbase-server: The patch generated 0 new + 279
unchanged - 1 fixed = 279 total (was 280) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
44s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m
57s{color} | {color:red} The patch causes 44 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
55s{color} | {color:red} The patch causes 44 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 26s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-20556 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12924039/HBASE-20556.branch-1.001.patch
|
| Optional Tests | asflicense javac
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480083#comment-16480083 ] Ted Yu commented on HBASE-20565: Is the scenario limited to FilterListWithAND only ? If so, we can consider turning off the max step rule for FilterListWithAND. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480080#comment-16480080 ] Zheng Hu commented on HBASE-20565: -- bq. Is it possible to formalize the condition(s) where Max Step Rule doesn't apply ? No so easy ? unless we abstract a method named shouldAppyMaxStepRule (or something) for Filter, and handle this in FilterListWithAND. Seems that it's not a good practice. In fact, FilterList concated by sub-filter-X and filter which maintain a offset will hit this bug ? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473738#comment-16473738 ] Zheng Hu edited comment on HBASE-20565 at 5/18/18 3:10 AM: --- It's complex now...if a FilterListWithAND has two sub filters A & B, and B has a row-level global state (such as column offset...), after HBASE-18410, we optimze the forward step to be maximum, so we consider the return code from sub-filters at global level, which means that a cell may not be included the sub-filter A, but we still need to pass the cell to sub-filter B for calculating B' return code (for global return code purpose), finally B's row-level global state messed up ... was (Author: openinx): It's complex now...if a FilterListWithAND has two sub filters A & B, and B has a row-level global state (such as column limit...), after HBASE-18410, we optimze the forward step to be maximum, so we consider the return code from sub-filters at global level, which means that a cell may not be included the sub-filter A, but we still need to pass the cell to sub-filter B for calculating B' return code (for global return code purpose), finally B's row-level global state messed up ... > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480069#comment-16480069 ] Ted Yu commented on HBASE-20565: bq. FilterList concated by ColumnRangeFilter & ColumnPaginationFilter (offset, limit) is in conflict with the Max Step Rule Is it possible to formalize the condition(s) where Max Step Rule doesn't apply ? If so, Max Step Rule can be turned off in those scenarios. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480064#comment-16480064 ] Zheng Hu commented on HBASE-20565: -- FilterList concated by ColumnRangeFilter & ColumnPaginationFilter (offset, limit) is in conflict with the Max Step Rule. but FilterList concated by ColumnRangeFilter & ColumnPaginationFilter(limit) or ColumnPaginationFilter(byte[], limit) has no problem.Does it make sense that setting an offset & limit for ColumnPaginationFilter ? This scene can be completely replaced by ColumnPaginationFilter(byte[], limit) ? [~anoop.hbase], how do you think ? > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20560) Revisit the TestReplicationDroppedTables ut
[ https://issues.apache.org/jira/browse/HBASE-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20560: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Revisit the TestReplicationDroppedTables ut > --- > > Key: HBASE-20560 > URL: https://issues.apache.org/jira/browse/HBASE-20560 > Project: HBase > Issue Type: Bug >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20560.patch, HBASE-20560.patch > > > After HBASE-20475, the ut TestReplicationDroppedTables has been more stable > now, but still failed in few times.. > Checked the code again, I found the problems: > 1. > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16465759=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16465759 > 2. > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16467225=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16467225 > So prepared a patch for the revisiting.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20560) Revisit the TestReplicationDroppedTables ut
[ https://issues.apache.org/jira/browse/HBASE-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480050#comment-16480050 ] Zheng Hu commented on HBASE-20560: -- The UT is stable now, closed this issue. > Revisit the TestReplicationDroppedTables ut > --- > > Key: HBASE-20560 > URL: https://issues.apache.org/jira/browse/HBASE-20560 > Project: HBase > Issue Type: Bug >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20560.patch, HBASE-20560.patch > > > After HBASE-20475, the ut TestReplicationDroppedTables has been more stable > now, but still failed in few times.. > Checked the code again, I found the problems: > 1. > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16465759=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16465759 > 2. > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16467225=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16467225 > So prepared a patch for the revisiting.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20446) Allow building HBase 1.x against Hadoop 3.1.0
[ https://issues.apache.org/jira/browse/HBASE-20446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480006#comment-16480006 ] Lars Hofhansl commented on HBASE-20446: --- I will push this to branch-1 only. > Allow building HBase 1.x against Hadoop 3.1.0 > - > > Key: HBASE-20446 > URL: https://issues.apache.org/jira/browse/HBASE-20446 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 1.5.0 > > Attachments: 20446.txt > > > Simple change, just leaving it here in case somebody needs this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20446) Allow building HBase 1.x against Hadoop 3.1.0
[ https://issues.apache.org/jira/browse/HBASE-20446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480005#comment-16480005 ] Lars Hofhansl commented on HBASE-20446: --- Thanks [~mdrob], good to know. There's no mention of that on the Hadoop release page (but it does say for - for example - 2.9.0). In fact for the 3.0.1 release it says that 3.0.0 is deprecated after that release (but then a 3.0.2 shows up two weeks later). I guess that referred to 3.0.0 specifically, due to the port change, and the 3.0 line. So the current stable release is 3.0.2? > Allow building HBase 1.x against Hadoop 3.1.0 > - > > Key: HBASE-20446 > URL: https://issues.apache.org/jira/browse/HBASE-20446 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 1.5.0 > > Attachments: 20446.txt > > > Simple change, just leaving it here in case somebody needs this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20556) Backport HBASE-16490 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479980#comment-16479980 ] Tak Lon (Stephen) Wu edited comment on HBASE-20556 at 5/18/18 1:23 AM: --- I submitted another patch for fixing the [checkstyle|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/diff-checkstyle-hbase-server.txt] failure, however, I found the failures in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] and [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] logs should not be related. # in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] logs, a very strange error jdk 7 incompatible with jdk 8 of `maven-compiler-plugin:3.6.1:compile: java.lang.UnsupportedClassVersionError: javax/tools/DiagnosticListener : Unsupported major.minor version 52.0`. I found other JIRAs that works on branch-1 has the same issue, did the maven version (3.6.1) has any problem with jdk 7 ?? or in the pom.xml (I didn't touch that) has any issue with jdk 7 ? this also happened on the local compile as well # in [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] log, all unit tests completed but at the end of the test process, I saw a lots of `ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?` I found this [surefire faq|http://maven.apache.org/surefire/maven-surefire-plugin/faq.html] and said system.exit() at any time is not supported (I also found some other JIRAs that works on branch-1 has this issue). So those Test classes called system.exit() should not be related to this change, but just be curious, does anyone know how to get it thru? was (Author: taklwu): I submitted another patch for fixing the checkstyle problem, however, I found the failures in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] and [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] logs should not be related. # in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] logs, a very strange error jdk 7 incompatible with jdk 8 of `maven-compiler-plugin:3.6.1:compile: java.lang.UnsupportedClassVersionError: javax/tools/DiagnosticListener : Unsupported major.minor version 52.0`. I found other JIRAs that works on branch-1 has the same issue, did the maven version (3.6.1) has any problem with jdk 7 ?? or in the pom.xml (I didn't touch that) has any issue with jdk 7 ? this also happened on the local compile as well # in [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] log, all unit tests completed but at the end of the test process, I saw a lots of `ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?` I found this [surefire faq|http://maven.apache.org/surefire/maven-surefire-plugin/faq.html] and said system.exit() at any time is not supported (I also found some other JIRAs that works on branch-1 has this issue). So those Test classes called system.exit() should not be related to this change, but just be curious, does anyone know how to get it thru? > Backport HBASE-16490 to branch-1 > > > Key: HBASE-20556 > URL: https://issues.apache.org/jira/browse/HBASE-20556 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20556.branch-1.001.patch, > HBASE-20556.branch-1.002.patch, HBASE-20556.branch-1.003.patch > > > As part of HBASE-20555, HBASE-16490 is the first patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20556) Backport HBASE-16490 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479980#comment-16479980 ] Tak Lon (Stephen) Wu edited comment on HBASE-20556 at 5/18/18 1:22 AM: --- I submitted another patch for fixing the checkstyle problem, however, I found the failures in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] and [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] logs should not be related. # in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] logs, a very strange error jdk 7 incompatible with jdk 8 of `maven-compiler-plugin:3.6.1:compile: java.lang.UnsupportedClassVersionError: javax/tools/DiagnosticListener : Unsupported major.minor version 52.0`. I found other JIRAs that works on branch-1 has the same issue, did the maven version (3.6.1) has any problem with jdk 7 ?? or in the pom.xml (I didn't touch that) has any issue with jdk 7 ? this also happened on the local compile as well # in [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] log, all unit tests completed but at the end of the test process, I saw a lots of `ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?` I found this [surefire faq|http://maven.apache.org/surefire/maven-surefire-plugin/faq.html] and said system.exit() at any time is not supported (I also found some other JIRAs that works on branch-1 has this issue). So those Test classes called system.exit() should not be related to this change, but just be curious, does anyone know how to get it thru? was (Author: taklwu): I submitted another patch for fixing the style, however, I found the failures in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] and [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] logs should not be related. # in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] logs, a very strange error jdk 7 incompatible with jdk 8 of `maven-compiler-plugin:3.6.1:compile: java.lang.UnsupportedClassVersionError: javax/tools/DiagnosticListener : Unsupported major.minor version 52.0`. I found other JIRAs that works on branch-1 has the same issue, did the maven version (3.6.1) has any problem with jdk 7 ?? or in the pom.xml (I didn't touch that) has any issue with jdk 7 ? this also happened on the local compile as well # in [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] log, all unit tests completed but at the end of the test process, I saw a lots of `ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?` I found this [surefire faq|http://maven.apache.org/surefire/maven-surefire-plugin/faq.html] and said system.exit() at any time is not supported (I also found some other JIRAs that works on branch-1 has this issue). So those Test classes called system.exit() should not be related to this change, but just be curious, does anyone know how to get it thru? > Backport HBASE-16490 to branch-1 > > > Key: HBASE-20556 > URL: https://issues.apache.org/jira/browse/HBASE-20556 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20556.branch-1.001.patch, > HBASE-20556.branch-1.002.patch, HBASE-20556.branch-1.003.patch > > > As part of HBASE-20555, HBASE-16490 is the first patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20556) Backport HBASE-16490 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479980#comment-16479980 ] Tak Lon (Stephen) Wu commented on HBASE-20556: -- I submitted another patch for fixing the style, however, I found the failures in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] and [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] logs should not be related. # in [complie|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_181.txt] logs, a very strange error jdk 7 incompatible with jdk 8 of `maven-compiler-plugin:3.6.1:compile: java.lang.UnsupportedClassVersionError: javax/tools/DiagnosticListener : Unsupported major.minor version 52.0`. I found other JIRAs that works on branch-1 has the same issue, did the maven version (3.6.1) has any problem with jdk 7 ?? or in the pom.xml (I didn't touch that) has any issue with jdk 7 ? this also happened on the local compile as well # in [unit|https://builds.apache.org/job/PreCommit-HBASE-Build/12854/artifact/patchprocess/patch-unit-hbase-server.txt] log, all unit tests completed but at the end of the test process, I saw a lots of `ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?` I found this [surefire faq|http://maven.apache.org/surefire/maven-surefire-plugin/faq.html] and said system.exit() at any time is not supported (I also found some other JIRAs that works on branch-1 has this issue). So those Test classes called system.exit() should not be related to this change, but just be curious, does anyone know how to get it thru? > Backport HBASE-16490 to branch-1 > > > Key: HBASE-20556 > URL: https://issues.apache.org/jira/browse/HBASE-20556 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20556.branch-1.001.patch, > HBASE-20556.branch-1.002.patch, HBASE-20556.branch-1.003.patch > > > As part of HBASE-20555, HBASE-16490 is the first patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20556) Backport HBASE-16490 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20556: - Attachment: HBASE-20556.branch-1.001.patch > Backport HBASE-16490 to branch-1 > > > Key: HBASE-20556 > URL: https://issues.apache.org/jira/browse/HBASE-20556 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20556.branch-1.001.patch, > HBASE-20556.branch-1.002.patch, HBASE-20556.branch-1.003.patch > > > As part of HBASE-20555, HBASE-16490 is the first patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20552) HBase RegionServer was shutdown due to UnexpectedStateException
[
https://issues.apache.org/jira/browse/HBASE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479955#comment-16479955
]
stack commented on HBASE-20552:
---
Long shot: Something up w/ your HDFS over there [~elserj] and crew where lease
recovery is dropping the end of the WAL? Doing some bad math on file length?
It does it for two different logs here... The Master WAL Proc and the
regionserver hosting hbase:meta's WAL. Some interesting version of HDFS? Thanks.
> HBase RegionServer was shutdown due to UnexpectedStateException
> ---
>
> Key: HBASE-20552
> URL: https://issues.apache.org/jira/browse/HBASE-20552
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Romil Choksi
>Assignee: Umesh Agashe
>Priority: Critical
> Attachments:
> 102143-master-ctr-e138-1518143905142-279227-01-03.hwx.site.log,
> 102143-master-ctr-e138-1518143905142-279227-01-05.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-02.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-07.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-08.hwx.site.log
>
>
> This was observed during cluster testing (source code sync'ed with hbase-2.0,
> built May 2nd):
> {code}
> 2018-05-02 05:44:10,089 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=2]
> master.MasterRpcServices: Region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474 reported
> a fatal error:
> * ABORTING region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN, location=ctr-e138-
> 1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on server=ctr-e138-
> 1518143905142-279227-01-02.hwx.site,16020,1525239334474 but state has
> otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:459)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException:
> rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1037)
> ... 7 more
> *
> Cause:
> org.apache.hadoop.hbase.YouAreDeadException:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
>table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:459)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException:
> rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
>
[jira] [Commented] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479941#comment-16479941
]
Hadoop QA commented on HBASE-20588:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 8s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20588 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12924010/HBASE-20588.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6f59f163e0cc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f917f4e7c8 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12868/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12868/testReport/ |
| Max. process+thread count | 4524 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12868/console |
| Powered by | Apache
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479910#comment-16479910
]
Hudson commented on HBASE-20582:
Results for branch branch-2
[build #748 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/748/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/748//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/748//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/748//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479897#comment-16479897
]
Hadoop QA commented on HBASE-20588:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 40s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}102m
11s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20588 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12924007/HBASE-20588.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a8529fc13918 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f917f4e7c8 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12867/testReport/ |
| Max. process+thread count | 4498 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12867/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Space quota change after
[jira] [Commented] (HBASE-20552) HBase RegionServer was shutdown due to UnexpectedStateException
[
https://issues.apache.org/jira/browse/HBASE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479831#comment-16479831
]
Umesh Agashe commented on HBASE-20552:
--
[~elserj], I don't have a repro. I thought I had a repro but it was due to the
bug which was inadvertently introduced in recent commit and got fixed in
addendum (HBASE-20564). So far I found 2 instances of missing edits around the
same time. First, in master proc wal where 003 is not able to read pids 468
onwards. And second, in meta region:
pid=475 on 005 started with:
{code:java}
2018-05-02 05:39:45,811 INFO [PEWorker-6] assignment.AssignProcedure: Starting
pid=475, ppid=471, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=test_hbase_ha_load_test_tool_hbase,
region=94f6ca283dbb4445b2bcdc321b734d28; rit=OFFLINE,
location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525238558502;
forceNewPlan=false, retain=true
{code}
After this it was updated twice on 005:
{code:java}
2018-05-02 05:39:45,983 INFO [PEWorker-1] assignment.RegionStateStore: pid=475
updating hbase:meta row=94f6ca283dbb4445b2bcdc321b734d28, regionState=OPENING
2018-05-02 05:39:46,580 INFO [PEWorker-1] assignment.RegionStateStore: pid=475
updating hbase:meta row=94f6ca283dbb4445b2bcdc321b734d28, regionState=OPEN,
openSeqNum=13401,
regionLocation=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
{code}
But when 003 read and printed meta, it has:
{code:java}
2018-05-02 05:44:08,236 INFO
[master/ctr-e138-1518143905142-279227-01-03:2]
assignment.RegionStateStore: Load hbase:meta entry
region=94f6ca283dbb4445b2bcdc321b734d28, regionState=OPEN,
lastHost=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474,
regionLocation=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525238558502
{code}
The location server including timestamp matches to when pid=471 started
"location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525238558502".
So 2 updates from pid=471 to meta are missing.
> HBase RegionServer was shutdown due to UnexpectedStateException
> ---
>
> Key: HBASE-20552
> URL: https://issues.apache.org/jira/browse/HBASE-20552
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Romil Choksi
>Assignee: Umesh Agashe
>Priority: Critical
> Attachments:
> 102143-master-ctr-e138-1518143905142-279227-01-03.hwx.site.log,
> 102143-master-ctr-e138-1518143905142-279227-01-05.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-02.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-07.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-08.hwx.site.log
>
>
> This was observed during cluster testing (source code sync'ed with hbase-2.0,
> built May 2nd):
> {code}
> 2018-05-02 05:44:10,089 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=2]
> master.MasterRpcServices: Region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474 reported
> a fatal error:
> * ABORTING region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN, location=ctr-e138-
> 1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on server=ctr-e138-
> 1518143905142-279227-01-02.hwx.site,16020,1525239334474 but state has
> otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:459)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException:
> rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
>
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20588:
---
Attachment: (was: HBASE-20588.master.001.patch)
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch,
> HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479787#comment-16479787
]
Ted Yu commented on HBASE-20588:
Running the new test without fix:
{code}
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_WRITES Puts are
disallowed due to a space quota.
at
org.apache.hadoop.hbase.quotas.policies.NoWritesViolationPolicyEnforcement.check(NoWritesViolationPolicyEnforcement.java:46)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2801)
{code}
Is it possible to add a test which shows the fix for 'NO_INSERTS Puts' error
shown in description ?
Thanks
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch,
> HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through {{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info0:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479757#comment-16479757
]
Nihal Jain commented on HBASE-20588:
Resubmitted patch after removing redundant looping over
currentSnapshots.entrySet().
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch,
> HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20588:
---
Attachment: HBASE-20588.master.001.patch
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch,
> HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20552) HBase RegionServer was shutdown due to UnexpectedStateException
[
https://issues.apache.org/jira/browse/HBASE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479767#comment-16479767
]
Josh Elser commented on HBASE-20552:
Trying to help pick this one up too..
{quote}bq. On M005, pid=471 is SCP for R007 which also hosts meta. Meta is
re-assigned with pid=472 to R002 which is followed by other region assignments
{quote}
I'm coming to think this is our problem, too.
pid=471 is an SCP for r007 from pv2-004.log which finished at 05:39:47,288 on
m005. When m003 takes over and reads the tracker from pv2-002.log, the largest
pid we have is pid=467.
My hunch (which I need to back up with code) is that because m003 never sees
the completed SCP, it thinks that r002 is holding this region (overriding what
meta say, maybe?), claiming it to be on r007 instead. The following is the
"largest" proc from the pv2-004 log that m003 reads.
{noformat}
2018-05-02 05:43:33,876 DEBUG
[master/ctr-e138-1518143905142-279227-01-03:2]
procedure2.ProcedureExecutor: Completed pid=465, state=SUCCESS;
MoveRegionProcedure hri=94f6ca283dbb4445b2bcdc321b734d28,
source=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474,
destination=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525238558502{noformat}
Then, m003 initializes RegionStateStore, saying:
{noformat}
2018-05-02 05:44:08,236 INFO
[master/ctr-e138-1518143905142-279227-01-03:2]
assignment.RegionStateStore: Load hbase:meta entry
region=94f6ca283dbb4445b2bcdc321b734d28, regionState=OPEN,
lastHost=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474,
regionLocation=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525238558502
{noformat}
This makes me wonder if Umesh's findings about pid=507 (SCP for r007 putting
the region back on r007) are related...
You get anywhere on a repro, [~uagashe]? I have some nodes running through this
internal scenario which has triggered this before. Might try my hand at
repro'ing in an IT, but unsure how hard that will be ;)
> HBase RegionServer was shutdown due to UnexpectedStateException
> ---
>
> Key: HBASE-20552
> URL: https://issues.apache.org/jira/browse/HBASE-20552
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Romil Choksi
>Assignee: Umesh Agashe
>Priority: Critical
> Attachments:
> 102143-master-ctr-e138-1518143905142-279227-01-03.hwx.site.log,
> 102143-master-ctr-e138-1518143905142-279227-01-05.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-02.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-07.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-08.hwx.site.log
>
>
> This was observed during cluster testing (source code sync'ed with hbase-2.0,
> built May 2nd):
> {code}
> 2018-05-02 05:44:10,089 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=2]
> master.MasterRpcServices: Region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474 reported
> a fatal error:
> * ABORTING region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN, location=ctr-e138-
> 1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on server=ctr-e138-
> 1518143905142-279227-01-02.hwx.site,16020,1525239334474 but state has
> otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:459)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException:
> rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
>
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20588:
---
Attachment: HBASE-20588.master.001.patch
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch,
> HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20588:
---
Description:
Steps followed
- Through {{hbase shell}}
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
NO_INSERTS{noformat}
- Run {{PE}} until the quota is reached
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=2000
sequentialWrite 1{noformat}
- Through {{HBase}} shell
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
- Through {{HBase}} shell verify the effective Quotas
{noformat}
> list_quotas
OWNER QUOTAS
0 row(s)
Took 0.0365 seconds{noformat}
- Wait for some time (at least 5 mins) and try to add data to the table
{noformat}
> put 'TestTable','r1','info0:0','v1'
ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
are disallowed due to a space quota.
at
org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
To resolve the issue, {{RSes}} need to be restarted which points to in memory
data not getting reset.
was:
Steps followed
- Through \{{hbase shell}}
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
NO_INSERTS{noformat}
- Run {{PE}} until the quota is reached
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=2000
sequentialWrite 1{noformat}
- Through {{HBase}} shell
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
- Through {{HBase}} shell verify the effective Quotas
{noformat}
> list_quotas
OWNER QUOTAS
0 row(s)
Took 0.0365 seconds{noformat}
- Wait for some time (at least 5 mins) and try to add data to the table
{noformat}
> put 'TestTable','r1','info:0','v1'
ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
are disallowed due to a space quota.
at
org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
To resolve the issue, {{RSes}} need to be restarted which points to in memory
data not getting reset.
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch,
> HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through {{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info0:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20588:
---
Fix Version/s: 3.0.0
Status: Patch Available (was: Open)
It seems we did not disable violation policy whenever quota was removed from
any table.
Submitted a fix, please review!
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20588:
---
Attachment: HBASE-20588.master.001.patch
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Priority: Major
> Attachments: HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain reassigned HBASE-20588:
--
Assignee: Nihal Jain
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Attachments: HBASE-20588.master.001.patch
>
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-20588:
--
Description:
Steps followed
- Through \{{hbase shell}}
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
NO_INSERTS{noformat}
- Run {{PE}} until the quota is reached
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=2000
sequentialWrite 1{noformat}
- Through {{HBase}} shell
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
- Through {{HBase}} shell verify the effective Quotas
{noformat}
> list_quotas
OWNER QUOTAS
0 row(s)
Took 0.0365 seconds{noformat}
- Wait for some time (at least 5 mins) and try to add data to the table
{noformat}
> put 'TestTable','r1','info:0','v1'
ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
are disallowed due to a space quota.
at
org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
To resolve the issue, {{RSes}} need to be restarted which points to in memory
data not getting reset.
was:
Steps followed
- Through h{{base shell}}
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
NO_INSERTS{noformat}
- Run {{PE}} until the quota is reached
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=2000
sequentialWrite 1{noformat}
- Through {{HBase}} shell
{noformat}
set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
- Through {{HBase}} shell verify the effective Quotas
{noformat}
> list_quotas
OWNER QUOTAS
0 row(s)
Took 0.0365 seconds{noformat}
- Wait for some time (at least 5 mins) and try to add data to the table
{noformat}
> put 'TestTable','r1','info:0','v1'
ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
are disallowed due to a space quota.
at
org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
To resolve the issue, {{RSes}} need to be restarted which points to in memory
data not getting reset.
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Priority: Major
>
> Steps followed
> - Through \{{hbase shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-13120) Allow disabling hadoop classpath and native library lookup
[
https://issues.apache.org/jira/browse/HBASE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-13120:
---
Assignee: Siddharth Wagle
> Allow disabling hadoop classpath and native library lookup
> --
>
> Key: HBASE-13120
> URL: https://issues.apache.org/jira/browse/HBASE-13120
> Project: HBase
> Issue Type: Improvement
> Components: hadoop2, scripts
>Affects Versions: 0.98.0
>Reporter: Siddharth Wagle
>Assignee: Siddharth Wagle
>Priority: Major
> Fix For: 1.0.1, 1.1.0, 0.98.12, 2.0.0
>
> Attachments: HBASE-13120-1.patch, HBASE-13120.patch
>
>
> - Current bin/hbase script sets the java.library.path to include hadoop
> native libs based on what version of hadoop is installed on the box.
> {code}
> HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" which
> hadoop 2>/dev/null)
> {code}
> - This effectively means that a self-contained HBase running with a different
> version of embedded hadoop jars will fail to work in case of version
> incompatibilities.
> - Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with
> cluster Hadoop installation.
> - It would be good to have a hbase-env variable to completely override native
> lib path or a config to disable native lib path lookup, in which case user
> has to provide it during start.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-13120) Allow disabling hadoop classpath and native library lookup
[
https://issues.apache.org/jira/browse/HBASE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13120:
Component/s: (was: hbase)
scripts
hadoop2
> Allow disabling hadoop classpath and native library lookup
> --
>
> Key: HBASE-13120
> URL: https://issues.apache.org/jira/browse/HBASE-13120
> Project: HBase
> Issue Type: Improvement
> Components: hadoop2, scripts
>Affects Versions: 0.98.0
>Reporter: Siddharth Wagle
>Priority: Major
> Fix For: 1.0.1, 1.1.0, 0.98.12, 2.0.0
>
> Attachments: HBASE-13120-1.patch, HBASE-13120.patch
>
>
> - Current bin/hbase script sets the java.library.path to include hadoop
> native libs based on what version of hadoop is installed on the box.
> {code}
> HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" which
> hadoop 2>/dev/null)
> {code}
> - This effectively means that a self-contained HBase running with a different
> version of embedded hadoop jars will fail to work in case of version
> incompatibilities.
> - Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with
> cluster Hadoop installation.
> - It would be good to have a hbase-env variable to completely override native
> lib path or a config to disable native lib path lookup, in which case user
> has to provide it during start.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20600) Loosing HBase data when region server goes down
Nandakishore created HBASE-20600: Summary: Loosing HBase data when region server goes down Key: HBASE-20600 URL: https://issues.apache.org/jira/browse/HBASE-20600 Project: HBase Issue Type: Bug Reporter: Nandakishore Hi Team, I have a issue with HBase cluster. I have hosted an HBase cluster with Phoenix on EMR emr-5.8.0 and storage as S3. I have 1 master and 5 slaves 4.x large nodes. I’m losing the data while querying a table after a region server dies. Here are the steps I followed. 1. Launched the cluster with dfs replication factory as 3. 2. Created the tables and loaded the data using Phoenix. 3. Cross checked the data I loaded into tables and I see the data. 4. Wantedly terminated a EC2 machine which is part of cluster, meaning killing region server. 5. I could see EMR resizing and bringing up the new node. 6. When I query the table after the whole cluster is stable, which usually took 5-10 minutes, I see losing some data which is on dead RS. Could you please let me know what possibly could go wrong. Also please let me know if I have to set any properties. I would be happy to provide more details if you need. Thanks, Nandakishore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20588) Space quota change after quota violation doesn't seem to take in effect
[
https://issues.apache.org/jira/browse/HBASE-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479671#comment-16479671
]
Nihal Jain commented on HBASE-20588:
Hi, I would like to work on this
> Space quota change after quota violation doesn't seem to take in effect
> ---
>
> Key: HBASE-20588
> URL: https://issues.apache.org/jira/browse/HBASE-20588
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Biju Nair
>Priority: Major
>
> Steps followed
> - Through h{{base shell}}
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => '2M', POLICY =>
> NO_INSERTS{noformat}
> - Run {{PE}} until the quota is reached
> {noformat}
> hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
> --rows=2000 sequentialWrite 1{noformat}
> - Through {{HBase}} shell
> {noformat}
> set_quota TYPE => SPACE, TABLE => 'TestTable', LIMIT => NONE{noformat}
> - Through {{HBase}} shell verify the effective Quotas
> {noformat}
> > list_quotas
> OWNER QUOTAS
>
>
> 0 row(s)
> Took 0.0365 seconds{noformat}
> - Wait for some time (at least 5 mins) and try to add data to the table
> {noformat}
> > put 'TestTable','r1','info:0','v1'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_INSERTS Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoInsertsViolationPolicyEnforcement.check(NoInsertsViolationPolicyEnforcement.java:47){noformat}
> To resolve the issue, {{RSes}} need to be restarted which points to in memory
> data not getting reset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479672#comment-16479672 ] Andrew Purtell commented on HBASE-20597: The findbugs warning is not due to this patch. The TestSimpleRpcScheduler failure is not related. (Cannot possibly be) > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20598) Upgrade to JRuby 9.2
[ https://issues.apache.org/jira/browse/HBASE-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479666#comment-16479666 ] Sean Busbey commented on HBASE-20598: - whoever takes this on: please doc supported ruby version changes, any changes to bundled gems, any license adjustments, and if/how the jruby community denotes "early access" vs "production ready" releases. > Upgrade to JRuby 9.2 > > > Key: HBASE-20598 > URL: https://issues.apache.org/jira/browse/HBASE-20598 > Project: HBase > Issue Type: Bug > Components: dependencies, shell >Reporter: Josh Elser >Priority: Major > Fix For: 3.0.0 > > > [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we > can get ourselves onto that from our current 9.1 release line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479650#comment-16479650
]
Hadoop QA commented on HBASE-20597:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 40s{color}
| {color:red} hbase-server generated 5 new + 183 unchanged - 5 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} hbase-server: The patch generated 0 new + 2
unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 16s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
56s{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed =
2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 44s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of
org.apache.hadoop.hbase.replication.HBaseReplicationEndpoint.regionServers;
locked 50% of time Unsynchronized access at HBaseReplicationEndpoint.java:50%
of time Unsynchronized access at HBaseReplicationEndpoint.java:[line 200] |
| |
org.apache.hadoop.hbase.replication.HBaseReplicationEndpoint.getRegionServers()
is unsynchronized,
org.apache.hadoop.hbase.replication.HBaseReplicationEndpoint.setRegionServers(List)
is synchronized At HBaseReplicationEndpoint.java:synchronized At
HBaseReplicationEndpoint.java:[lines 191-200] |
| Failed junit tests | hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20597 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12923971/HBASE-20597.patch |
| Optional Tests |
[jira] [Commented] (HBASE-20571) JMXJsonServlet generates invalid JSON if it has NaN in metrics
[
https://issues.apache.org/jira/browse/HBASE-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479646#comment-16479646
]
Hudson commented on HBASE-20571:
Results for branch branch-1.4
[build #324 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> JMXJsonServlet generates invalid JSON if it has NaN in metrics
> --
>
> Key: HBASE-20571
> URL: https://issues.apache.org/jira/browse/HBASE-20571
> Project: HBase
> Issue Type: Bug
> Components: UI
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20571.branch-1.003.patch,
> HBASE-20571.branch-1.004.patch, HBASE-20571.branch-2.0.001.patch,
> HBASE-20571.branch-2.0.002.patch, HBASE-20571.branch-2.0.002.patch,
> HBASE-20571.branch-2.0.003.patch
>
>
> {{/jmx}} servlet responses invalid JSON, if some metrics are NaN:
> {code}
> "l1CacheHitCount" : 0,
> "l1CacheMissCount" : 0,
> "l1CacheHitRatio" : NaN,
> "l1CacheMissRatio" : NaN,
> "l2CacheHitCount" : 0,
> "l2CacheMissCount" : 0,
> "l2CacheHitRatio" : 0.0,
> "l2CacheMissRatio" : 0.0,
> {code}
> NaN is an invalid character sequence in JSON. We should not response NaN in
> metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20500) [rsgroup] should keep at least one server in default group
[
https://issues.apache.org/jira/browse/HBASE-20500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479647#comment-16479647
]
Hudson commented on HBASE-20500:
Results for branch branch-1.4
[build #324 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/324//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [rsgroup] should keep at least one server in default group
> --
>
> Key: HBASE-20500
> URL: https://issues.apache.org/jira/browse/HBASE-20500
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5
>
> Attachments: HBASE-20500-branch-2.v1.patch,
> HBASE-20500-master.v1.patch, HBASE-20500-master.v2.patch,
> HBASE-20500-master.v3.patch, HBASE-20500-master.v4.patch,
> HBASE-20500-master.v5.patch
>
>
> we move all the servers from default group
> the default group will has no servers,
> then we create a new table ,
> it will failed case of the default group has no servers
> eorr info is :
> EROOR: ConstraintException:Target RSGroup must have at lease on server
>
> we should keep at least one server in 'default' RSGroup
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20581) HBase book documentation wrong for REST operations on schema endpoints
[
https://issues.apache.org/jira/browse/HBASE-20581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20581:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> HBase book documentation wrong for REST operations on schema endpoints
> --
>
> Key: HBASE-20581
> URL: https://issues.apache.org/jira/browse/HBASE-20581
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Kevin Risden
>Assignee: Josh Elser
>Priority: Critical
> Attachments: HBASE-20581.001.patch
>
>
> On [https://hbase.apache.org/book.html#_using_rest_endpoints]
> The documentation states that to update a table schema (the configuration for
> a column family), the {{PUT}} HTTP verb will update the current configuration
> with the "fragment" of configuration provided, while the {{POST}} HTTP verb
> will replace the current configuration with whatever is provided.
> In reality, the opposite is true: {{POST}} updates the configuration, {{PUT}}
> replaces. The old javadoc for the o.a.h.h.rest package got it right, but the
> entry on the HBase book transposed this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20581) HBase book documentation wrong for REST operations on schema endpoints
[
https://issues.apache.org/jira/browse/HBASE-20581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20581:
---
Fix Version/s: 3.0.0
> HBase book documentation wrong for REST operations on schema endpoints
> --
>
> Key: HBASE-20581
> URL: https://issues.apache.org/jira/browse/HBASE-20581
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Kevin Risden
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20581.001.patch
>
>
> On [https://hbase.apache.org/book.html#_using_rest_endpoints]
> The documentation states that to update a table schema (the configuration for
> a column family), the {{PUT}} HTTP verb will update the current configuration
> with the "fragment" of configuration provided, while the {{POST}} HTTP verb
> will replace the current configuration with whatever is provided.
> In reality, the opposite is true: {{POST}} updates the configuration, {{PUT}}
> replaces. The old javadoc for the o.a.h.h.rest package got it right, but the
> entry on the HBase book transposed this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20591) nightly job doesn't respect maven options
[ https://issues.apache.org/jira/browse/HBASE-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479584#comment-16479584 ] Sean Busbey commented on HBASE-20591: - bump for review. nightly in master last night failed again with this issue. > nightly job doesn't respect maven options > - > > Key: HBASE-20591 > URL: https://issues.apache.org/jira/browse/HBASE-20591 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Attachments: HBASE-20591.0.patch > > > some recent nightly runs have had failures that look like HBASE-20068, but at > the maven install step. > looks like our yetus runs are ignoring the "use build specific maven repos" > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Implement a meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479544#comment-16479544
]
Andrew Purtell commented on HBASE-19722:
Latest patch looks much better.
This looks like a typo:
{code}
getReginoIdFromOp
{code}
Do you think {{getTableNameFromOp}}, {{getRegionIdFromOp}}, and
{{isMetaTableOp}} are generally useful and should go into the client API as
static helper methods? They look useful. Is Object the common base class or is
there something higher up in the hierarchy that is still common like
{{Operation}}? (Just a thought, I didn't look.)
When using the String constructors to make strings from byte[] should specify a
charset. We use UTF_8 throughout HBase.
I'm a little concerned about the allocation demand of tracking every client, of
perhaps many thousands, instead of top-N. However trying to do top-N would be
complicated. We'd need to keep all the state to do top-N around and frequently
clean up stale entries when the set of top-N changes. So I think all of that
can be deferred until later. We can try this out as is first.
> Implement a meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.master.010.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20597:
---
Status: Patch Available (was: Open)
I ran {{mvn test -Dtest=\*Replication\*}} before posting the patches and
everything passed.
> Use a lock to serialize access to a shared reference to ZooKeeperWatcher in
> HBaseReplicationEndpoint
>
>
> Key: HBASE-20597
> URL: https://issues.apache.org/jira/browse/HBASE-20597
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.4, 1.3.2
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20597-branch-1.patch, HBASE-20597.patch
>
>
> The code that closes down a ZKW that fails to initialize when attempting to
> connect to the remote cluster is not MT safe and can in theory leak
> ZooKeeperWatcher instances. The allocation of a new ZKW and store to the
> reference is not atomic. Might have concurrent allocations with only one
> winning store, leading to leaked ZKW instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20581) HBase book documentation wrong for REST operations on schema endpoints
[
https://issues.apache.org/jira/browse/HBASE-20581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479522#comment-16479522
]
Josh Elser commented on HBASE-20581:
Thanks, [~balazs.meszaros], let me push this before I forget (again) :)
> HBase book documentation wrong for REST operations on schema endpoints
> --
>
> Key: HBASE-20581
> URL: https://issues.apache.org/jira/browse/HBASE-20581
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Kevin Risden
>Assignee: Josh Elser
>Priority: Critical
> Attachments: HBASE-20581.001.patch
>
>
> On [https://hbase.apache.org/book.html#_using_rest_endpoints]
> The documentation states that to update a table schema (the configuration for
> a column family), the {{PUT}} HTTP verb will update the current configuration
> with the "fragment" of configuration provided, while the {{POST}} HTTP verb
> will replace the current configuration with whatever is provided.
> In reality, the opposite is true: {{POST}} updates the configuration, {{PUT}}
> replaces. The old javadoc for the o.a.h.h.rest package got it right, but the
> entry on the HBase book transposed this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20597: --- Attachment: HBASE-20597.patch HBASE-20597-branch-1.patch > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20599) NPE in HRegionServer.reportFileArchivalForQuotas
[
https://issues.apache.org/jira/browse/HBASE-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved HBASE-20599.
Resolution: Duplicate
Fix Version/s: (was: 3.0.0)
Yeesh, I'm striking out today. Dupe of HBASE-20531
> NPE in HRegionServer.reportFileArchivalForQuotas
>
>
> Key: HBASE-20599
> URL: https://issues.apache.org/jira/browse/HBASE-20599
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
>
> Straightforward NPE while a RS is going down:
> {noformat}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1622)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1450)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3688)
> at
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2691)
> at
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2622)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:902)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1600)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1597)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more{noformat}
> FYI [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20599) NPE in HRegionServer.reportFileArchivalForQuotas
[
https://issues.apache.org/jira/browse/HBASE-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479489#comment-16479489
]
Josh Elser commented on HBASE-20599:
Oops, my bad, Stack. This is on master, not branch-2 stuff. Some of the
lower-latency space quota improvements didn't land on 2.x
> NPE in HRegionServer.reportFileArchivalForQuotas
>
>
> Key: HBASE-20599
> URL: https://issues.apache.org/jira/browse/HBASE-20599
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0
>
>
> Straightforward NPE while a RS is going down:
> {noformat}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1622)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1450)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3688)
> at
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2691)
> at
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2622)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:902)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1600)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1597)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more{noformat}
> FYI [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20599) NPE in HRegionServer.reportFileArchivalForQuotas
[
https://issues.apache.org/jira/browse/HBASE-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20599:
---
Fix Version/s: (was: 2.0.1)
(was: 2.1.0)
3.0.0
> NPE in HRegionServer.reportFileArchivalForQuotas
>
>
> Key: HBASE-20599
> URL: https://issues.apache.org/jira/browse/HBASE-20599
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0
>
>
> Straightforward NPE while a RS is going down:
> {noformat}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1622)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1450)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3688)
> at
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2691)
> at
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2622)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:902)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1600)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1597)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more{noformat}
> FYI [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20599) NPE in HRegionServer.reportFileArchivalForQuotas
Josh Elser created HBASE-20599:
--
Summary: NPE in HRegionServer.reportFileArchivalForQuotas
Key: HBASE-20599
URL: https://issues.apache.org/jira/browse/HBASE-20599
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.1.0, 2.0.1
Straightforward NPE while a RS is going down:
{noformat}
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1622)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1450)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3688)
at
org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2691)
at
org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2622)
at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:902)
at
org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1600)
at
org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1597)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more{noformat}
FYI [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20595) Remove the concept of 'special tables' from rsgroups
[ https://issues.apache.org/jira/browse/HBASE-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479467#comment-16479467 ] Andrew Purtell commented on HBASE-20595: Never mind what I said above about another special rsgroup for system tables. I think we just need to set a flag if a rsgroup has a system table in it and guarantee that group always has at least one server. That works for a wider range of cases including the single node case. > Remove the concept of 'special tables' from rsgroups > > > Key: HBASE-20595 > URL: https://issues.apache.org/jira/browse/HBASE-20595 > Project: HBase > Issue Type: Task > Components: Region Assignment, rsgroup >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > > Regionserver groups needs to specially handle what it calls "special tables", > tables upon which core or other modular functionality depends. They need to > be excluded from normal rsgroup processing during bootstrap to avoid circular > dependencies or errors due to insufficiently initialized state. I think we > also want to ensure that such tables are always given a rsgroup assignment > with nonzero servers. (IIRC another issue already raises that point, we can > link it later.) > Special tables include: > * The system tables in the 'hbase:' namespace > * The ACL table if the AccessController coprocessor is installed > * The Labels table if the VisibilityController coprocessor is installed > * The Quotas table if the FS quotas feature is active > Either we need a facility where "special tables" can be registered, which > should be in core. Or, we institute a blanket rule that core and all > extensions that need a "special table" must put them into the 'hbase:' > namespace, so the TableName#isSystemTable() test will return TRUE for all, > and then rsgroups simply needs to test for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20595) Remove the concept of 'special tables' from rsgroups
[ https://issues.apache.org/jira/browse/HBASE-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479457#comment-16479457 ] Andrew Purtell edited comment on HBASE-20595 at 5/17/18 5:53 PM: - bq. Or, we institute a blanket rule that core and all extensions that need a "special table" must put them into the 'hbase:' namespace, so the TableName#isSystemTable() test will return TRUE for all, and then rsgroups simply needs to test for that. This is what I am thinking as far as how we achieve removal of the 'special tables' concept. Special == system. I haven't looked at all of the places where we create such tables though to determine if it is a compat problem, a move of the table to the system namespace. Pretty sure the security coprocessors are fine. Quotas is only in trunk so that would be ok too. Anyway, any objections to this? was (Author: apurtell): bq. Or, we institute a blanket rule that core and all extensions that need a "special table" must put them into the 'hbase:' namespace, so the TableName#isSystemTable() test will return TRUE for all, and then rsgroups simply needs to test for that. This is what I am thinking as far as how we achieve removal of the 'special tables' concept. Special == system. I haven't looked at all of the places where we create such tables though to determine if it is a compat problem. Pretty sure the security coprocessors are fine. Quotas is only in trunk so that would be ok too. Anyway, any objections to this? > Remove the concept of 'special tables' from rsgroups > > > Key: HBASE-20595 > URL: https://issues.apache.org/jira/browse/HBASE-20595 > Project: HBase > Issue Type: Task > Components: Region Assignment, rsgroup >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > > Regionserver groups needs to specially handle what it calls "special tables", > tables upon which core or other modular functionality depends. They need to > be excluded from normal rsgroup processing during bootstrap to avoid circular > dependencies or errors due to insufficiently initialized state. I think we > also want to ensure that such tables are always given a rsgroup assignment > with nonzero servers. (IIRC another issue already raises that point, we can > link it later.) > Special tables include: > * The system tables in the 'hbase:' namespace > * The ACL table if the AccessController coprocessor is installed > * The Labels table if the VisibilityController coprocessor is installed > * The Quotas table if the FS quotas feature is active > Either we need a facility where "special tables" can be registered, which > should be in core. Or, we institute a blanket rule that core and all > extensions that need a "special table" must put them into the 'hbase:' > namespace, so the TableName#isSystemTable() test will return TRUE for all, > and then rsgroups simply needs to test for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20595) Remove the concept of 'special tables' from rsgroups
[ https://issues.apache.org/jira/browse/HBASE-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479457#comment-16479457 ] Andrew Purtell commented on HBASE-20595: bq. Or, we institute a blanket rule that core and all extensions that need a "special table" must put them into the 'hbase:' namespace, so the TableName#isSystemTable() test will return TRUE for all, and then rsgroups simply needs to test for that. This is what I am thinking as far as how we achieve removal of the 'special tables' concept. Special == system. I haven't looked at all of the places where we create such tables though to determine if it is a compat problem. Pretty sure the security coprocessors are fine. Quotas is only in trunk so that would be ok too. Anyway, any objections to this? > Remove the concept of 'special tables' from rsgroups > > > Key: HBASE-20595 > URL: https://issues.apache.org/jira/browse/HBASE-20595 > Project: HBase > Issue Type: Task > Components: Region Assignment, rsgroup >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > > Regionserver groups needs to specially handle what it calls "special tables", > tables upon which core or other modular functionality depends. They need to > be excluded from normal rsgroup processing during bootstrap to avoid circular > dependencies or errors due to insufficiently initialized state. I think we > also want to ensure that such tables are always given a rsgroup assignment > with nonzero servers. (IIRC another issue already raises that point, we can > link it later.) > Special tables include: > * The system tables in the 'hbase:' namespace > * The ACL table if the AccessController coprocessor is installed > * The Labels table if the VisibilityController coprocessor is installed > * The Quotas table if the FS quotas feature is active > Either we need a facility where "special tables" can be registered, which > should be in core. Or, we institute a blanket rule that core and all > extensions that need a "special table" must put them into the 'hbase:' > namespace, so the TableName#isSystemTable() test will return TRUE for all, > and then rsgroups simply needs to test for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20595) Remove the concept of 'special tables' from rsgroups
[ https://issues.apache.org/jira/browse/HBASE-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479450#comment-16479450 ] Andrew Purtell edited comment on HBASE-20595 at 5/17/18 5:49 PM: - bq. we'll probably need to document "can't turn on rsgroup feature" as a limitation of single-node deployments. Ah, no, better that the constraint that system tables cannot be placed into the same rsgroup as user tables be what is optional, and allow single node deployments to put everything into "default" was (Author: apurtell): bq. we'll probably need to document "can't turn on rsgroup feature" as a limitation of single-node deployments. Ah, no, better that the constraint that system tables cannot be placed into the same rsgroup as user tables be what is optional. > Remove the concept of 'special tables' from rsgroups > > > Key: HBASE-20595 > URL: https://issues.apache.org/jira/browse/HBASE-20595 > Project: HBase > Issue Type: Task > Components: Region Assignment, rsgroup >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > > Regionserver groups needs to specially handle what it calls "special tables", > tables upon which core or other modular functionality depends. They need to > be excluded from normal rsgroup processing during bootstrap to avoid circular > dependencies or errors due to insufficiently initialized state. I think we > also want to ensure that such tables are always given a rsgroup assignment > with nonzero servers. (IIRC another issue already raises that point, we can > link it later.) > Special tables include: > * The system tables in the 'hbase:' namespace > * The ACL table if the AccessController coprocessor is installed > * The Labels table if the VisibilityController coprocessor is installed > * The Quotas table if the FS quotas feature is active > Either we need a facility where "special tables" can be registered, which > should be in core. Or, we institute a blanket rule that core and all > extensions that need a "special table" must put them into the 'hbase:' > namespace, so the TableName#isSystemTable() test will return TRUE for all, > and then rsgroups simply needs to test for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20595) Remove the concept of 'special tables' from rsgroups
[ https://issues.apache.org/jira/browse/HBASE-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479450#comment-16479450 ] Andrew Purtell commented on HBASE-20595: bq. we'll probably need to document "can't turn on rsgroup feature" as a limitation of single-node deployments. Ah, no, better that the constraint that system tables cannot be placed into the same rsgroup as user tables be what is optional. > Remove the concept of 'special tables' from rsgroups > > > Key: HBASE-20595 > URL: https://issues.apache.org/jira/browse/HBASE-20595 > Project: HBase > Issue Type: Task > Components: Region Assignment, rsgroup >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > > Regionserver groups needs to specially handle what it calls "special tables", > tables upon which core or other modular functionality depends. They need to > be excluded from normal rsgroup processing during bootstrap to avoid circular > dependencies or errors due to insufficiently initialized state. I think we > also want to ensure that such tables are always given a rsgroup assignment > with nonzero servers. (IIRC another issue already raises that point, we can > link it later.) > Special tables include: > * The system tables in the 'hbase:' namespace > * The ACL table if the AccessController coprocessor is installed > * The Labels table if the VisibilityController coprocessor is installed > * The Quotas table if the FS quotas feature is active > Either we need a facility where "special tables" can be registered, which > should be in core. Or, we institute a blanket rule that core and all > extensions that need a "special table" must put them into the 'hbase:' > namespace, so the TableName#isSystemTable() test will return TRUE for all, > and then rsgroups simply needs to test for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479433#comment-16479433 ] Andrew Purtell edited comment on HBASE-20597 at 5/17/18 5:36 PM: - Not sure if this is only a theoretical problem but might as well fix it. If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread). was (Author: apurtell): If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread). > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479433#comment-16479433 ] Andrew Purtell edited comment on HBASE-20597 at 5/17/18 5:36 PM: - Not sure if this is only a theoretical problem but might as well fix it. If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread)? was (Author: apurtell): Not sure if this is only a theoretical problem but might as well fix it. If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread). > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20598) Upgrade to JRuby 9.2
[ https://issues.apache.org/jira/browse/HBASE-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20598: -- Fix Version/s: 3.0.0 > Upgrade to JRuby 9.2 > > > Key: HBASE-20598 > URL: https://issues.apache.org/jira/browse/HBASE-20598 > Project: HBase > Issue Type: Bug > Components: dependencies, shell >Reporter: Josh Elser >Priority: Major > Fix For: 3.0.0 > > > [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we > can get ourselves onto that from our current 9.1 release line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479436#comment-16479436 ] Andrew Purtell commented on HBASE-20597: Based on a comment found in this code looks like findbugs has been warning about this. > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479433#comment-16479433 ] Andrew Purtell commented on HBASE-20597: If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread). > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20597: --- Description: The code that closes down a ZKW that fails to initialize when attempting to connect to the remote cluster is not MT safe and can in theory leak ZooKeeperWatcher instances. The allocation of a new ZKW and store to the reference is not atomic. Might have concurrent allocations with only one winning store, leading to leaked ZKW instances. (was: The code that closes down a ZKW that fails to initialize when attempting to connect to the remote cluster is not MT safe and can in theory leak ZooKeeperWatcher instances. The allocation of a new ZKW and store to the reference is not atomic. Might have concurrent allocations with only one winning store, leading to leaked ZKW instances. If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread). ) > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479431#comment-16479431
]
Josh Elser commented on HBASE-20582:
{quote}file a follow on issue for jruby 9.2?
{quote}
Done sir HBASE-20598
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20598) Upgrade to JRuby 9.2
Josh Elser created HBASE-20598: -- Summary: Upgrade to JRuby 9.2 Key: HBASE-20598 URL: https://issues.apache.org/jira/browse/HBASE-20598 Project: HBase Issue Type: Bug Components: dependencies, shell Reporter: Josh Elser [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we can get ourselves onto that from our current 9.1 release line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
Andrew Purtell created HBASE-20597: -- Summary: Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint Key: HBASE-20597 URL: https://issues.apache.org/jira/browse/HBASE-20597 Project: HBase Issue Type: Bug Affects Versions: 1.4.4, 1.3.2 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 The code that closes down a ZKW that fails to initialize when attempting to connect to the remote cluster is not MT safe and can in theory leak ZooKeeperWatcher instances. The allocation of a new ZKW and store to the reference is not atomic. Might have concurrent allocations with only one winning store, leading to leaked ZKW instances. If the connection problem is persistent, like loss of shared trust between the clusters, we may accumulate unclosed ZKW instances over time, with a ZK send thread and event thread each, and eventually have enough leaked threads to cause OOME (cannot allocate native thread). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20582:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the discussion everyone!
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18948) HBase tags are server side only.
[ https://issues.apache.org/jira/browse/HBASE-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479398#comment-16479398 ] Josh Elser commented on HBASE-18948: [~trigunab], your patch still needs some work. Please see the above discussion around how tags *are* available for Coprocessors. > HBase tags are server side only. > > > Key: HBASE-18948 > URL: https://issues.apache.org/jira/browse/HBASE-18948 > Project: HBase > Issue Type: Improvement > Components: API, documentation >Reporter: Thiriguna Bharat Rao >Assignee: Thiriguna Bharat Rao >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18948.patch > > > HBase tags are server side only. In the Apache HBase documentation, in > section 62.1.1 http://hbase.apache.org/book.html#_implementation_details , I > am going to add a sentence to state explicitly that "Tags are not available > for get/set from client operations including coprocessors". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20369) Document incompatibilities between HBase 1.1.2 and HBase 2.0
[
https://issues.apache.org/jira/browse/HBASE-20369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479394#comment-16479394
]
Josh Elser commented on HBASE-20369:
[~trigunab], looks like your v6 patch still has multiple commits:
{noformat}
Subject: [PATCH 1/4] Added a new appendix section{noformat}
You could either use git-rebase to squash 4 commits into 1, or just use
git-diff to create the patch.
> Document incompatibilities between HBase 1.1.2 and HBase 2.0
>
>
> Key: HBASE-20369
> URL: https://issues.apache.org/jira/browse/HBASE-20369
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Thiriguna Bharat Rao
>Assignee: Thiriguna Bharat Rao
>Priority: Critical
> Labels: patch
> Attachments: HBASE-20369.patch, HBASE-20369_v1.patch,
> HBASE-20369_v2.patch, HBASE-20369_v3.patch, HBASE-20369_v4.patch,
> HBASE-20369_v5.patch, HBASE-20369_v6.patch, book.adoc
>
>
> Hi,
> I compiled a draft document for the HBase incompatibilities from the raw
> source content that was available in HBase Beta 1 site. Can someone please
> review and provide a feedback or share your comments on this document?
> Appreciate your support and time.
>
> Best Regards,
> Triguna
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20460) Doc offheap write-path
[ https://issues.apache.org/jira/browse/HBASE-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479395#comment-16479395 ] Anoop Sam John commented on HBASE-20460: Also will add one more line. The max off heap size in hbase-env.sh to consider this MSLAB chunk pool size also and changed accordingly. > Doc offheap write-path > -- > > Key: HBASE-20460 > URL: https://issues.apache.org/jira/browse/HBASE-20460 > Project: HBase > Issue Type: Bug > Components: documentation, Offheaping >Reporter: stack >Priority: Critical > Fix For: 2.1.0 > > > We have an empty section in refguide that needs filling in on how to enable > offheap write-path, how to know you've set it up right or not, how to tune > it, and how it relates to direct memory allocation and offheap read-path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20460) Doc offheap write-path
[ https://issues.apache.org/jira/browse/HBASE-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479352#comment-16479352 ] Anoop Sam John commented on HBASE-20460: Hi [~stack].. How abt below details? In hbase-2.0.0, HBASE-15179 made the HBase write path to work off heap. By default the memstores use MSLAB to avoid memory fragmentation. It creates bigger fixed sized chunks and memstore cell's data will get copied into these chunks. These chunks can be pooled also and from 2.0.0 the MSLAB pool is by default ON. The offheaping make use of the MSLAB pool. It creates the MSLAB chunks as Direct ByteBuffers and pool them. The max size for such an off heap chunk pool can be specified using the config 'hbase.regionserver.offheap.global.memstore.size'. Each of the chunk will be of 2MB size by default. When a Cell is added to a memstore, the Cell key, value bytes get copied into these off heap buffers and a Cell POJO will refer to this memory area. It can reduce the on heap size occupancy of the memstores to a great extend and reduce the max heap size requirement for RSs in a write heavy workload. We track the on heap and off heap size of each of the memstore separately and same with for Region and Global level. The flush decisions consider both sizes. At Region level, it consider both sizes sum and compares against the region flush size (128 MB by default). Globally on heap size occupancy of all memstores are tracked as well as off heap size. When any of these sizes breaches the lower mark or the max size, regions are selected for forced flushes. We had also changed the RPC server req reading part to make use of the ByteBufferPool's DBBs. But now , by default Netty RPC server is getting used which is not having this pool usage. So am not adding that info. > Doc offheap write-path > -- > > Key: HBASE-20460 > URL: https://issues.apache.org/jira/browse/HBASE-20460 > Project: HBase > Issue Type: Bug > Components: documentation, Offheaping >Reporter: stack >Priority: Critical > Fix For: 2.1.0 > > > We have an empty section in refguide that needs filling in on how to enable > offheap write-path, how to know you've set it up right or not, how to tune > it, and how it relates to direct memory allocation and offheap read-path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20589) Don't need to assign meta to a new RS when standby master become active
[ https://issues.apache.org/jira/browse/HBASE-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479325#comment-16479325 ] stack commented on HBASE-20589: --- bq. Master should create hbase:meta region node from zk when the meta region already opened? This sounds right. If we are not doing this, seems like a mistake. For user-space regions, see how we populate regionstatenode via a scan of hbase:meta table which has last region state in it. Thanks for digging into this [~zghaobac]. Good find. And thanks for keeping with it. > Don't need to assign meta to a new RS when standby master become active > --- > > Key: HBASE-20589 > URL: https://issues.apache.org/jira/browse/HBASE-20589 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-20589.master.001.patch, > HBASE-20589.master.002.patch, HBASE-20589.master.003.patch, > HBASE-20589.master.003.patch, HBASE-20589.master.004.patch > > > I found this problem when I write ut for HBASE-20569. Now the master > finishActiveMasterInitialization introduce a new > RecoverMetaProcedure(HBASE-18261) and it has a sub procedure AssignProcedure. > AssignProcedure will skip assign a region when regions state is OPEN and > server is online. But for the new regiog state node is created with state > OFFLINE. So it will assign the meta to a new RS. And kill the old RS when old > RS report to master. This will make the master initialization cost a long > time. I will attatch a ut to show this. FYI [~stack] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Singhal reassigned HBASE-20582:
-
Assignee: Josh Elser (was: Ankit Singhal)
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479306#comment-16479306
]
Mike Drob commented on HBASE-20582:
---
file a follow on issue for jruby 9.2?
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20488) PE tool prints full name in help message
[
https://issues.apache.org/jira/browse/HBASE-20488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479265#comment-16479265
]
Hudson commented on HBASE-20488:
Results for branch branch-2
[build #747 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/747/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/747//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/747//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/747//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> PE tool prints full name in help message
>
>
> Key: HBASE-20488
> URL: https://issues.apache.org/jira/browse/HBASE-20488
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 3.0.0
>Reporter: Peter Somogyi
>Assignee: Xu Cang
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20488.master.001.patch,
> HBASE-20488.master.003.patch, HBASE-20488.master.004.patch
>
>
> The PerfomanceEvaluation tool currently prints its full path in help message
> instead of short name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20583) SplitLogWorker should handle FileNotFoundException when split a wal
[
https://issues.apache.org/jira/browse/HBASE-20583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479248#comment-16479248
]
Guanghao Zhang commented on HBASE-20583:
{quote}bq.And do you want to change the log splitting framework from zk based
to proc-v2 based?
{quote}
Yeah. This will remove zk dependency when failover.
{quote}bq.I like the [~Apache9] idea. New issue?
{quote}
Yeah, new issue. We already do same thing for sync replication. The standby
clsuter will use proc-v2 to replay the WALs and no zk dependency.
> SplitLogWorker should handle FileNotFoundException when split a wal
> ---
>
> Key: HBASE-20583
> URL: https://issues.apache.org/jira/browse/HBASE-20583
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20583.master.001.patch,
> HBASE-20583.master.001.patch
>
>
> When a split task is finished, master will delete the wal first, then remove
> the task's zk node. So if master crashed after delelte the wal, the zk task
> node may be leaved on zk. When master resubmit this task, the task will
> failed by FileNotFoundException.
> We also handle FileNotFoundException in WALSplitter. But not handle this in
> SplitLogWorker.
>
> {code:java}
> try {
> in = getReader(path, reporter);
> } catch (EOFException e) {
> if (length <= 0) {
> // TODO should we ignore an empty, not-last log file if skip.errors
> // is false? Either way, the caller should decide what to do. E.g.
> // ignore if this is the last log in sequence.
> // TODO is this scenario still possible if the log has been
> // recovered (i.e. closed)
> LOG.warn("Could not open {} for reading. File is empty", path, e);
> }
> // EOFException being ignored
> return null;
> }
> } catch (IOException e) {
> if (e instanceof FileNotFoundException) {
> // A wal file may not exist anymore. Nothing can be recovered so move on
> LOG.warn("File {} does not exist anymore", path, e);
> return null;
> }
> }{code}
> {code:java}
> // Here fs.getFileStatus may throw FileNotFoundException, too. We should
> handle this exception as the WALSplitter.getReader.
> try {
> if (!WALSplitter.splitLogFile(walDir, fs.getFileStatus(new Path(walDir,
> filename)),
> fs, conf, p, sequenceIdChecker,
> server.getCoordinatedStateManager().getSplitLogWorkerCoordination(),
> factory)) {
> return Status.PREEMPTED;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20589) Don't need to assign meta to a new RS when standby master become active
[ https://issues.apache.org/jira/browse/HBASE-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479249#comment-16479249 ] Guanghao Zhang commented on HBASE-20589: Master should create hbase:meta region node from zk when the meta region already opened? > Don't need to assign meta to a new RS when standby master become active > --- > > Key: HBASE-20589 > URL: https://issues.apache.org/jira/browse/HBASE-20589 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-20589.master.001.patch, > HBASE-20589.master.002.patch, HBASE-20589.master.003.patch, > HBASE-20589.master.003.patch, HBASE-20589.master.004.patch > > > I found this problem when I write ut for HBASE-20569. Now the master > finishActiveMasterInitialization introduce a new > RecoverMetaProcedure(HBASE-18261) and it has a sub procedure AssignProcedure. > AssignProcedure will skip assign a region when regions state is OPEN and > server is online. But for the new regiog state node is created with state > OFFLINE. So it will assign the meta to a new RS. And kill the old RS when old > RS report to master. This will make the master initialization cost a long > time. I will attatch a ut to show this. FYI [~stack] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20589) Don't need to assign meta to a new RS when standby master become active
[
https://issues.apache.org/jira/browse/HBASE-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479211#comment-16479211
]
Guanghao Zhang commented on HBASE-20589:
{quote}What is "...new regiog state..."? You say "...node is created with state
OFFLINE...". Which node is this? The node in master memory for hbase:meta?
{quote}
The region node for hbase:meta. Because the master is new, it create a new node
for hbase:meta and the state is OFFLINE. This will make the AssignProcedure
assign hbase:meta to a new RS.
> Don't need to assign meta to a new RS when standby master become active
> ---
>
> Key: HBASE-20589
> URL: https://issues.apache.org/jira/browse/HBASE-20589
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-20589.master.001.patch,
> HBASE-20589.master.002.patch, HBASE-20589.master.003.patch,
> HBASE-20589.master.003.patch, HBASE-20589.master.004.patch
>
>
> I found this problem when I write ut for HBASE-20569. Now the master
> finishActiveMasterInitialization introduce a new
> RecoverMetaProcedure(HBASE-18261) and it has a sub procedure AssignProcedure.
> AssignProcedure will skip assign a region when regions state is OPEN and
> server is online. But for the new regiog state node is created with state
> OFFLINE. So it will assign the meta to a new RS. And kill the old RS when old
> RS report to master. This will make the master initialization cost a long
> time. I will attatch a ut to show this. FYI [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20595) Remove the concept of 'special tables' from rsgroups
[ https://issues.apache.org/jira/browse/HBASE-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479202#comment-16479202 ] Sean Busbey commented on HBASE-20595: - that makes sense. is the "system group" treated like a normal rsgroup in terms of exclusivity? If so, how do we handle it and the user group needing to be distinct? if we just delegate it to manual operator config that means we can't have the rsgroup on by default ever right? we'll probably need to document "can't turn on rsgroup feature" as a limitation of single-node deployments. > Remove the concept of 'special tables' from rsgroups > > > Key: HBASE-20595 > URL: https://issues.apache.org/jira/browse/HBASE-20595 > Project: HBase > Issue Type: Task > Components: Region Assignment, rsgroup >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > > Regionserver groups needs to specially handle what it calls "special tables", > tables upon which core or other modular functionality depends. They need to > be excluded from normal rsgroup processing during bootstrap to avoid circular > dependencies or errors due to insufficiently initialized state. I think we > also want to ensure that such tables are always given a rsgroup assignment > with nonzero servers. (IIRC another issue already raises that point, we can > link it later.) > Special tables include: > * The system tables in the 'hbase:' namespace > * The ACL table if the AccessController coprocessor is installed > * The Labels table if the VisibilityController coprocessor is installed > * The Quotas table if the FS quotas feature is active > Either we need a facility where "special tables" can be registered, which > should be in core. Or, we institute a blanket rule that core and all > extensions that need a "special table" must put them into the 'hbase:' > namespace, so the TableName#isSystemTable() test will return TRUE for all, > and then rsgroups simply needs to test for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479194#comment-16479194
]
Sean Busbey commented on HBASE-20582:
-
thumbsup.gif
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20589) Don't need to assign meta to a new RS when standby master become active
[ https://issues.apache.org/jira/browse/HBASE-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479190#comment-16479190 ] stack commented on HBASE-20589: --- What is "...new regiog state..."? You say "...node is created with state OFFLINE...". Which node is this? The node in master memory for hbase:meta? Sorry for all the questions. Trying to understand what has changed. Generally we go through the hbase:meta AssignProcedure steps to confirm that the hbase:meta is in the place it claims to be... Even unto asking the current server to open the hbase:meta. It will reject the request saying it already open if carrying the region and then we'll know its indeed where it claims to be. The meta state is marked as OFFLINE on new master start and it used to be what was in zk which would have been the last state set on last hbase:meta assign? On the patch, I'm not mad about calling finishTransition(env, regionNode) .. from other than the last step in the assign procedure. Whats the reason for not setting next step as finish and calling finishTransition instead? Set next state in starttransition to REGION_TRANSITION_FINISH ? Thanks. > Don't need to assign meta to a new RS when standby master become active > --- > > Key: HBASE-20589 > URL: https://issues.apache.org/jira/browse/HBASE-20589 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-20589.master.001.patch, > HBASE-20589.master.002.patch, HBASE-20589.master.003.patch, > HBASE-20589.master.003.patch, HBASE-20589.master.004.patch > > > I found this problem when I write ut for HBASE-20569. Now the master > finishActiveMasterInitialization introduce a new > RecoverMetaProcedure(HBASE-18261) and it has a sub procedure AssignProcedure. > AssignProcedure will skip assign a region when regions state is OPEN and > server is online. But for the new regiog state node is created with state > OFFLINE. So it will assign the meta to a new RS. And kill the old RS when old > RS report to master. This will make the master initialization cost a long > time. I will attatch a ut to show this. FYI [~stack] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20583) SplitLogWorker should handle FileNotFoundException when split a wal
[
https://issues.apache.org/jira/browse/HBASE-20583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479164#comment-16479164
]
stack commented on HBASE-20583:
---
To be clear, thats +1 for branch-2.0. Nice bug fix. Thanks [~zghaobac].
> SplitLogWorker should handle FileNotFoundException when split a wal
> ---
>
> Key: HBASE-20583
> URL: https://issues.apache.org/jira/browse/HBASE-20583
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20583.master.001.patch,
> HBASE-20583.master.001.patch
>
>
> When a split task is finished, master will delete the wal first, then remove
> the task's zk node. So if master crashed after delelte the wal, the zk task
> node may be leaved on zk. When master resubmit this task, the task will
> failed by FileNotFoundException.
> We also handle FileNotFoundException in WALSplitter. But not handle this in
> SplitLogWorker.
>
> {code:java}
> try {
> in = getReader(path, reporter);
> } catch (EOFException e) {
> if (length <= 0) {
> // TODO should we ignore an empty, not-last log file if skip.errors
> // is false? Either way, the caller should decide what to do. E.g.
> // ignore if this is the last log in sequence.
> // TODO is this scenario still possible if the log has been
> // recovered (i.e. closed)
> LOG.warn("Could not open {} for reading. File is empty", path, e);
> }
> // EOFException being ignored
> return null;
> }
> } catch (IOException e) {
> if (e instanceof FileNotFoundException) {
> // A wal file may not exist anymore. Nothing can be recovered so move on
> LOG.warn("File {} does not exist anymore", path, e);
> return null;
> }
> }{code}
> {code:java}
> // Here fs.getFileStatus may throw FileNotFoundException, too. We should
> handle this exception as the WALSplitter.getReader.
> try {
> if (!WALSplitter.splitLogFile(walDir, fs.getFileStatus(new Path(walDir,
> filename)),
> fs, conf, p, sequenceIdChecker,
> server.getCoordinatedStateManager().getSplitLogWorkerCoordination(),
> factory)) {
> return Status.PREEMPTED;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20583) SplitLogWorker should handle FileNotFoundException when split a wal
[
https://issues.apache.org/jira/browse/HBASE-20583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479162#comment-16479162
]
stack commented on HBASE-20583:
---
+1
I like the [~Apache9] idea. New issue?
> SplitLogWorker should handle FileNotFoundException when split a wal
> ---
>
> Key: HBASE-20583
> URL: https://issues.apache.org/jira/browse/HBASE-20583
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20583.master.001.patch,
> HBASE-20583.master.001.patch
>
>
> When a split task is finished, master will delete the wal first, then remove
> the task's zk node. So if master crashed after delelte the wal, the zk task
> node may be leaved on zk. When master resubmit this task, the task will
> failed by FileNotFoundException.
> We also handle FileNotFoundException in WALSplitter. But not handle this in
> SplitLogWorker.
>
> {code:java}
> try {
> in = getReader(path, reporter);
> } catch (EOFException e) {
> if (length <= 0) {
> // TODO should we ignore an empty, not-last log file if skip.errors
> // is false? Either way, the caller should decide what to do. E.g.
> // ignore if this is the last log in sequence.
> // TODO is this scenario still possible if the log has been
> // recovered (i.e. closed)
> LOG.warn("Could not open {} for reading. File is empty", path, e);
> }
> // EOFException being ignored
> return null;
> }
> } catch (IOException e) {
> if (e instanceof FileNotFoundException) {
> // A wal file may not exist anymore. Nothing can be recovered so move on
> LOG.warn("File {} does not exist anymore", path, e);
> return null;
> }
> }{code}
> {code:java}
> // Here fs.getFileStatus may throw FileNotFoundException, too. We should
> handle this exception as the WALSplitter.getReader.
> try {
> if (!WALSplitter.splitLogFile(walDir, fs.getFileStatus(new Path(walDir,
> filename)),
> fs, conf, p, sequenceIdChecker,
> server.getCoordinatedStateManager().getSplitLogWorkerCoordination(),
> factory)) {
> return Status.PREEMPTED;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479118#comment-16479118
]
Josh Elser commented on HBASE-20582:
Confirming: TestReplicationKillMasterRSWithSeparateOldWALs did pass for me
locally on branch-2 and master.
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479101#comment-16479101
]
Josh Elser commented on HBASE-20582:
Test failure was
{noformat}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 69.215
s <<< FAILURE! - in
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs
[ERROR]
killOneMasterRS(org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs)
Time elapsed: 66.462 s <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=16, exceptions:
Wed May 16 19:07:48 UTC 2018, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=61018: Call to bcdc9c6c7e68/172.17.0.2:38822
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=79, waitTime=58812, rpcTimeout=58800 row 'ccc' on table 'test' at
region=test,ccc,1526497597339.d09793b3665a04d8bc5dda08f0e15de3.,
hostname=bcdc9c6c7e68,38822,1526497586868, seqNum=2
at
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs.killOneMasterRS(TestReplicationKillMasterRSWithSeparateOldWALs.java:45)
Caused by: java.net.SocketTimeoutException: callTimeout=6,
callDuration=61018: Call to bcdc9c6c7e68/172.17.0.2:38822 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=79,
waitTime=58812, rpcTimeout=58800 row 'ccc' on table 'test' at
region=test,ccc,1526497597339.d09793b3665a04d8bc5dda08f0e15de3.,
hostname=bcdc9c6c7e68,38822,1526497586868, seqNum=2
Caused by: java.io.IOException: Call to bcdc9c6c7e68/172.17.0.2:38822 failed on
local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=79,
waitTime=58812, rpcTimeout=58800
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=79,
waitTime=58812, rpcTimeout=58800{noformat}
Re-running this locally, but I suspect this to be unrelated to this change.
Folks happy with this for 2.1? ([~busbey] I think you had the strongest
opinions)
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20424) Allow writing WAL to local and remote cluster concurrently
[ https://issues.apache.org/jira/browse/HBASE-20424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479046#comment-16479046 ] Duo Zhang commented on HBASE-20424: --- Review board link: https://reviews.apache.org/r/67182/ > Allow writing WAL to local and remote cluster concurrently > -- > > Key: HBASE-20424 > URL: https://issues.apache.org/jira/browse/HBASE-20424 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20424-HBASE-19064.patch > > > For better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20424) Allow writing WAL to local and remote cluster concurrently
[
https://issues.apache.org/jira/browse/HBASE-20424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479038#comment-16479038
]
Hadoop QA commented on HBASE-20424:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
58s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 32s{color}
| {color:red} hbase-server generated 2 new + 186 unchanged - 2 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} hbase-server: The patch generated 3 new + 97 unchanged
- 2 fixed = 100 total (was 99) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m
39s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20424 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12923906/HBASE-20424-HBASE-19064.patch
[jira] [Commented] (HBASE-20583) SplitLogWorker should handle FileNotFoundException when split a wal
[
https://issues.apache.org/jira/browse/HBASE-20583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479034#comment-16479034
]
Duo Zhang commented on HBASE-20583:
---
+1.
And do you want to change the log splitting framework from zk based to proc-v2
based? I think this could solve the problem by a cleaner way.
> SplitLogWorker should handle FileNotFoundException when split a wal
> ---
>
> Key: HBASE-20583
> URL: https://issues.apache.org/jira/browse/HBASE-20583
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20583.master.001.patch,
> HBASE-20583.master.001.patch
>
>
> When a split task is finished, master will delete the wal first, then remove
> the task's zk node. So if master crashed after delelte the wal, the zk task
> node may be leaved on zk. When master resubmit this task, the task will
> failed by FileNotFoundException.
> We also handle FileNotFoundException in WALSplitter. But not handle this in
> SplitLogWorker.
>
> {code:java}
> try {
> in = getReader(path, reporter);
> } catch (EOFException e) {
> if (length <= 0) {
> // TODO should we ignore an empty, not-last log file if skip.errors
> // is false? Either way, the caller should decide what to do. E.g.
> // ignore if this is the last log in sequence.
> // TODO is this scenario still possible if the log has been
> // recovered (i.e. closed)
> LOG.warn("Could not open {} for reading. File is empty", path, e);
> }
> // EOFException being ignored
> return null;
> }
> } catch (IOException e) {
> if (e instanceof FileNotFoundException) {
> // A wal file may not exist anymore. Nothing can be recovered so move on
> LOG.warn("File {} does not exist anymore", path, e);
> return null;
> }
> }{code}
> {code:java}
> // Here fs.getFileStatus may throw FileNotFoundException, too. We should
> handle this exception as the WALSplitter.getReader.
> try {
> if (!WALSplitter.splitLogFile(walDir, fs.getFileStatus(new Path(walDir,
> filename)),
> fs, conf, p, sequenceIdChecker,
> server.getCoordinatedStateManager().getSplitLogWorkerCoordination(),
> factory)) {
> return Status.PREEMPTED;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20561) The way we stop a ReplicationSource may cause the RS down
[
https://issues.apache.org/jira/browse/HBASE-20561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479013#comment-16479013
]
Duo Zhang commented on HBASE-20561:
---
Much better now. The only concern is that is it safe to convert all
SystemErrorException to InterruptedException?
And maybe we could also add a graceful wait interval before interrupting the
thread when stop a ReplicationSource?
> The way we stop a ReplicationSource may cause the RS down
> -
>
> Key: HBASE-20561
> URL: https://issues.apache.org/jira/browse/HBASE-20561
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-20561.master.001.patch,
> HBASE-20561.master.002.patch
>
>
> See this:
> https://builds.apache.org/job/HBASE-Flaky-Tests/31125/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL-output.txt
> {noformat}
> 2018-05-09 15:07:00,887 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.RefreshPeerCallable(52): Received a peer change event, peerId=2,
> type=REMOVE_PEER
> 2018-05-09 15:07:00,890 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.ReplicationSource(485): Closing source
> 2-asf916.gq1.ygridcore.net,36287,1525878368395 because: Replication stream
> was removed by a user
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-0,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:871)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.removeWAL(ZKReplicationQueueStorage.java:198)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$cleanOldLogs$8(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:454)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:506)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:489)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:231)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:133)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:103)
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-1,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:990)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:663)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1690)
> at
>
[jira] [Commented] (HBASE-20589) Don't need to assign meta to a new RS when standby master become active
[
https://issues.apache.org/jira/browse/HBASE-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479007#comment-16479007
]
Hadoop QA commented on HBASE-20589:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 3 new + 30 unchanged
- 0 fixed = 33 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}160m 41s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestRestartCluster |
| | hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
| | hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| |
hadoop.hbase.replication.multiwal.TestReplicationSyncUpToolWithMultipleAsyncWAL
|
| | hadoop.hbase.master.procedure.TestTruncateTableProcedure |
| | hadoop.hbase.replication.TestReplicationSyncUpTool |
| | hadoop.hbase.replication.TestReplicationDisableInactivePeer |
| | hadoop.hbase.master.procedure.TestCloneSnapshotProcedure |
| | hadoop.hbase.master.procedure.TestModifyNamespaceProcedure |
| | hadoop.hbase.client.TestMetaWithReplicas |
| | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.master.assignment.TestSplitTableRegionProcedure |
| | hadoop.hbase.TestMultiVersions |
| | hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | hadoop.hbase.master.procedure.TestProcedureAdmin |
| | hadoop.hbase.master.procedure.TestDeleteNamespaceProcedure |
| | hadoop.hbase.master.TestMasterOperationsForRegionReplicas |
| |
[jira] [Commented] (HBASE-20591) nightly job doesn't respect maven options
[
https://issues.apache.org/jira/browse/HBASE-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478989#comment-16478989
]
Hudson commented on HBASE-20591:
Results for branch HBASE-20591
[build #3 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20591/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20591/3//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20591/3//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20591/3//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> nightly job doesn't respect maven options
> -
>
> Key: HBASE-20591
> URL: https://issues.apache.org/jira/browse/HBASE-20591
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HBASE-20591.0.patch
>
>
> some recent nightly runs have had failures that look like HBASE-20068, but at
> the maven install step.
> looks like our yetus runs are ignoring the "use build specific maven repos"
> flag.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20571) JMXJsonServlet generates invalid JSON if it has NaN in metrics
[
https://issues.apache.org/jira/browse/HBASE-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478981#comment-16478981
]
Hudson commented on HBASE-20571:
Results for branch master
[build #334 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/334/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> JMXJsonServlet generates invalid JSON if it has NaN in metrics
> --
>
> Key: HBASE-20571
> URL: https://issues.apache.org/jira/browse/HBASE-20571
> Project: HBase
> Issue Type: Bug
> Components: UI
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20571.branch-1.003.patch,
> HBASE-20571.branch-1.004.patch, HBASE-20571.branch-2.0.001.patch,
> HBASE-20571.branch-2.0.002.patch, HBASE-20571.branch-2.0.002.patch,
> HBASE-20571.branch-2.0.003.patch
>
>
> {{/jmx}} servlet responses invalid JSON, if some metrics are NaN:
> {code}
> "l1CacheHitCount" : 0,
> "l1CacheMissCount" : 0,
> "l1CacheHitRatio" : NaN,
> "l1CacheMissRatio" : NaN,
> "l2CacheHitCount" : 0,
> "l2CacheMissCount" : 0,
> "l2CacheHitRatio" : 0.0,
> "l2CacheMissRatio" : 0.0,
> {code}
> NaN is an invalid character sequence in JSON. We should not response NaN in
> metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[
https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478976#comment-16478976
]
Hudson commented on HBASE-20520:
Results for branch master
[build #334 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/334/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
> --
>
> Key: HBASE-20520
> URL: https://issues.apache.org/jira/browse/HBASE-20520
> Project: HBase
> Issue Type: Sub-task
> Components: conf
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20520.branch-2.0.001.patch,
> HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch,
> HBASE-20520.branch-2.0.004.patch, HBASE-20520.branch-2.0.005.patch
>
>
> Good one found by our [~mdrob]. Problem in the parent issue where failed
> attempt doubling default block size but halving the size at which roll -- so
> we end up in roughly same place only we make less small WALs in those cases
> where we are writing furiously and fail to roll before starting a new block.
> From Drob: ".../we drop default log roll to 0.5, but we don't actually
> increase the block size... AbstractProtobufLogWriter.java:164 still uses
> default HDFS block sizing and AbstractFSWAL.java:408 block size is only used
> in a log message, never actually makes it to files...".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20593) HBase website landing page should link to HBaseCon Asia 2018
[
https://issues.apache.org/jira/browse/HBASE-20593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478979#comment-16478979
]
Hudson commented on HBASE-20593:
Results for branch master
[build #334 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/334/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HBase website landing page should link to HBaseCon Asia 2018
>
>
> Key: HBASE-20593
> URL: https://issues.apache.org/jira/browse/HBASE-20593
> Project: HBase
> Issue Type: Task
> Components: website
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20593.0.patch, HBASE-20593.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478980#comment-16478980
]
Hudson commented on HBASE-20567:
Results for branch master
[build #334 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/334/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/334//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20567.master.001.patch,
> HBASE-20567.master.002.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
