[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737947#comment-16737947 ] xufeng commented on HBASE-7191: --- Can we move the connect() method to the position after offlineHdfsIntegrityRepair() method? > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Priority: Major > Attachments: 7191-2.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21614:
--
Attachment: HBASE-21614-v1.patch
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614-v1.patch, HBASE-21614.01.patch,
> HBASE-21614.patch, HBASE-21614.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21694) Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell commands
[ https://issues.apache.org/jira/browse/HBASE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-21694: --- Attachment: HBASE-21694.master.004.patch > Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell > commands > > > Key: HBASE-21694 > URL: https://issues.apache.org/jira/browse/HBASE-21694 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21694.master.001.patch, > HBASE-21694.master.002.patch, HBASE-21694.master.003.patch, > HBASE-21694.master.004.patch > > > To add or remove table-cfs to peers' exclude-table-cfs list, the current way > is use set_peer_exclude_tableCFs. But one has to copy all exclude-table-cfs > of the peer firstly and modify it, which is error-prone and time-consuming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737927#comment-16737927
]
Hadoop QA commented on HBASE-21614:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 37s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestClientClusterMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21614 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954268/HBASE-21614.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 83d5dc24d437 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737931#comment-16737931 ] xufeng commented on HBASE-7191: --- anybody here? I encounter this issue. hbase hbck can not repair offline hdfs problem. hbase failed to start because there is no version file on hdfs. try to do "hbase hbck -fixVersionFile"，but the tool will connect master, so failed. anyone can help me？ thank you。 > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Priority: Major > Attachments: 7191-2.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737930#comment-16737930
]
Duo Zhang commented on HBASE-21614:
---
Oh, let me provide a new patch. When expected is empty, we will return true
instead of false in the past. This is a bit strange but maybe useful for some
scenario. Let me at least add comment about this behavior.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614.01.patch, HBASE-21614.patch,
> HBASE-21614.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21663) Add replica scan support
[ https://issues.apache.org/jira/browse/HBASE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737920#comment-16737920 ] Duo Zhang commented on HBASE-21663: --- Review board link: https://reviews.apache.org/r/69696/ > Add replica scan support > > > Key: HBASE-21663 > URL: https://issues.apache.org/jira/browse/HBASE-21663 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21663.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21663) Add replica scan support
[ https://issues.apache.org/jira/browse/HBASE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21663: -- Fix Version/s: 2.0.5 2.1.3 2.2.0 3.0.0 > Add replica scan support > > > Key: HBASE-21663 > URL: https://issues.apache.org/jira/browse/HBASE-21663 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21663.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21663) Add replica scan support
[ https://issues.apache.org/jira/browse/HBASE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21663: -- Component/s: read replicas Client asyncclient > Add replica scan support > > > Key: HBASE-21663 > URL: https://issues.apache.org/jira/browse/HBASE-21663 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21663.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21663) Add replica scan support
[ https://issues.apache.org/jira/browse/HBASE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21663: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Add replica scan support > > > Key: HBASE-21663 > URL: https://issues.apache.org/jira/browse/HBASE-21663 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21663.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21663) Add replica scan support
[ https://issues.apache.org/jira/browse/HBASE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21663: -- Attachment: HBASE-21663.patch > Add replica scan support > > > Key: HBASE-21663 > URL: https://issues.apache.org/jira/browse/HBASE-21663 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > Attachments: HBASE-21663.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21695) Fix flaky test TestRegionServerAbortTimeout
[
https://issues.apache.org/jira/browse/HBASE-21695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737911#comment-16737911
]
Guanghao Zhang commented on HBASE-21695:
{quote}bq. Did something change sir to cause this breakage do you know? Thanks.
{quote}
Maybe something related to SCP or Procedure V2? I checked the failed log
before. The ut timeout because the procedure can't finished and there are no
regionservers and only master can't stop...
> Fix flaky test TestRegionServerAbortTimeout
> ---
>
> Key: HBASE-21695
> URL: https://issues.apache.org/jira/browse/HBASE-21695
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-21695.master.001.patch,
> HBASE-21695.master.001.patch, HBASE-21695.master.001.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/master/Flaky_20Test_20Report/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21626) log the regions blocking WAL from being archived
[ https://issues.apache.org/jira/browse/HBASE-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737891#comment-16737891 ] stack commented on HBASE-21626: --- [~sershe] The patch is great. I went to commit but it doesn't have your email in the patch (have you looked at the little ./dev-support/submit-patch.py tool... or if you don't mind, make your patch with git format-patch so it includes your email detail so we can properly attribute your work). Otherwise, tell me what to shove in there and I'll do it manually. Thanks. > log the regions blocking WAL from being archived > > > Key: HBASE-21626 > URL: https://issues.apache.org/jira/browse/HBASE-21626 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21626.01.patch, HBASE-21626.02.patch, > HBASE-21626.patch > > > The WALs not being archived for a long time can result in a long recovery > later. It's useful to know what regions are blocking the WALs from being > archived, to be able to debug flush logic and tune configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21697) Add 2.1.2 to the download page
[ https://issues.apache.org/jira/browse/HBASE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737890#comment-16737890 ] Duo Zhang commented on HBASE-21697: --- Alright. Just want to confirm that this is intentional, not by accident. Thanks sir. > Add 2.1.2 to the download page > -- > > Key: HBASE-21697 > URL: https://issues.apache.org/jira/browse/HBASE-21697 > Project: HBase > Issue Type: Sub-task > Components: 2.1.2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737892#comment-16737892
]
Hadoop QA commented on HBASE-21699:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 41s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
6s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21699 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954267/hbase-21699.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7499ea7cc8f9 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5c902b48e5 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15512/testReport/ |
| Max. process+thread count | 262 (vs. ulimit of 1) |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15512/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Create table failed
[jira] [Commented] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[
https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737889#comment-16737889
]
Nihal Jain commented on HBASE-21645:
{quote}IMO we should also set region replica count as 1 in descriptor if table
creation is allowed in this case.
{quote}
This can be done, but if we do this we should handle it consistently all
across. What I mean is currently if we try to create/modify table with say, min
version with a negative illegal value, we throw/warn but do not override the
value.
We should handle the over-riding of incorrect/meaningless value for all such
cases.
> Perform sanity check and disallow table creation/modification with region
> replication < 1
> -
>
> Key: HBASE-21645
> URL: https://issues.apache.org/jira/browse/HBASE-21645
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 1.5.0, 2.1.1, 2.1.2
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21645.master.001.patch,
> HBASE-21645.master.001.patch, HBASE-21645.master.002.patch
>
>
> We should perform sanity check and disallow table creation with region
> replication < 1 or modification of an existing table with new region
> replication value < 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[ https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737885#comment-16737885 ] Pankaj Kumar commented on HBASE-21645: -- Thanks [~nihaljain.cs] for the fix. With this approach we will just WARN the user and allow table creation with default replica count ie. 1. What would be the replica count in table descriptor? It would be zero which is actually not correct. IMO we should also set region replica count as 1 in descriptor if table creation is allowed in this case. > Perform sanity check and disallow table creation/modification with region > replication < 1 > - > > Key: HBASE-21645 > URL: https://issues.apache.org/jira/browse/HBASE-21645 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 1.5.0, 2.1.1, 2.1.2 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21645.master.001.patch, > HBASE-21645.master.001.patch, HBASE-21645.master.002.patch > > > We should perform sanity check and disallow table creation with region > replication < 1 or modification of an existing table with new region > replication value < 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21691) Fix flaky test TestRecoveredEdits
[
https://issues.apache.org/jira/browse/HBASE-21691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737886#comment-16737886
]
Hadoop QA commented on HBASE-21691:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} hbase-server: The patch generated 0 new + 8
unchanged - 2 fixed = 8 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 47s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}252m 46s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}301m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | hadoop.hbase.master.TestSplitWALManager |
| | hadoop.hbase.master.procedure.TestServerCrashProcedureWithReplicas |
| | hadoop.hbase.replication.TestReplicationDisableInactivePeer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21691 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954247/HBASE-21691.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux dc3468ae6786 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ebe3d1d1d9 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
[jira] [Commented] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737884#comment-16737884
]
Hadoop QA commented on HBASE-21699:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
0s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-21699 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954266/hbase-21699.branch-2.0.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6c7ddcb12a18 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.0 / b9bcbd0a23 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15511/testReport/ |
| Max. process+thread count | 291 (vs. ulimit of 1) |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15511/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically
[jira] [Commented] (HBASE-21697) Add 2.1.2 to the download page
[ https://issues.apache.org/jira/browse/HBASE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737881#comment-16737881 ] stack commented on HBASE-21697: --- [~Apache9] Yeah. I was thinking not needed since 2.1.1 and 2.0.3 have been removed. What you thinking [~Apache9]? > Add 2.1.2 to the download page > -- > > Key: HBASE-21697 > URL: https://issues.apache.org/jira/browse/HBASE-21697 > Project: HBase > Issue Type: Sub-task > Components: 2.1.2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737877#comment-16737877
]
stack commented on HBASE-21614:
---
Retry
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614.01.patch, HBASE-21614.patch,
> HBASE-21614.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21614:
--
Attachment: HBASE-21614.patch
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614.01.patch, HBASE-21614.patch,
> HBASE-21614.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21661) Provide Thrift2 implementation of Table/Admin
[ https://issues.apache.org/jira/browse/HBASE-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737880#comment-16737880 ] stack commented on HBASE-21661: --- Yeah, could be another issue [~allan163] (there probably is one already). The mvn proto plugin is nice because it does appropriate binary (Could look into what it would take when moving thrift to the hbase-connector module?). +1 on patch (Thanks for the RN and updating the description to make it more plain what is going on here -- building the case for thrift2 over thrift1). Thanks [~allan163] > Provide Thrift2 implementation of Table/Admin > - > > Key: HBASE-21661 > URL: https://issues.apache.org/jira/browse/HBASE-21661 > Project: HBase > Issue Type: Sub-task >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21661.patch, HBASE-21661.v2.patch, > HBASE-21661.v3.patch, HBASE-21661.v4.patch, HBASE-21661.v5.patch > > > Provide Thrift2 implementation of Table/Admin, making Java user to use thrift > client more easily(Some environment which can not expose ZK or RS Servers > directly require thrift or REST protocol even using Java). > Another Example of this is RemoteHTable and RemoteAdmin, they are REST > connectors. > ThriftAdmin/ThriftTable provides another good reason for Java user to use > thrift2 and not thrift1. Before, they need to convert HBase Get/Put/Scan... > to TGet/TPut/TScan and construct a thrift client all by themselves. With > ThriftAdmin/ThriftTable, they can use thrift protocol just like > HTable/HBaseAdmin. They are just like RemoteHTable/RemoteAdmin for REST > protocols. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737876#comment-16737876
]
stack commented on HBASE-21614:
---
+1 on patch. Makes sense to me. Nice digging [~sershe] sir.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614.01.patch, HBASE-21614.patch,
> HBASE-21614.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737873#comment-16737873
]
Guanghao Zhang commented on HBASE-21699:
Add a test to admin_test.rb, too?
> Create table failed when using SPLITS_FILE => 'splits.txt'
> ---
>
> Key: HBASE-21699
> URL: https://issues.apache.org/jira/browse/HBASE-21699
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4
>Reporter: huan
>Priority: Minor
> Attachments: hbase-21699.001.patch
>
>
> Hi all:
> When I ran
> {code:java}
> create 't1', 'f1', SPLITS_FILE => 'splits.txt'
> {code}
> on HBase2.0.0, it failed, and no detailed error info, just like below:
> {code:java}
> ERROR:
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> {code}
> So I opened the debug:
> {code:java}
> hbase shell -d
> {code}
> and
> {code:java}
> ERROR:
> Backtrace:
> org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
> org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
> But it works on branch 1.2.0.
> so I view the source code, I find the issue is because the below code:
> {code:java}
> // admin.rb
> if arg.key?(SPLITS_FILE)
> splits_file = arg.delete(SPLITS_FILE)
> unless File.exist?(splits_file)
> raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
> end
> arg[SPLITS] = []
> File.foreach(splits_file) do |line|
> arg[SPLITS].push(line.chomp)
> end
> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
> end
> {code}
> {code:java}
> // HTableDescriptor part
> public HTableDescriptor setValue(String key, String value) {
> getDelegateeForModification().setValue(Bytes.toBytes(key),
> Bytes.toBytes(value));
> return this;
> }
> {code}
> {code:java}
> // Bytes part
> public static byte[] toBytes(String s) {
> try {
> return s.getBytes(UTF8_CSN);
> } catch (UnsupportedEncodingException e) {
> // should never happen!
> throw new IllegalArgumentException("UTF8 decoding is not supported", e);
> }
> }
> {code}
> Call flow is:
> {code:java}
> admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
> Bytes.toBytes(key) && Bytes.toBytes(value) {code}
> from Bytes.toBytes, if s is null, the function will throw
> NullPointerException, but HTableDescriptor.setValue(String key, String value)
> does not check key and value.
> in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
> after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
> HTableDescriptor.setValue(String key, String value) does not check key and
> value is root cause.
> why branch below 2.0.0 works fine, because old code is :
> {code:java}
> public HTableDescriptor setValue(String key, String value) {
> if (value == null) {
> remove(key);
> } else {
> setValue(Bytes.toBytes(key), Bytes.toBytes(value));
> }
> return this;
> }
> {code}
> it check the value.
> since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
> it included:
> {code:java}
> public ModifyableTableDescriptor setValue(String key, String value) {
> return setValue(toBytesOrNull(key, Bytes::toBytes),
> toBytesOrNull(value, Bytes::toBytes));
> }
> {code}
> it checked key and value, but HTableDescriptor.setValue(String key, String
> value) does not call it, so
> just change HTableDescriptor.setValue(String key, String value) to call it,
> it will works
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[ https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737872#comment-16737872 ] Nihal Jain commented on HBASE-21645: Thanks [~zghaobac] for review and commit. > Perform sanity check and disallow table creation/modification with region > replication < 1 > - > > Key: HBASE-21645 > URL: https://issues.apache.org/jira/browse/HBASE-21645 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 1.5.0, 2.1.1, 2.1.2 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21645.master.001.patch, > HBASE-21645.master.001.patch, HBASE-21645.master.002.patch > > > We should perform sanity check and disallow table creation with region > replication < 1 or modification of an existing table with new region > replication value < 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Attachment: (was: hbase-21699.branch-2.0.patch)
> Create table failed when using SPLITS_FILE => 'splits.txt'
> ---
>
> Key: HBASE-21699
> URL: https://issues.apache.org/jira/browse/HBASE-21699
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4
>Reporter: huan
>Priority: Minor
> Attachments: hbase-21699.001.patch
>
>
> Hi all:
> When I ran
> {code:java}
> create 't1', 'f1', SPLITS_FILE => 'splits.txt'
> {code}
> on HBase2.0.0, it failed, and no detailed error info, just like below:
> {code:java}
> ERROR:
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> {code}
> So I opened the debug:
> {code:java}
> hbase shell -d
> {code}
> and
> {code:java}
> ERROR:
> Backtrace:
> org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
> org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
> But it works on branch 1.2.0.
> so I view the source code, I find the issue is because the below code:
> {code:java}
> // admin.rb
> if arg.key?(SPLITS_FILE)
> splits_file = arg.delete(SPLITS_FILE)
> unless File.exist?(splits_file)
> raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
> end
> arg[SPLITS] = []
> File.foreach(splits_file) do |line|
> arg[SPLITS].push(line.chomp)
> end
> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
> end
> {code}
> {code:java}
> // HTableDescriptor part
> public HTableDescriptor setValue(String key, String value) {
> getDelegateeForModification().setValue(Bytes.toBytes(key),
> Bytes.toBytes(value));
> return this;
> }
> {code}
> {code:java}
> // Bytes part
> public static byte[] toBytes(String s) {
> try {
> return s.getBytes(UTF8_CSN);
> } catch (UnsupportedEncodingException e) {
> // should never happen!
> throw new IllegalArgumentException("UTF8 decoding is not supported", e);
> }
> }
> {code}
> Call flow is:
> {code:java}
> admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
> Bytes.toBytes(key) && Bytes.toBytes(value) {code}
> from Bytes.toBytes, if s is null, the function will throw
> NullPointerException, but HTableDescriptor.setValue(String key, String value)
> does not check key and value.
> in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
> after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
> HTableDescriptor.setValue(String key, String value) does not check key and
> value is root cause.
> why branch below 2.0.0 works fine, because old code is :
> {code:java}
> public HTableDescriptor setValue(String key, String value) {
> if (value == null) {
> remove(key);
> } else {
> setValue(Bytes.toBytes(key), Bytes.toBytes(value));
> }
> return this;
> }
> {code}
> it check the value.
> since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
> it included:
> {code:java}
> public ModifyableTableDescriptor setValue(String key, String value) {
> return setValue(toBytesOrNull(key, Bytes::toBytes),
> toBytesOrNull(value, Bytes::toBytes));
> }
> {code}
> it checked key and value, but HTableDescriptor.setValue(String key, String
> value) does not call it, so
> just change HTableDescriptor.setValue(String key, String value) to call it,
> it will works
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Attachment: hbase-21699.001.patch
> Create table failed when using SPLITS_FILE => 'splits.txt'
> ---
>
> Key: HBASE-21699
> URL: https://issues.apache.org/jira/browse/HBASE-21699
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4
>Reporter: huan
>Priority: Minor
> Attachments: hbase-21699.001.patch
>
>
> Hi all:
> When I ran
> {code:java}
> create 't1', 'f1', SPLITS_FILE => 'splits.txt'
> {code}
> on HBase2.0.0, it failed, and no detailed error info, just like below:
> {code:java}
> ERROR:
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> {code}
> So I opened the debug:
> {code:java}
> hbase shell -d
> {code}
> and
> {code:java}
> ERROR:
> Backtrace:
> org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
> org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
> But it works on branch 1.2.0.
> so I view the source code, I find the issue is because the below code:
> {code:java}
> // admin.rb
> if arg.key?(SPLITS_FILE)
> splits_file = arg.delete(SPLITS_FILE)
> unless File.exist?(splits_file)
> raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
> end
> arg[SPLITS] = []
> File.foreach(splits_file) do |line|
> arg[SPLITS].push(line.chomp)
> end
> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
> end
> {code}
> {code:java}
> // HTableDescriptor part
> public HTableDescriptor setValue(String key, String value) {
> getDelegateeForModification().setValue(Bytes.toBytes(key),
> Bytes.toBytes(value));
> return this;
> }
> {code}
> {code:java}
> // Bytes part
> public static byte[] toBytes(String s) {
> try {
> return s.getBytes(UTF8_CSN);
> } catch (UnsupportedEncodingException e) {
> // should never happen!
> throw new IllegalArgumentException("UTF8 decoding is not supported", e);
> }
> }
> {code}
> Call flow is:
> {code:java}
> admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
> Bytes.toBytes(key) && Bytes.toBytes(value) {code}
> from Bytes.toBytes, if s is null, the function will throw
> NullPointerException, but HTableDescriptor.setValue(String key, String value)
> does not check key and value.
> in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
> after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
> HTableDescriptor.setValue(String key, String value) does not check key and
> value is root cause.
> why branch below 2.0.0 works fine, because old code is :
> {code:java}
> public HTableDescriptor setValue(String key, String value) {
> if (value == null) {
> remove(key);
> } else {
> setValue(Bytes.toBytes(key), Bytes.toBytes(value));
> }
> return this;
> }
> {code}
> it check the value.
> since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
> it included:
> {code:java}
> public ModifyableTableDescriptor setValue(String key, String value) {
> return setValue(toBytesOrNull(key, Bytes::toBytes),
> toBytesOrNull(value, Bytes::toBytes));
> }
> {code}
> it checked key and value, but HTableDescriptor.setValue(String key, String
> value) does not call it, so
> just change HTableDescriptor.setValue(String key, String value) to call it,
> it will works
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21695) Fix flaky test TestRegionServerAbortTimeout
[ https://issues.apache.org/jira/browse/HBASE-21695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737868#comment-16737868 ] stack commented on HBASE-21695: --- Patch looks good [~zghaobac] I was looking at 2.0 branch builds until recently and they weren't doing bad. Did something change sir to cause this breakage do you know? Thanks. > Fix flaky test TestRegionServerAbortTimeout > --- > > Key: HBASE-21695 > URL: https://issues.apache.org/jira/browse/HBASE-21695 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21695.master.001.patch, > HBASE-21695.master.001.patch, HBASE-21695.master.001.patch > > > [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/master/Flaky_20Test_20Report/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Affects Version/s: (was: 3.0.0)
> Create table failed when using SPLITS_FILE => 'splits.txt'
> ---
>
> Key: HBASE-21699
> URL: https://issues.apache.org/jira/browse/HBASE-21699
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4
>Reporter: huan
>Priority: Minor
> Attachments: hbase-21699.branch-2.0.patch
>
>
> Hi all:
> When I ran
> {code:java}
> create 't1', 'f1', SPLITS_FILE => 'splits.txt'
> {code}
> on HBase2.0.0, it failed, and no detailed error info, just like below:
> {code:java}
> ERROR:
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> {code}
> So I opened the debug:
> {code:java}
> hbase shell -d
> {code}
> and
> {code:java}
> ERROR:
> Backtrace:
> org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
> org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
> But it works on branch 1.2.0.
> so I view the source code, I find the issue is because the below code:
> {code:java}
> // admin.rb
> if arg.key?(SPLITS_FILE)
> splits_file = arg.delete(SPLITS_FILE)
> unless File.exist?(splits_file)
> raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
> end
> arg[SPLITS] = []
> File.foreach(splits_file) do |line|
> arg[SPLITS].push(line.chomp)
> end
> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
> end
> {code}
> {code:java}
> // HTableDescriptor part
> public HTableDescriptor setValue(String key, String value) {
> getDelegateeForModification().setValue(Bytes.toBytes(key),
> Bytes.toBytes(value));
> return this;
> }
> {code}
> {code:java}
> // Bytes part
> public static byte[] toBytes(String s) {
> try {
> return s.getBytes(UTF8_CSN);
> } catch (UnsupportedEncodingException e) {
> // should never happen!
> throw new IllegalArgumentException("UTF8 decoding is not supported", e);
> }
> }
> {code}
> Call flow is:
> {code:java}
> admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
> Bytes.toBytes(key) && Bytes.toBytes(value) {code}
> from Bytes.toBytes, if s is null, the function will throw
> NullPointerException, but HTableDescriptor.setValue(String key, String value)
> does not check key and value.
> in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
> after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
> HTableDescriptor.setValue(String key, String value) does not check key and
> value is root cause.
> why branch below 2.0.0 works fine, because old code is :
> {code:java}
> public HTableDescriptor setValue(String key, String value) {
> if (value == null) {
> remove(key);
> } else {
> setValue(Bytes.toBytes(key), Bytes.toBytes(value));
> }
> return this;
> }
> {code}
> it check the value.
> since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
> it included:
> {code:java}
> public ModifyableTableDescriptor setValue(String key, String value) {
> return setValue(toBytesOrNull(key, Bytes::toBytes),
> toBytesOrNull(value, Bytes::toBytes));
> }
> {code}
> it checked key and value, but HTableDescriptor.setValue(String key, String
> value) does not call it, so
> just change HTableDescriptor.setValue(String key, String value) to call it,
> it will works
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Attachment: hbase-21699.branch-2.0.patch
Status: Patch Available (was: Open)
> Create table failed when using SPLITS_FILE => 'splits.txt'
> ---
>
> Key: HBASE-21699
> URL: https://issues.apache.org/jira/browse/HBASE-21699
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.4, 2.1.2, 2.0.3, 2.0.2, 2.1.1, 2.0.1, 2.0.0, 3.0.0
>Reporter: huan
>Priority: Minor
> Attachments: hbase-21699.branch-2.0.patch
>
>
> Hi all:
> When I ran
> {code:java}
> create 't1', 'f1', SPLITS_FILE => 'splits.txt'
> {code}
> on HBase2.0.0, it failed, and no detailed error info, just like below:
> {code:java}
> ERROR:
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> {code}
> So I opened the debug:
> {code:java}
> hbase shell -d
> {code}
> and
> {code:java}
> ERROR:
> Backtrace:
> org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
> org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
> But it works on branch 1.2.0.
> so I view the source code, I find the issue is because the below code:
> {code:java}
> // admin.rb
> if arg.key?(SPLITS_FILE)
> splits_file = arg.delete(SPLITS_FILE)
> unless File.exist?(splits_file)
> raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
> end
> arg[SPLITS] = []
> File.foreach(splits_file) do |line|
> arg[SPLITS].push(line.chomp)
> end
> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
> end
> {code}
> {code:java}
> // HTableDescriptor part
> public HTableDescriptor setValue(String key, String value) {
> getDelegateeForModification().setValue(Bytes.toBytes(key),
> Bytes.toBytes(value));
> return this;
> }
> {code}
> {code:java}
> // Bytes part
> public static byte[] toBytes(String s) {
> try {
> return s.getBytes(UTF8_CSN);
> } catch (UnsupportedEncodingException e) {
> // should never happen!
> throw new IllegalArgumentException("UTF8 decoding is not supported", e);
> }
> }
> {code}
> Call flow is:
> {code:java}
> admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
> Bytes.toBytes(key) && Bytes.toBytes(value) {code}
> from Bytes.toBytes, if s is null, the function will throw
> NullPointerException, but HTableDescriptor.setValue(String key, String value)
> does not check key and value.
> in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
> after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
> HTableDescriptor.setValue(String key, String value) does not check key and
> value is root cause.
> why branch below 2.0.0 works fine, because old code is :
> {code:java}
> public HTableDescriptor setValue(String key, String value) {
> if (value == null) {
> remove(key);
> } else {
> setValue(Bytes.toBytes(key), Bytes.toBytes(value));
> }
> return this;
> }
> {code}
> it check the value.
> since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
> it included:
> {code:java}
> public ModifyableTableDescriptor setValue(String key, String value) {
> return setValue(toBytesOrNull(key, Bytes::toBytes),
> toBytesOrNull(value, Bytes::toBytes));
> }
> {code}
> it checked key and value, but HTableDescriptor.setValue(String key, String
> value) does not call it, so
> just change HTableDescriptor.setValue(String key, String value) to call it,
> it will works
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Description:
Hi all:
When I ran
{code:java}
create 't1', 'f1', SPLITS_FILE => 'splits.txt'
{code}
on HBase2.0.0, it failed, and no detailed error info, just like below:
{code:java}
ERROR:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:
{code}
So I opened the debug:
{code:java}
hbase shell -d
{code}
and
{code:java}
ERROR:
Backtrace:
org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
But it works on branch 1.2.0.
so I view the source code, I find the issue is because the below code:
{code:java}
// admin.rb
if arg.key?(SPLITS_FILE)
splits_file = arg.delete(SPLITS_FILE)
unless File.exist?(splits_file)
raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
end
arg[SPLITS] = []
File.foreach(splits_file) do |line|
arg[SPLITS].push(line.chomp)
end
htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
end
{code}
{code:java}
// HTableDescriptor part
public HTableDescriptor setValue(String key, String value) {
getDelegateeForModification().setValue(Bytes.toBytes(key),
Bytes.toBytes(value));
return this;
}
{code}
{code:java}
// Bytes part
public static byte[] toBytes(String s) {
try {
return s.getBytes(UTF8_CSN);
} catch (UnsupportedEncodingException e) {
// should never happen!
throw new IllegalArgumentException("UTF8 decoding is not supported", e);
}
}
{code}
Call flow is:
{code:java}
admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
Bytes.toBytes(key) && Bytes.toBytes(value) {code}
from Bytes.toBytes, if s is null, the function will throw NullPointerException,
but HTableDescriptor.setValue(String key, String value) does not check key and
value.
in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
HTableDescriptor.setValue(String key, String value) does not check key and
value is root cause.
why branch below 2.0.0 works fine, because old code is :
{code:java}
public HTableDescriptor setValue(String key, String value) {
if (value == null) {
remove(key);
} else {
setValue(Bytes.toBytes(key), Bytes.toBytes(value));
}
return this;
}
{code}
it check the value.
since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
it included:
{code:java}
public ModifyableTableDescriptor setValue(String key, String value) {
return setValue(toBytesOrNull(key, Bytes::toBytes),
toBytesOrNull(value, Bytes::toBytes));
}
{code}
it checked key and value, but HTableDescriptor.setValue(String key, String
value) does not call it, so
just change HTableDescriptor.setValue(String key, String value) to call it, it
will works
was:
Hi all:
When I ran
{code:java}
create 't1', 'f1', SPLITS_FILE => 'splits.txt'
{code}
on HBase2.0.0, it failed, and no detailed error info, just like below:
{code:java}
ERROR:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:
{code}
So I opened the debug:
{code:java}
hbase shell -d
{code}
and
{code:java}
ERROR:
Backtrace:
org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
But it works on branch 1.2.0.
so I view the source code, I find the issue is because the below code:
{code:java}
// admin.rb
if arg.key?(SPLITS_FILE)
splits_file = arg.delete(SPLITS_FILE)
unless File.exist?(splits_file)
raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
end
arg[SPLITS] = []
File.foreach(splits_file) do |line|
arg[SPLITS].push(line.chomp)
end
htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
end
{code}
{code:java}
// HTableDescriptor part
public HTableDescriptor setValue(String key, String value) {
getDelegateeForModification().setValue(Bytes.toBytes(key),
Bytes.toBytes(value));
return this;
}
{code}
{code:java}
// Bytes part
public static byte[] toBytes(String s) {
try {
return s.getBytes(UTF8_CSN);
} catch (UnsupportedEncodingException e) {
// should never happen!
throw new IllegalArgumentException("UTF8 decoding is not supported", e);
}
}
{code}
Call flow is:
{code:java}
[jira] [Commented] (HBASE-21698) Move version in branch-2.1 from 2.1.2 to 2.1.3-SNAPSHOT
[
https://issues.apache.org/jira/browse/HBASE-21698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737840#comment-16737840
]
Hudson commented on HBASE-21698:
Results for branch branch-2.1
[build #752 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/752/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/752//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/752//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/752//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move version in branch-2.1 from 2.1.2 to 2.1.3-SNAPSHOT
> ---
>
> Key: HBASE-21698
> URL: https://issues.apache.org/jira/browse/HBASE-21698
> Project: HBase
> Issue Type: Sub-task
> Components: release
>Reporter: stack
>Assignee: stack
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21695) Fix flaky test TestRegionServerAbortTimeout
[
https://issues.apache.org/jira/browse/HBASE-21695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737832#comment-16737832
]
Hadoop QA commented on HBASE-21695:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 39s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}148m 25s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaHighReplication
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21695 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954248/HBASE-21695.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux effb0837b8d3 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ebe3d1d1d9 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/15505/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15505/testReport/ |
| Max. process+thread count | 4884 (vs. ulimit of 1) |
|
[jira] [Commented] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[ https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737823#comment-16737823 ] Guanghao Zhang commented on HBASE-21645: Thanks [~nihaljain.cs] for contributing. > Perform sanity check and disallow table creation/modification with region > replication < 1 > - > > Key: HBASE-21645 > URL: https://issues.apache.org/jira/browse/HBASE-21645 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 1.5.0, 2.1.1, 2.1.2 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21645.master.001.patch, > HBASE-21645.master.001.patch, HBASE-21645.master.002.patch > > > We should perform sanity check and disallow table creation with region > replication < 1 or modification of an existing table with new region > replication value < 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[ https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21645: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.5 2.1.3 2.2.0 3.0.0 Status: Resolved (was: Patch Available) Pushed to branch-2.0+. > Perform sanity check and disallow table creation/modification with region > replication < 1 > - > > Key: HBASE-21645 > URL: https://issues.apache.org/jira/browse/HBASE-21645 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 1.5.0, 2.1.1, 2.1.2 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21645.master.001.patch, > HBASE-21645.master.001.patch, HBASE-21645.master.002.patch > > > We should perform sanity check and disallow table creation with region > replication < 1 or modification of an existing table with new region > replication value < 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Description:
Hi all:
When I ran
{code:java}
create 't1', 'f1', SPLITS_FILE => 'splits.txt'
{code}
on HBase2.0.0, it failed, and no detailed error info, just like below:
{code:java}
ERROR:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:
{code}
So I opened the debug:
{code:java}
hbase shell -d
{code}
and
{code:java}
ERROR:
Backtrace:
org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
But it works on branch 1.2.0.
so I view the source code, I find the issue is because the below code:
{code:java}
// admin.rb
if arg.key?(SPLITS_FILE)
splits_file = arg.delete(SPLITS_FILE)
unless File.exist?(splits_file)
raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
end
arg[SPLITS] = []
File.foreach(splits_file) do |line|
arg[SPLITS].push(line.chomp)
end
htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
end
{code}
{code:java}
// HTableDescriptor part
public HTableDescriptor setValue(String key, String value) {
getDelegateeForModification().setValue(Bytes.toBytes(key),
Bytes.toBytes(value));
return this;
}
{code}
{code:java}
// Bytes part
public static byte[] toBytes(String s) {
try {
return s.getBytes(UTF8_CSN);
} catch (UnsupportedEncodingException e) {
// should never happen!
throw new IllegalArgumentException("UTF8 decoding is not supported", e);
}
}
{code}
Call flow is:
{code:java}
admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
Bytes.toBytes(key) && Bytes.toBytes(value) {code}
from Bytes.toBytes, if s is null, the function will throw NullPointerException,
but HTableDescriptor.setValue(String key, String value) does not check key and
value.
in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
HTableDescriptor.setValue(String key, String value) does not check key and
value is root cause.
why branch below 2.0.0 works fine, because old code is :
{code:java}
public HTableDescriptor setValue(String key, String value) {
if (value == null) {
remove(key);
} else {
setValue(Bytes.toBytes(key), Bytes.toBytes(value));
}
return this;
}
{code}
it check the value.
since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
it included:
{code:java}
public ModifyableTableDescriptor setValue(String key, String value) {
return setValue(toBytesOrNull(key, Bytes::toBytes),
toBytesOrNull(value, Bytes::toBytes));
}
{code}
it checked key and value, but HTableDescriptor.setValue(String key, String
value) does not call it, so
just change HTableDescriptor.setValue(String key, String value) to call it, it
will works
was:
Hi all:
When I ran
{code:java}
create 't1', 'f1', SPLITS_FILE => 'splits.txt'
{code}
on HBase2.0.0, it failed, and no detailed error info, just like below:
{code:java}
ERROR:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:
{code}
So I opened the debug:
{code:java}
hbase shell -d
{code}
and
{code:java}
ERROR:
Backtrace:
org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
But it works on branch 1.2.0.
so I view the source code, I find the issue is because the below code:
{code:java}
// admin.rb
if arg.key?(SPLITS_FILE)
splits_file = arg.delete(SPLITS_FILE)
unless File.exist?(splits_file)
raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
end
arg[SPLITS] = []
File.foreach(splits_file) do |line|
arg[SPLITS].push(line.chomp)
end
htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
end
{code}
{code:java}
// HTableDescriptor part
public HTableDescriptor setValue(String key, String value) {
getDelegateeForModification().setValue(Bytes.toBytes(key),
Bytes.toBytes(value));
return this;
}
{code}
{code:java}
// Bytes part
public static byte[] toBytes(String s) {
try {
return s.getBytes(UTF8_CSN);
} catch (UnsupportedEncodingException e) {
// should never happen!
throw new IllegalArgumentException("UTF8 decoding is not
[jira] [Commented] (HBASE-21694) Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell commands
[ https://issues.apache.org/jira/browse/HBASE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737817#comment-16737817 ] Guanghao Zhang commented on HBASE-21694: [~Yi Mei] Add one line change in hbase-server module to trigger hbase-server unit tests? > Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell > commands > > > Key: HBASE-21694 > URL: https://issues.apache.org/jira/browse/HBASE-21694 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21694.master.001.patch, > HBASE-21694.master.002.patch, HBASE-21694.master.003.patch > > > To add or remove table-cfs to peers' exclude-table-cfs list, the current way > is use set_peer_exclude_tableCFs. But one has to copy all exclude-table-cfs > of the peer firstly and modify it, which is error-prone and time-consuming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21694) Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell commands
[ https://issues.apache.org/jira/browse/HBASE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737817#comment-16737817 ] Guanghao Zhang edited comment on HBASE-21694 at 1/9/19 3:57 AM: [~Yi Mei] Add one line change in hbase-server module to trigger hbase-server unit tests? Now the HADOOP QA only run hbase-client tests and hbase-shell tests. was (Author: zghaobac): [~Yi Mei] Add one line change in hbase-server module to trigger hbase-server unit tests? > Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell > commands > > > Key: HBASE-21694 > URL: https://issues.apache.org/jira/browse/HBASE-21694 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21694.master.001.patch, > HBASE-21694.master.002.patch, HBASE-21694.master.003.patch > > > To add or remove table-cfs to peers' exclude-table-cfs list, the current way > is use set_peer_exclude_tableCFs. But one has to copy all exclude-table-cfs > of the peer firstly and modify it, which is error-prone and time-consuming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737819#comment-16737819
]
Hadoop QA commented on HBASE-21614:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 37s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
10s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 17s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestClientClusterMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21614 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954257/HBASE-21614.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 34d9be86c723 4.4.0-131-generic #157~14.04.1-Ubuntu SMP Fri Jul
13 08:53:17 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
[
https://issues.apache.org/jira/browse/HBASE-21699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-21699:
-
Description:
Hi all:
When I ran
{code:java}
create 't1', 'f1', SPLITS_FILE => 'splits.txt'
{code}
on HBase2.0.0, it failed, and no detailed error info, just like below:
{code:java}
ERROR:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:
{code}
So I opened the debug:
{code:java}
hbase shell -d
{code}
and
{code:java}
ERROR:
Backtrace:
org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
But it works on branch 1.2.0.
so I view the source code, I find the issue is because the below code:
{code:java}
// admin.rb
if arg.key?(SPLITS_FILE)
splits_file = arg.delete(SPLITS_FILE)
unless File.exist?(splits_file)
raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
end
arg[SPLITS] = []
File.foreach(splits_file) do |line|
arg[SPLITS].push(line.chomp)
end
htd.setValue(SPLITS_FILE, arg[SPLITS_FILE])
end
{code}
{code:java}
// HTableDescriptor part
public HTableDescriptor setValue(String key, String value) {
getDelegateeForModification().setValue(Bytes.toBytes(key),
Bytes.toBytes(value));
return this;
}
{code}
{code:java}
// Bytes part
public static byte[] toBytes(String s) {
try {
return s.getBytes(UTF8_CSN);
} catch (UnsupportedEncodingException e) {
// should never happen!
throw new IllegalArgumentException("UTF8 decoding is not supported", e);
}
}
{code}
Call flow is:
{code:java}
admin.rb ---> htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) --->
Bytes.toBytes(key) && Bytes.toBytes(value) {code}
from Bytes.toBytes, if s is null, the function will throw NullPointerException,
but HTableDescriptor.setValue(String key, String value) does not check key and
value.
in admin.rb, it use arg.delete(SPLITS_FILE) to get the value, but this means,
after using arg.delete(SPLITS_FILE), arg[SPLITS_FILE] will return nil. so
HTableDescriptor.setValue(String key, String value) does not check key and
value is root cause.
why branch below 2.0.0 works fine, because old code is :
{code:java}
public HTableDescriptor setValue(String key, String value) {
if (value == null) {
remove(key);
} else {
setValue(Bytes.toBytes(key), Bytes.toBytes(value));
}
return this;
}
{code}
it check the value.
since branch 2.0.0, HBase add new function called 'TableDescriptorBuilder'
it included:
{code:java}
public ModifyableTableDescriptor setValue(String key, String value) {
return setValue(toBytesOrNull(key, Bytes::toBytes),
toBytesOrNull(value, Bytes::toBytes));
}
{code}
it checked key and value, but HTableDescriptor.setValue(String key, String
value) does not call it, so
just change HTableDescriptor.setValue(String key, String value) to call it, it
will works
> Create table failed when using SPLITS_FILE => 'splits.txt'
> ---
>
> Key: HBASE-21699
> URL: https://issues.apache.org/jira/browse/HBASE-21699
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 3.0.0, 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4
>Reporter: huan
>Priority: Minor
>
> Hi all:
> When I ran
> {code:java}
> create 't1', 'f1', SPLITS_FILE => 'splits.txt'
> {code}
>
> on HBase2.0.0, it failed, and no detailed error info, just like below:
>
> {code:java}
> ERROR:
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> {code}
> So I opened the debug:
>
>
> {code:java}
> hbase shell -d
> {code}
> and
>
>
> {code:java}
> ERROR:
> Backtrace:
> org.apache.hadoop.hbase.util.Bytes.toBytes(org/apache/hadoop/hbase/util/Bytes.java:732)
> org.apache.hadoop.hbase.HTableDescriptor.setValue(org/apache/hadoop/hbase/HTableDescriptor.java:190){code}
>
>
> But it works on branch 1.2.0.
> so I view the source code, I find the issue is because the below code:
> {code:java}
> // admin.rb
> if arg.key?(SPLITS_FILE)
> splits_file = arg.delete(SPLITS_FILE)
> unless File.exist?(splits_file)
> raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
> end
> arg[SPLITS] = []
> File.foreach(splits_file) do |line|
>
[jira] [Commented] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737812#comment-16737812
]
Sean Busbey commented on HBASE-21618:
-
that's great. thanks [~zghaobac]!
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21699) Create table failed when using SPLITS_FILE => 'splits.txt'
huan created HBASE-21699: Summary: Create table failed when using SPLITS_FILE => 'splits.txt' Key: HBASE-21699 URL: https://issues.apache.org/jira/browse/HBASE-21699 Project: HBase Issue Type: Bug Components: Client Affects Versions: 2.0.4, 2.1.2, 2.0.3, 2.0.2, 2.1.1, 2.0.1, 2.0.0, 3.0.0 Reporter: huan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21661) Provide Thrift2 implementation of Table/Admin
[ https://issues.apache.org/jira/browse/HBASE-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737781#comment-16737781 ] Allan Yang commented on HBASE-21661: [~stack], any other comments before I can commit this, sir? > Provide Thrift2 implementation of Table/Admin > - > > Key: HBASE-21661 > URL: https://issues.apache.org/jira/browse/HBASE-21661 > Project: HBase > Issue Type: Sub-task >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21661.patch, HBASE-21661.v2.patch, > HBASE-21661.v3.patch, HBASE-21661.v4.patch, HBASE-21661.v5.patch > > > Provide Thrift2 implementation of Table/Admin, making Java user to use thrift > client more easily(Some environment which can not expose ZK or RS Servers > directly require thrift or REST protocol even using Java). > Another Example of this is RemoteHTable and RemoteAdmin, they are REST > connectors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-21618.
Resolution: Fixed
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.0.4, 2.1.2
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-21618:
---
Release Note: There was a bug when scan with the same
startRow(inclusive=true) and stopRow(inclusive=false). The old incorrect
behavior is return one result. After this fix, the new correct behavior is
return nothing.
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reopened HBASE-21618:
Reopen to add a release note.
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737802#comment-16737802
]
Sean Busbey commented on HBASE-21618:
-
yeah it changes incorrect behavior and is (rightly) marked Critical. As-is the
only thing that'll show up in our release notes is the jira subject, which only
mentions the old incorrect behavior (without labeling it as such).
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21661) Provide Thrift2 implementation of Table/Admin
[ https://issues.apache.org/jira/browse/HBASE-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-21661: --- Release Note: ThriftAdmin/ThriftTable are implemented based on Thrift2. With ThriftAdmin/ThriftTable, People can use thrift2 protocol just like HTable/HBaseAdmin. Example of using ThriftConnection Configuration conf = HBaseConfiguration.create(); conf.set(ClusterConnection.HBASE_CLIENT_CONNECTION_IMPL,ThriftConnection.class.getName()); Connection conn = ConnectionFactory.createConnection(conf); Table table = conn.getTable(tablename) It is just like a normal Connection, similar use experience with the default ConnectionImplementation > Provide Thrift2 implementation of Table/Admin > - > > Key: HBASE-21661 > URL: https://issues.apache.org/jira/browse/HBASE-21661 > Project: HBase > Issue Type: Sub-task >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21661.patch, HBASE-21661.v2.patch, > HBASE-21661.v3.patch, HBASE-21661.v4.patch, HBASE-21661.v5.patch > > > Provide Thrift2 implementation of Table/Admin, making Java user to use thrift > client more easily(Some environment which can not expose ZK or RS Servers > directly require thrift or REST protocol even using Java). > Another Example of this is RemoteHTable and RemoteAdmin, they are REST > connectors. > ThriftAdmin/ThriftTable provides another good reason for Java user to use > thrift2 and not thrift1. Before, they need to convert HBase Get/Put/Scan... > to TGet/TPut/TScan and construct a thrift client all by themselves. With > ThriftAdmin/ThriftTable, they can use thrift protocol just like > HTable/HBaseAdmin. They are just like RemoteHTable/RemoteAdmin for REST > protocols. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21694) Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell commands
[
https://issues.apache.org/jira/browse/HBASE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737784#comment-16737784
]
Hadoop QA commented on HBASE-21694:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 101 new + 392 unchanged - 2 fixed
= 493 total (was 394) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 16s{color} | {color:orange} The patch generated 52 new + 532 unchanged - 0
fixed = 584 total (was 532) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
34s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
4s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21694 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954254/HBASE-21694.master.003.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux 5034ccbd4cb5
[jira] [Updated] (HBASE-21661) Provide Thrift2 implementation of Table/Admin
[ https://issues.apache.org/jira/browse/HBASE-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-21661: --- Description: Provide Thrift2 implementation of Table/Admin, making Java user to use thrift client more easily(Some environment which can not expose ZK or RS Servers directly require thrift or REST protocol even using Java). Another Example of this is RemoteHTable and RemoteAdmin, they are REST connectors. ThriftAdmin/ThriftTable provides another good reason for Java user to use thrift2 and not thrift1. Before, they need to convert HBase Get/Put/Scan... to TGet/TPut/TScan and construct a thrift client all by themselves. With ThriftAdmin/ThriftTable, they can use thrift protocol just like HTable/HBaseAdmin. They are just like RemoteHTable/RemoteAdmin for REST protocols. was: Provide Thrift2 implementation of Table/Admin, making Java user to use thrift client more easily(Some environment which can not expose ZK or RS Servers directly require thrift or REST protocol even using Java). Another Example of this is RemoteHTable and RemoteAdmin, they are REST connectors. > Provide Thrift2 implementation of Table/Admin > - > > Key: HBASE-21661 > URL: https://issues.apache.org/jira/browse/HBASE-21661 > Project: HBase > Issue Type: Sub-task >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21661.patch, HBASE-21661.v2.patch, > HBASE-21661.v3.patch, HBASE-21661.v4.patch, HBASE-21661.v5.patch > > > Provide Thrift2 implementation of Table/Admin, making Java user to use thrift > client more easily(Some environment which can not expose ZK or RS Servers > directly require thrift or REST protocol even using Java). > Another Example of this is RemoteHTable and RemoteAdmin, they are REST > connectors. > ThriftAdmin/ThriftTable provides another good reason for Java user to use > thrift2 and not thrift1. Before, they need to convert HBase Get/Put/Scan... > to TGet/TPut/TScan and construct a thrift client all by themselves. With > ThriftAdmin/ThriftTable, they can use thrift protocol just like > HTable/HBaseAdmin. They are just like RemoteHTable/RemoteAdmin for REST > protocols. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21634) Print error message when user uses unacceptable values for LIMIT while setting quotas.
[
https://issues.apache.org/jira/browse/HBASE-21634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737774#comment-16737774
]
Hadoop QA commented on HBASE-21634:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
9s{color} | {color:red} The patch generated 9 new + 124 unchanged - 1 fixed =
133 total (was 125) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 8s{color} | {color:orange} The patch generated 48 new + 278 unchanged - 0
fixed = 326 total (was 278) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
20s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21634 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954255/hbase-21634.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit rubocop
ruby_lint |
| uname | Linux 23ab8747e8e2 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ebe3d1d1d9 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| rubocop | v0.62.0 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/15509/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| ruby-lint |
https://builds.apache.org/job/PreCommit-HBASE-Build/15509/artifact/patchprocess/diff-patch-ruby-lint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15509/testReport/ |
| Max. process+thread count | 2226 (vs. ulimit of 1) |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15509/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Print error message when user uses unacceptable values for LIMIT while
> setting quotas.
> --
>
> Key: HBASE-21634
> URL: https://issues.apache.org/jira/browse/HBASE-21634
> Project: HBase
> Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-21634.master.001.patch
>
>
> When unacceptable value(like 2.3G or 70H) to LIMIT are passed while setting
> quotas, we currently do not print any error message (informing the user about
> the erroneous input). Like below:
> {noformat}
> hbase(main):002:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '2.3G',
> POLICY => NO_WRITES
> Took 0.0792 seconds
> hbase(main):003:0> list_quotas
> OWNERQUOTAS
> TABLE => t2 TYPE => SPACE,
> TABLE => t2, LIMIT => 2B, VIOLATION_POLICY
[jira] [Updated] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21614:
--
Attachment: HBASE-21614.patch
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614.01.patch, HBASE-21614.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21694) Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell commands
[ https://issues.apache.org/jira/browse/HBASE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737766#comment-16737766 ] Guanghao Zhang commented on HBASE-21694: +1. Wait the Hadoop QA result. > Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell > commands > > > Key: HBASE-21694 > URL: https://issues.apache.org/jira/browse/HBASE-21694 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21694.master.001.patch, > HBASE-21694.master.002.patch, HBASE-21694.master.003.patch > > > To add or remove table-cfs to peers' exclude-table-cfs list, the current way > is use set_peer_exclude_tableCFs. But one has to copy all exclude-table-cfs > of the peer firstly and modify it, which is error-prone and time-consuming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21634) Print error message when user uses unacceptable values for LIMIT while setting quotas.
[
https://issues.apache.org/jira/browse/HBASE-21634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-21634:
---
Attachment: hbase-21634.master.001.patch
Status: Patch Available (was: In Progress)
Added test cases as well for both Space & Throttle quota (invalid limit values).
> Print error message when user uses unacceptable values for LIMIT while
> setting quotas.
> --
>
> Key: HBASE-21634
> URL: https://issues.apache.org/jira/browse/HBASE-21634
> Project: HBase
> Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-21634.master.001.patch
>
>
> When unacceptable value(like 2.3G or 70H) to LIMIT are passed while setting
> quotas, we currently do not print any error message (informing the user about
> the erroneous input). Like below:
> {noformat}
> hbase(main):002:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '2.3G',
> POLICY => NO_WRITES
> Took 0.0792 seconds
> hbase(main):003:0> list_quotas
> OWNERQUOTAS
> TABLE => t2 TYPE => SPACE,
> TABLE => t2, LIMIT => 2B, VIOLATION_POLICY => NO_WRITES
> 1 row(s)
> Took 0.0512 seconds
> hbase(main):006:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '70H',
> POLICY => NO_WRITES
> Took 0.0088 seconds
> hbase(main):007:0> list_quotas
> OWNERQUOTAS
> TABLE => t2 TYPE => SPACE,
> TABLE => t2, LIMIT => 70B, VIOLATION_POLICY => NO_WRITES
> 1 row(s)
> Took 0.0448 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21591) Support ability to have host based permissions
[
https://issues.apache.org/jira/browse/HBASE-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737756#comment-16737756
]
Andrew Purtell commented on HBASE-21591:
This seems fine, given the available detail to evaluate the proposal. Try to
minimize coprocessor API changes. For example, consider using the RPC static
helper method to retrieve the client IP address from the RPC context and put it
in the observer context instead of passing it through as a new method
parameter. Changes to private interfaces will all be fine.
> Support ability to have host based permissions
> --
>
> Key: HBASE-21591
> URL: https://issues.apache.org/jira/browse/HBASE-21591
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Trivial
>
> Today, one can put in an ACL rule where a user is not permitted to read data
> but can insert data (e.g. {{grant 'user', 'table', 'W'}}). However, one can
> not implement HBase as a "drop-box" for data where by in a secure network,
> one can read and write data but outside that secure network one can only
> write data; and I do not believe this is possible with custom access
> controllers, unless one "wraps" HBase; e.g. with the HBase REST server.
> I have been pushing for this model (e.g. [Of Data Dropboxes and Data
> Gloveboxes|https://thestrangeloop.com/2018/of-data-dropboxes-and-data-gloveboxes.html]
> or
> [slides|http://clayb.net/presentations/Of%20Data%20Dropboxes%20and%20Data%20Gloveboxes.pdf])
> in a number of technologies for some data compartmentalization initiatives.
> I propose passing the requester's host information through the HBase
> authentication stack so that the ACL model in HBase can work akin to the SQL
> semantics of {{user@host}} or {{user@}}.The expected impact would
> be to HBase private interfaces only, so far in POC'ing it seems the following
> would be impacted:
> Access Control Classes/ACL Table Management:
> * AccessControlUtil
> * UserPermission
> * AccessChecker
> * AccessControlFilter
> * AccessController
> * AuthResult
> * TableAuthManager
> * AccessControl.proto
> Co-Processor APIs for Checking Authentication:
> * CoprocessorHost
> * ObserverContext
> * ObserverContextImpl
> * RSRpcServices
> * RSGroupAdminEndpoint
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21614:
--
Fix Version/s: 2.2.0
3.0.0
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21614:
--
Component/s: amv2
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737753#comment-16737753
]
Duo Zhang commented on HBASE-21614:
---
OK, seems reasonable. Let me commit.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737751#comment-16737751
]
Duo Zhang commented on HBASE-21614:
---
Oh, just aw the second paragraph. Let me check.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20925) Canary test to expose table availability rate
[
https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737752#comment-16737752
]
Hadoop QA commented on HBASE-20925:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-20925 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20925 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935757/HBASE-20925.master.005.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15508/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Canary test to expose table availability rate
> --
>
> Key: HBASE-20925
> URL: https://issues.apache.org/jira/browse/HBASE-20925
> Project: HBase
> Issue Type: Improvement
> Components: canary
>Affects Versions: 3.0.0, 2.0.0, 1.4.6
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Major
> Labels: Canary
> Attachments: HBASE-20925.master.001.patch,
> HBASE-20925.master.002.patch, HBASE-20925.master.003.patch,
> HBASE-20925.master.004.patch, HBASE-20925.master.005.patch
>
>
> Canary test to expose table availability rate.
>
> It will print table availability rate such as below.
>
>
> *2018-07-27 17:11:06,823 INFO [CanaryMonitor-1532736665083] tool.Canary:
> *
> *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary:
> === Summary: ===*
> *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read
> success rate for table : MyTable is: 1.0 .*
> *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read
> success rate for table : mytable3 is: 0.9*
> *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read
> success rate for table : mytable2 is: 0.8*
> *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read
> success rate for table : mytable4 is: 1.0*
> *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary:
> ===END==*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737750#comment-16737750
]
Duo Zhang commented on HBASE-21614:
---
{quote}
and the server crash procedure will not unblock the RIT because it only
unblocks RITs for open regions on the server.
{quote}
Can you confirm this again? IIRC, when opening, we will set the region location
to the target server and update meta, and when closing, we will not change the
region location when updating meta, only after the region is closed, we will
set the region location to null, and also set the previous region location to
lastHost.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20925) Canary test to expose table availability rate
[ https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737747#comment-16737747 ] Andrew Purtell commented on HBASE-20925: Is this still needed? What is the utility for the rate computation? Would it better to print a line for every successful or failed probe and let the tool post-processing the output make this computation if it is needed? Raw output would be more useful and flexible than this cooked metric? > Canary test to expose table availability rate > -- > > Key: HBASE-20925 > URL: https://issues.apache.org/jira/browse/HBASE-20925 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 3.0.0, 2.0.0, 1.4.6 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Major > Labels: Canary > Attachments: HBASE-20925.master.001.patch, > HBASE-20925.master.002.patch, HBASE-20925.master.003.patch, > HBASE-20925.master.004.patch, HBASE-20925.master.005.patch > > > Canary test to expose table availability rate. > > It will print table availability rate such as below. > > > *2018-07-27 17:11:06,823 INFO [CanaryMonitor-1532736665083] tool.Canary: > * > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: > === Summary: ===* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : MyTable is: 1.0 .* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable3 is: 0.9* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable2 is: 0.8* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable4 is: 1.0* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: > ===END==* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21697) Add 2.1.2 to the download page
[ https://issues.apache.org/jira/browse/HBASE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737745#comment-16737745 ] Duo Zhang commented on HBASE-21697: --- Sir, the warning for 2.1.1 and 2.0.3 on the download page seems disappeared? > Add 2.1.2 to the download page > -- > > Key: HBASE-21697 > URL: https://issues.apache.org/jira/browse/HBASE-21697 > Project: HBase > Issue Type: Sub-task > Components: 2.1.2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21694) Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell commands
[ https://issues.apache.org/jira/browse/HBASE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-21694: --- Attachment: HBASE-21694.master.003.patch > Add append_peer_exclude_tableCFs and remove_peer_exclude_tableCFs shell > commands > > > Key: HBASE-21694 > URL: https://issues.apache.org/jira/browse/HBASE-21694 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Attachments: HBASE-21694.master.001.patch, > HBASE-21694.master.002.patch, HBASE-21694.master.003.patch > > > To add or remove table-cfs to peers' exclude-table-cfs list, the current way > is use set_peer_exclude_tableCFs. But one has to copy all exclude-table-cfs > of the peer firstly and modify it, which is error-prone and time-consuming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21625) a runnable procedure v2 does not run
[
https://issues.apache.org/jira/browse/HBASE-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737741#comment-16737741
]
Duo Zhang commented on HBASE-21625:
---
There is no waiting, the programming model is event driven. The remote rpc call
will return immediately, and later the rs will report back, through another rpc
call.
So the problem here maybe:
1. The RS does not call reportRIT.
2. The RS is dead actually but hangs there forever and master does not schedule
a SCP for it.
3. The RS is dead and has been proceeded by master but there is a race which
causes that we haven't interrupted the TRSP so the OpenRegionProcedure hangs
forever.
> a runnable procedure v2 does not run
>
>
> Key: HBASE-21625
> URL: https://issues.apache.org/jira/browse/HBASE-21625
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> This is on master snapshot as of a few weeks ago.
> Haven't looked at the code much yet, but it seems rather fundamental. The
> procedure comes from meta replica assignment (HBASE-21624), in case it
> matters w.r.t. the engine initialization; however, the master is functional
> and other procedures run fine. I can also see lots of other open region
> procedures with a similar patterns that were initialized before this one and
> have run fine.
> Currently, there are no other runnable procedures on master - a lot of
> succeeded procedures since then, the parent blocked on this procedure, and
> one unrelated RIT procedure waiting with timeout and being updated
> periodically.
> The procedure itself is
> {noformat}
> 157156157155 RUNNABLEhadoop
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure Wed Dec 19
> 17:20:27 PST 2018Wed Dec 19 17:20:28 PST 2018[ { region => {
> regionId => '1', tableName => { ... }, startKey => '', endKey => '', offline
> => 'false', split => 'false', replicaId => '1' }, targetServer => { hostName
> => 'server1', port => '17020', startCode => '1545266805778' } }, {} ]
> {noformat}
> This is in PST so it's been like that for ~19 hours.
> The only line involving this PID in the log is {noformat}
> 2018-12-19 17:20:27,974 INFO [PEWorker-4] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=157156, ppid=157155, state=RUNNABLE,
> hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}]
> {noformat}
> There are no other useful logs for either this PID, parent PID, or region in
> question since. This PEWorker (4) is also alive and did some work since then,
> so it's not like the thread errored out somewhere.
> All the PEWorker-s are waiting for work:
> {noformat}
> Thread 158 (PEWorker-16):
> State: TIMED_WAITING
> Blocked count: 1340
> Waited count: 5064
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>
> org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:171)
>
> org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:153)
>
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1949)
> {noformat}
> The main assignment procedure for this region is blocked on it:
> {noformat}
> 157155WAITING hadoop TransitRegionStateProcedure
> table=hbase:meta, region=534574363, ASSIGN Wed Dec 19 17:20:27 PST 2018
> Wed Dec 19 17:20:27 PST 2018[ { state => [ '1', '2', '3' ] }, {
> regionId => '1', tableName => { ... }, startKey => '', endKey => '', offline
> => 'false', split => 'false', replicaId => '1' }, { initialState =>
> 'REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE', lastState =>
> 'REGION_STATE_TRANSITION_CONFIRM_OPENED', assignCandidate => { hostName =>
> 'server1', port => '17020', startCode => '1545266805778' }, forceNewPlan =>
> 'false' } ]
> 2018-12-19 17:20:27,673 INFO [PEWorker-9]
> procedure.MasterProcedureScheduler: Took xlock for pid=157155,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false;
> TransitRegionStateProcedure table=hbase:meta, region=..., ASSIGN
> 2018-12-19 17:20:27,809 INFO [PEWorker-9]
> assignment.TransitRegionStateProcedure: Starting pid=157155,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
> TransitRegionStateProcedure table=hbase:meta, region=..., ASSIGN;
> rit=OFFLINE, location=server1,17020,1545266805778; forceNewPlan=false,
> retain=false
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737729#comment-16737729
]
Guanghao Zhang commented on HBASE-21618:
[~busbey] This is a bug fix. Need a release note, too?
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[ https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737719#comment-16737719 ] Guanghao Zhang commented on HBASE-21645: +1 > Perform sanity check and disallow table creation/modification with region > replication < 1 > - > > Key: HBASE-21645 > URL: https://issues.apache.org/jira/browse/HBASE-21645 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 1.5.0, 2.1.1, 2.1.2 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Attachments: HBASE-21645.master.001.patch, > HBASE-21645.master.001.patch, HBASE-21645.master.002.patch > > > We should perform sanity check and disallow table creation with region > replication < 1 or modification of an existing table with new region > replication value < 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21695) Fix flaky test TestRegionServerAbortTimeout
[ https://issues.apache.org/jira/browse/HBASE-21695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21695: --- Attachment: HBASE-21695.master.001.patch > Fix flaky test TestRegionServerAbortTimeout > --- > > Key: HBASE-21695 > URL: https://issues.apache.org/jira/browse/HBASE-21695 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21695.master.001.patch, > HBASE-21695.master.001.patch, HBASE-21695.master.001.patch > > > [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/master/Flaky_20Test_20Report/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21691) Fix flaky test TestRecoveredEdits
[ https://issues.apache.org/jira/browse/HBASE-21691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21691: --- Attachment: HBASE-21691.master.001.patch > Fix flaky test TestRecoveredEdits > - > > Key: HBASE-21691 > URL: https://issues.apache.org/jira/browse/HBASE-21691 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21691.master.001.patch, > HBASE-21691.master.001.patch, HBASE-21691.master.001.patch > > > TestRecoveredEdits failed a lot times in precommit jobs. > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/master/Flaky_20Test_20Report/ > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21625) a runnable procedure v2 does not run
[
https://issues.apache.org/jira/browse/HBASE-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737695#comment-16737695
]
Sergey Shelukhin commented on HBASE-21625:
--
Hmm, the remote call didn't come back for 19 hours then, or it wasn't made. I
also don't see any jstacks waiting for remote call...
> a runnable procedure v2 does not run
>
>
> Key: HBASE-21625
> URL: https://issues.apache.org/jira/browse/HBASE-21625
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> This is on master snapshot as of a few weeks ago.
> Haven't looked at the code much yet, but it seems rather fundamental. The
> procedure comes from meta replica assignment (HBASE-21624), in case it
> matters w.r.t. the engine initialization; however, the master is functional
> and other procedures run fine. I can also see lots of other open region
> procedures with a similar patterns that were initialized before this one and
> have run fine.
> Currently, there are no other runnable procedures on master - a lot of
> succeeded procedures since then, the parent blocked on this procedure, and
> one unrelated RIT procedure waiting with timeout and being updated
> periodically.
> The procedure itself is
> {noformat}
> 157156157155 RUNNABLEhadoop
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure Wed Dec 19
> 17:20:27 PST 2018Wed Dec 19 17:20:28 PST 2018[ { region => {
> regionId => '1', tableName => { ... }, startKey => '', endKey => '', offline
> => 'false', split => 'false', replicaId => '1' }, targetServer => { hostName
> => 'server1', port => '17020', startCode => '1545266805778' } }, {} ]
> {noformat}
> This is in PST so it's been like that for ~19 hours.
> The only line involving this PID in the log is {noformat}
> 2018-12-19 17:20:27,974 INFO [PEWorker-4] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=157156, ppid=157155, state=RUNNABLE,
> hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}]
> {noformat}
> There are no other useful logs for either this PID, parent PID, or region in
> question since. This PEWorker (4) is also alive and did some work since then,
> so it's not like the thread errored out somewhere.
> All the PEWorker-s are waiting for work:
> {noformat}
> Thread 158 (PEWorker-16):
> State: TIMED_WAITING
> Blocked count: 1340
> Waited count: 5064
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>
> org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:171)
>
> org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:153)
>
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1949)
> {noformat}
> The main assignment procedure for this region is blocked on it:
> {noformat}
> 157155WAITING hadoop TransitRegionStateProcedure
> table=hbase:meta, region=534574363, ASSIGN Wed Dec 19 17:20:27 PST 2018
> Wed Dec 19 17:20:27 PST 2018[ { state => [ '1', '2', '3' ] }, {
> regionId => '1', tableName => { ... }, startKey => '', endKey => '', offline
> => 'false', split => 'false', replicaId => '1' }, { initialState =>
> 'REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE', lastState =>
> 'REGION_STATE_TRANSITION_CONFIRM_OPENED', assignCandidate => { hostName =>
> 'server1', port => '17020', startCode => '1545266805778' }, forceNewPlan =>
> 'false' } ]
> 2018-12-19 17:20:27,673 INFO [PEWorker-9]
> procedure.MasterProcedureScheduler: Took xlock for pid=157155,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false;
> TransitRegionStateProcedure table=hbase:meta, region=..., ASSIGN
> 2018-12-19 17:20:27,809 INFO [PEWorker-9]
> assignment.TransitRegionStateProcedure: Starting pid=157155,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
> TransitRegionStateProcedure table=hbase:meta, region=..., ASSIGN;
> rit=OFFLINE, location=server1,17020,1545266805778; forceNewPlan=false,
> retain=false
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737692#comment-16737692
]
Sergey Shelukhin edited comment on HBASE-21614 at 1/9/19 12:44 AM:
---
[~Apache9] After master restart the region can be in opening/closing state in
meta on a dead server, RIT will be stuck forever trying to confirm it's closed
on that dead server, and the server crash procedure will not unblock the RIT
because it only unblocks RITs for open regions on the server.
This is caused by inconsistent master structure state in steady state vs
startup - in steady state we add regions to server region list in open, opening
and closing, but in startup logic we only add open regions.
was (Author: sershe):
[~Apache9] After master restart the region can be in opening/closing state in
meta on a dead server, RIT will be stuck forever trying to confirm it's closed
on that dead server, and the server crash procedure will not unblock the RIT
because it only unblocks RITs for open regions on the server.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015,
[jira] [Comment Edited] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737692#comment-16737692
]
Sergey Shelukhin edited comment on HBASE-21614 at 1/9/19 12:43 AM:
---
[~Apache9] After master restart the region can be in opening/closing state in
meta on a dead server, RIT will be stuck forever trying to confirm it's closed
on that dead server, and the server crash procedure will not unblock the RIT
because it only unblocks RITs for open regions on the server.
was (Author: sershe):
[~Apache9] After master restart the region can be in opening/closing state in
meta, RIT will be stuck forever trying to confirm it's closed on dead server,
and the server crash procedure will not unblock the RIT because it only
unblocks RITs for open regions on the server.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure
[jira] [Commented] (HBASE-21614) RIT recovery with ServerCrashProcedure doesn't account for all regions
[
https://issues.apache.org/jira/browse/HBASE-21614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737692#comment-16737692
]
Sergey Shelukhin commented on HBASE-21614:
--
[~Apache9] After master restart the region can be in opening/closing state in
meta, RIT will be stuck forever trying to confirm it's closed on dead server,
and the server crash procedure will not unblock the RIT because it only
unblocks RITs for open regions on the server.
> RIT recovery with ServerCrashProcedure doesn't account for all regions
> --
>
> Key: HBASE-21614
> URL: https://issues.apache.org/jira/browse/HBASE-21614
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21614.01.patch
>
>
> Master is restarting after a previous master crashed while recovering some
> regions from a dead server.
> Master recovers RIT for the region, however the RIT has no location (logged,
> at least) in CONFIRM_CLOSE state. That is a potential problem #0.5 - confirm
> where? But that should be covered by meta, so not a big deal, right. As such
> it doesn't seem to add the region to server map anywhere
> {noformat}
> 2018-12-17 14:51:14,606 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Attach pid=38015,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE to
> rit=OFFLINE, location=null, table=t1, region=region1 to restore RIT
> {noformat}
> -However, in this case ServerCrashProcedure for the server kicks off BEFORE
> meta is loaded-.
> -That seems to be a problem #1 - it immediately gets regions to later
> recover, so in this case it gets nothing-.
> -I've grepped our logs for successful cases of SCP interacting with region
> transition at startup, and in all cases the meta was loaded before SCP-.
> -Seems like a race condition-. Looks like SCP handles it
> {noformat}
> 2018-12-17 14:51:14,625 INFO [master/:17000:becomeActiveMaster]
> master.RegionServerTracker: Starting RegionServerTracker; 0 have existing
> ServerCrashProcedures, 103 possibly 'live' servers, and 1 'splitting'.
> 2018-12-17 14:51:20,770 INFO [master/:17000:becomeActiveMaster]
> master.ServerManager: Processing expiration of server1,17020,1544636616174 on
> ,17000,1545087053243
> 2018-12-17 14:51:20,921 INFO [master/:17000:becomeActiveMaster]
> assignment.AssignmentManager: Added server1,17020,1544636616174 to dead
> servers which carryingMeta=false, submitted ServerCrashProcedure pid=111298
> 2018-12-17 14:51:30,728 INFO [PEWorker-13] procedure.ServerCrashProcedure:
> Start pid=111298, state=RUNNABLE:SERVER_CRASH_START, hasLock=true;
> ServerCrashProcedure server=server1,17020,1544636616174, splitWal=true,
> meta=false
> {noformat}
> Meta is only loaded 11-12 seconds later.
> If one looks at meta-loading code however, there is one more problem #2 - the
> region is in CLOSING state, so the {{addRegionToServer}} is not going to be
> called - it's only called for OPENED regions.
> Expanding on the above, I've only seen SCP unblock stuck region transition at
> startup when region started out in meta as OPEN.
> {noformat}
> 2018-12-17 14:51:42,403 INFO [master/:17000:becomeActiveMaster]
> assignment.RegionStateStore: Load hbase:meta entry region=region1,
> regionState=CLOSING, lastHost=server1,17020,1544636616174,
> regionLocation=server1,17020,1544636616174, openSeqNum=629131
> {noformat}
> SCP predictably finishes without doing anything; no other logs for this pid
> {noformat}
> 2018-12-17 14:52:19,046 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=111298, state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=server1,17020,1544636616174, splitWal=true, meta=false in 58.0010sec
> {noformat}
> After that, region is still stuck trying to be closed in
> TransitRegionStateProcedure; it's in the same state for hours including
> across master restarts.
> {noformat}
> 2018-12-17 15:09:35,216 WARN [PEWorker-14]
> assignment.TransitRegionStateProcedure: Failed transition, suspend 604secs
> pid=38015, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true;
> TransitRegionStateProcedure table=t1, region=region1, REOPEN/MOVE;
> rit=CLOSING, location=server1,17020,1544636616174; waiting on rectified
> condition fixed by other Procedure or operator intervention
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21626) log the regions blocking WAL from being archived
[
https://issues.apache.org/jira/browse/HBASE-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737680#comment-16737680
]
Hadoop QA commented on HBASE-21626:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 38s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}131m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21626 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954228/HBASE-21626.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a48230e8cf87 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d26acbe1db |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15504/testReport/ |
| Max. process+thread count | 5060 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15504/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> log the regions
[jira] [Resolved] (HBASE-21698) Move version in branch-2.1 from 2.1.2 to 2.1.3-SNAPSHOT
[
https://issues.apache.org/jira/browse/HBASE-21698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-21698.
---
Resolution: Fixed
Pushed to branch-2.1. Here is what I did:
592 git pull --rebase
593 mvn clean org.codehaus.mojo:versions-maven-plugin:2.5:set
-DnewVersion=2.1.3-SNAPSHOT
594 find . -name pom.xml -exec git add {} \;
595 git status
596 git commit -m "HBASE-21698 Move version in branch-2.1 from 2.1.2 to
2.1.3-SNAPSHOT"
597 git push origin HEAD:branch-2.1
> Move version in branch-2.1 from 2.1.2 to 2.1.3-SNAPSHOT
> ---
>
> Key: HBASE-21698
> URL: https://issues.apache.org/jira/browse/HBASE-21698
> Project: HBase
> Issue Type: Sub-task
> Components: release
>Reporter: stack
>Assignee: stack
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21698) Move version in branch-2.1 from 2.1.2 to 2.1.3-SNAPSHOT
stack created HBASE-21698: - Summary: Move version in branch-2.1 from 2.1.2 to 2.1.3-SNAPSHOT Key: HBASE-21698 URL: https://issues.apache.org/jira/browse/HBASE-21698 Project: HBase Issue Type: Sub-task Components: release Reporter: stack Assignee: stack -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21697) Add 2.1.2 to the download page
[ https://issues.apache.org/jira/browse/HBASE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-21697. --- Resolution: Fixed Fix Version/s: 2.1.2 Pushed edits to master branch. > Add 2.1.2 to the download page > -- > > Key: HBASE-21697 > URL: https://issues.apache.org/jira/browse/HBASE-21697 > Project: HBase > Issue Type: Sub-task > Components: 2.1.2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21697) Add 2.1.2 to the download page
stack created HBASE-21697: - Summary: Add 2.1.2 to the download page Key: HBASE-21697 URL: https://issues.apache.org/jira/browse/HBASE-21697 Project: HBase Issue Type: Sub-task Components: 2.1.2 Reporter: stack Assignee: stack -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21683) Reset readsEnabled flag after successfully flushing the primary region
[
https://issues.apache.org/jira/browse/HBASE-21683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737637#comment-16737637
]
Hudson commented on HBASE-21683:
Results for branch branch-2
[build #1598 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reset readsEnabled flag after successfully flushing the primary region
> --
>
> Key: HBASE-21683
> URL: https://issues.apache.org/jira/browse/HBASE-21683
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21683.patch, HBASE-21683.patch
>
>
> Found the problem when implementing HBASE-21671, TestRegionReplicaFailover is
> easy to fail. In this test we set
> REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY to true, which means the
> secondary replicas can not be read until the primary region has been
> successfully flushed. But in RegionReplicaFlushHandler, there is only one
> place where we call region.setReadsEnabled(true), this is incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21296) [2.1] Upgrade Jetty dependencies to latest in major-line
[ https://issues.apache.org/jira/browse/HBASE-21296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737639#comment-16737639 ] stack commented on HBASE-21296: --- Ok. +1 on commit. Will roll a 2.1.3 soon. > [2.1] Upgrade Jetty dependencies to latest in major-line > > > Key: HBASE-21296 > URL: https://issues.apache.org/jira/browse/HBASE-21296 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.1.3 > > Attachments: HBASE-21282.001.branch-2.0.patch > > > Looks like we have dependencies on both jetty 9.2 and 9.3, but we're lagging > pretty far behind in both. We can upgrade both of these to the latest (august > 2018). > > I'll also have to take a look at why we're using two separate versions (maybe > we didn't want to switch from jetty-jsp to apache-jsp on 9.2->9.3?). Not sure > if there's a good reason for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21588) Procedure v2 wal splitting implementation
[
https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737638#comment-16737638
]
Hudson commented on HBASE-21588:
Results for branch branch-2
[build #1598 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1598//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Procedure v2 wal splitting implementation
> -
>
> Key: HBASE-21588
> URL: https://issues.apache.org/jira/browse/HBASE-21588
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21588.branch-2.001.patch,
> HBASE-21588.branch-2.002.patch, HBASE-21588.master.003.patch,
> HBASE-21588.master.004.patch, HBASE-21588.master.005.patch,
> HBASE-21588.master.006.patch, HBASE-21588.master.007.patch,
> HBASE-21588.master.008.patch, HBASE-21588.master.009.patch,
> HBASE-21588.master.010.patch, HBASE-21588.master.011.patch,
> HBASE-21588.master.012.patch, HBASE-21588.master.013.patch,
> HBASE-21588.master.014.patch, HBASE-21588.master.015.patch,
> HBASE-21588.master.016.patch, HBASE-21588.master.017.patch,
> HBASE-21588.master.018.patch, HBASE-21588.master.018.patch,
> HBASE-21588.master.019.patch, HBASE-21588.master.019.patch,
> HBASE-21588.master.020.patch
>
>
> create a sub task to submit the implementation of procedure v2 wal splitting
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21687) Update Findbugs Maven Plugin to 3.0.4 to work with Maven 3.6.0+ [branches-1]
[
https://issues.apache.org/jira/browse/HBASE-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737612#comment-16737612
]
Hudson commented on HBASE-21687:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #515 (See
[https://builds.apache.org/job/HBase-1.3-IT/515/])
HBASE-21687 Update Findbugs Maven Plugin to 3.0.4 to work with Maven (apurtell:
[https://github.com/apache/hbase/commit/42fca2364aa3fd2f9b7fdbba92dc2b16616f31d6])
* (edit) pom.xml
> Update Findbugs Maven Plugin to 3.0.4 to work with Maven 3.6.0+ [branches-1]
>
>
> Key: HBASE-21687
> URL: https://issues.apache.org/jira/browse/HBASE-21687
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>
> Attachments: HBASE-21687-branch-1.v0.patch
>
>
> Trying to build any branch-1 using Maven 3.6.0 fails:
> {code}
> [ERROR] Failed to execute goal
> org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs (default) on project
> hbase: Unable to parse configuration of mojo
> org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs for parameter
> pluginArtifacts: Cannot assign configuration entry 'pluginArtifacts' with
> value '${plugin.artifacts}' of type
> java.util.Collections.UnmodifiableRandomAccessList to property of type
> java.util.ArrayList -> [Help 1]
> {code}
> also shows up on nightly in the "check that we can make a source artifact"
> test, because "Maven (latest)" recently updated to maven 3.6.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement
[
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737584#comment-16737584
]
Hadoop QA commented on HBASE-20662:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} hbase-client: The patch generated 0 new + 5
unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} hbase-server: The patch generated 0 new + 149
unchanged - 4 fixed = 149 total (was 153) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
13s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 53s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
43s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}139m
29s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20662 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954208/HBASE-20662.master.005.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 10d42a870c99 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21601) corrupted WAL is not handled in all places (NegativeArraySizeException)
[
https://issues.apache.org/jira/browse/HBASE-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737564#comment-16737564
]
Sergey Shelukhin commented on HBASE-21601:
--
Looks like we might need to look closer at the file... l cannot tell from
KeyValueUtil/CellUtil/KeyValue/etc code where exactly the cell is created, but
it seems like the requisite number of bytes should always be read for the
record, assuming we don't get an IOException or EOF of some sort... or the
lower level, byte-reading logic would throw an error.
So, we may be reading the record fully from the file, but getting some garbage
bytes; or there's a bug somewhere that allows a partial read to happen, so the
offset calculations in KeyValue/CellUtil return bogus offsets.
> corrupted WAL is not handled in all places (NegativeArraySizeException)
> ---
>
> Key: HBASE-21601
> URL: https://issues.apache.org/jira/browse/HBASE-21601
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> 2018-12-13 17:01:12,208 ERROR [RS_LOG_REPLAY_OPS-regionserver/...]
> executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.RuntimeException: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hbase.wal.WALSplitter$PipelineController.checkForErrors(WALSplitter.java:846)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$OutputSink.finishWriting(WALSplitter.java:1203)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.finishWritingAndClose(WALSplitter.java:1267)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:349)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:178)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:90)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:113)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.filterCellByStore(WALSplitter.java:1542)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1586)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1560)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1085)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1077)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1047)
> {noformat}
> Unfortunately I cannot share the file.
> The issue appears to be straightforward - for whatever reason the family
> length is negative. Not sure how such a cell got created, I suspect the file
> was corrupted.
> {code}
> byte[] output = new byte[cell.getFamilyLength()];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21601) corrupted WAL is not handled in all places (NegativeArraySizeException)
[
https://issues.apache.org/jira/browse/HBASE-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737564#comment-16737564
]
Sergey Shelukhin edited comment on HBASE-21601 at 1/8/19 9:57 PM:
--
Looks like we might need to look closer at a problematic file... l cannot tell
from KeyValueUtil/CellUtil/KeyValue/etc code where exactly the cell is created,
but it seems like the requisite number of bytes should always be read for the
record, assuming we don't get an IOException or EOF of some sort... or the
lower level, byte-reading logic would throw an error.
So, we may be reading the record fully from the file, but getting some garbage
bytes; or there's a bug somewhere that allows a partial read to happen, so the
offset calculations in KeyValue/CellUtil return bogus offsets.
was (Author: sershe):
Looks like we might need to look closer at the file... l cannot tell from
KeyValueUtil/CellUtil/KeyValue/etc code where exactly the cell is created, but
it seems like the requisite number of bytes should always be read for the
record, assuming we don't get an IOException or EOF of some sort... or the
lower level, byte-reading logic would throw an error.
So, we may be reading the record fully from the file, but getting some garbage
bytes; or there's a bug somewhere that allows a partial read to happen, so the
offset calculations in KeyValue/CellUtil return bogus offsets.
> corrupted WAL is not handled in all places (NegativeArraySizeException)
> ---
>
> Key: HBASE-21601
> URL: https://issues.apache.org/jira/browse/HBASE-21601
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> 2018-12-13 17:01:12,208 ERROR [RS_LOG_REPLAY_OPS-regionserver/...]
> executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.RuntimeException: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hbase.wal.WALSplitter$PipelineController.checkForErrors(WALSplitter.java:846)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$OutputSink.finishWriting(WALSplitter.java:1203)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.finishWritingAndClose(WALSplitter.java:1267)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:349)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:178)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:90)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:113)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.filterCellByStore(WALSplitter.java:1542)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1586)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1560)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1085)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1077)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1047)
> {noformat}
> Unfortunately I cannot share the file.
> The issue appears to be straightforward - for whatever reason the family
> length is negative. Not sure how such a cell got created, I suspect the file
> was corrupted.
> {code}
> byte[] output = new byte[cell.getFamilyLength()];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21601) corrupted WAL is not handled in all places (NegativeArraySizeException)
[
https://issues.apache.org/jira/browse/HBASE-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737549#comment-16737549
]
Sergey Shelukhin commented on HBASE-21601:
--
As far as I see, skipErrors is only applied to IOException-s from certain
places, so these particular errors would not actually be caught by it.
> corrupted WAL is not handled in all places (NegativeArraySizeException)
> ---
>
> Key: HBASE-21601
> URL: https://issues.apache.org/jira/browse/HBASE-21601
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> 2018-12-13 17:01:12,208 ERROR [RS_LOG_REPLAY_OPS-regionserver/...]
> executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.RuntimeException: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hbase.wal.WALSplitter$PipelineController.checkForErrors(WALSplitter.java:846)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$OutputSink.finishWriting(WALSplitter.java:1203)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.finishWritingAndClose(WALSplitter.java:1267)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:349)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:178)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:90)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:113)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.filterCellByStore(WALSplitter.java:1542)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1586)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1560)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1085)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1077)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1047)
> {noformat}
> Unfortunately I cannot share the file.
> The issue appears to be straightforward - for whatever reason the family
> length is negative. Not sure how such a cell got created, I suspect the file
> was corrupted.
> {code}
> byte[] output = new byte[cell.getFamilyLength()];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21626) log the regions blocking WAL from being archived
[ https://issues.apache.org/jira/browse/HBASE-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-21626: - Attachment: (was: HBASE-21626.02.patch) > log the regions blocking WAL from being archived > > > Key: HBASE-21626 > URL: https://issues.apache.org/jira/browse/HBASE-21626 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21626.01.patch, HBASE-21626.patch > > > The WALs not being archived for a long time can result in a long recovery > later. It's useful to know what regions are blocking the WALs from being > archived, to be able to debug flush logic and tune configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21626) log the regions blocking WAL from being archived
[ https://issues.apache.org/jira/browse/HBASE-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-21626: - Attachment: HBASE-21626.02.patch > log the regions blocking WAL from being archived > > > Key: HBASE-21626 > URL: https://issues.apache.org/jira/browse/HBASE-21626 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21626.01.patch, HBASE-21626.02.patch, > HBASE-21626.patch > > > The WALs not being archived for a long time can result in a long recovery > later. It's useful to know what regions are blocking the WALs from being > archived, to be able to debug flush logic and tune configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21687) Update Findbugs Maven Plugin to 3.0.4 to work with Maven 3.6.0+ [branches-1]
[
https://issues.apache.org/jira/browse/HBASE-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737543#comment-16737543
]
Hudson commented on HBASE-21687:
Results for branch branch-1.2
[build #616 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/616/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/616//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/616//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/616//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Update Findbugs Maven Plugin to 3.0.4 to work with Maven 3.6.0+ [branches-1]
>
>
> Key: HBASE-21687
> URL: https://issues.apache.org/jira/browse/HBASE-21687
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>
> Attachments: HBASE-21687-branch-1.v0.patch
>
>
> Trying to build any branch-1 using Maven 3.6.0 fails:
> {code}
> [ERROR] Failed to execute goal
> org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs (default) on project
> hbase: Unable to parse configuration of mojo
> org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs for parameter
> pluginArtifacts: Cannot assign configuration entry 'pluginArtifacts' with
> value '${plugin.artifacts}' of type
> java.util.Collections.UnmodifiableRandomAccessList to property of type
> java.util.ArrayList -> [Help 1]
> {code}
> also shows up on nightly in the "check that we can make a source artifact"
> test, because "Maven (latest)" recently updated to maven 3.6.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21626) log the regions blocking WAL from being archived
[ https://issues.apache.org/jira/browse/HBASE-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737544#comment-16737544 ] Sergey Shelukhin commented on HBASE-21626: -- [~stack] changed the config name. The sequence ID accounting is already effectively aware of regions, given that this method iterates thru numbers by region... I changed the parameter name to be more generic like the rest of the class, but this patch basically just returns more details about why the method returned false, without altering the logic. > log the regions blocking WAL from being archived > > > Key: HBASE-21626 > URL: https://issues.apache.org/jira/browse/HBASE-21626 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21626.01.patch, HBASE-21626.02.patch, > HBASE-21626.patch > > > The WALs not being archived for a long time can result in a long recovery > later. It's useful to know what regions are blocking the WALs from being > archived, to be able to debug flush logic and tune configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21626) log the regions blocking WAL from being archived
[ https://issues.apache.org/jira/browse/HBASE-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-21626: - Attachment: HBASE-21626.02.patch > log the regions blocking WAL from being archived > > > Key: HBASE-21626 > URL: https://issues.apache.org/jira/browse/HBASE-21626 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21626.01.patch, HBASE-21626.02.patch, > HBASE-21626.patch > > > The WALs not being archived for a long time can result in a long recovery > later. It's useful to know what regions are blocking the WALs from being > archived, to be able to debug flush logic and tune configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21696) Upgrade findbugs maven plugin to 3.0.4
[
https://issues.apache.org/jira/browse/HBASE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737523#comment-16737523
]
Hadoop QA commented on HBASE-21696:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} branch-1 passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
47s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} branch-1 passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
43s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 0s{color} |
{color:red} The patch has 1 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 33s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| | pom.xml |
| Failed junit tests | hadoop.hbase.mapred.TestTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21696 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954207/HBASE-21696-branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit shadedjars
hadoopcheck xml compile |
| uname | Linux 28e56aa60ef0 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-1 / f9afd92 |
| maven | version: Apache Maven 3.0.5 |
|
[jira] [Commented] (HBASE-21645) Perform sanity check and disallow table creation/modification with region replication < 1
[
https://issues.apache.org/jira/browse/HBASE-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737525#comment-16737525
]
Hadoop QA commented on HBASE-21645:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}145m
36s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21645 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954204/HBASE-21645.master.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux abcf97462886 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7d6ce3569d |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15500/testReport/ |
| Max. process+thread count | 5103 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15500/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Perform
[jira] [Commented] (HBASE-21296) [2.1] Upgrade Jetty dependencies to latest in major-line
[ https://issues.apache.org/jira/browse/HBASE-21296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737516#comment-16737516 ] Josh Elser commented on HBASE-21296: [~stack], I think it's still a good idea to bring this in for the "dependency hygiene" aspect of it. > [2.1] Upgrade Jetty dependencies to latest in major-line > > > Key: HBASE-21296 > URL: https://issues.apache.org/jira/browse/HBASE-21296 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.1.3 > > Attachments: HBASE-21282.001.branch-2.0.patch > > > Looks like we have dependencies on both jetty 9.2 and 9.3, but we're lagging > pretty far behind in both. We can upgrade both of these to the latest (august > 2018). > > I'll also have to take a look at why we're using two separate versions (maybe > we didn't want to switch from jetty-jsp to apache-jsp on 9.2->9.3?). Not sure > if there's a good reason for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21672) Allow skipping HDFS block distribution computation
[
https://issues.apache.org/jira/browse/HBASE-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737513#comment-16737513
]
Sean Busbey commented on HBASE-21672:
-
Here's my concern: as an operator why should I have to know this detail of the
implementation? this is effectively a hidden "magically go faster" button. why
can't this be something that we take care of for the operator? Either by
whitelisting FileSystems that should skip it or pushing the providers of those
FileSystems to implement something that tells us as a downstream user that
there isn't going to be locality? Or doing a start up check that tells us there
isn't going to be locality (e.g. for the case where we are talking to HDFS but
that HDFS is a distinct set of nodes from our HBase nodes)?
> Allow skipping HDFS block distribution computation
> --
>
> Key: HBASE-21672
> URL: https://issues.apache.org/jira/browse/HBASE-21672
> Project: HBase
> Issue Type: Improvement
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Labels: S3
>
> We should have a configuration to skip HDFS block distribution calculation in
> HBase. For example on file systems that do not surface locality such as S3,
> calculating block distribution would not be any useful.
> Currentlly, we do not have a way to skip hdfs block distribution computation.
> For this, we can provide a new configuration key, say
> {{hbase.block.distribution.skip.computation}} (which would be {{false}} by
> default).
> Users using filesystems such as s3 may choose to make this {{true}}, thus
> skipping block distribution computation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21685) Change repository urls to Gitbox
[
https://issues.apache.org/jira/browse/HBASE-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737512#comment-16737512
]
Hadoop QA commented on HBASE-21685:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
47s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
43s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 39s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}205m
48s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}295m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21685 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954185/HBASE-21685.master.001.patch
|
| Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc
unit shadedjars hadoopcheck xml compile refguide mvnsite |
| uname | Linux c22df3d3ddaf 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21577) do not close regions when RS is dying due to a broken WAL
[ https://issues.apache.org/jira/browse/HBASE-21577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737472#comment-16737472 ] Sergey Shelukhin commented on HBASE-21577: -- The issue happens mostly with other regions, DroppedSnapshot almost always indicates some filesystem issue. As other regions are closing, they go thru the retries in various places. [~Apache9] does the patch look good to you after update? > do not close regions when RS is dying due to a broken WAL > - > > Key: HBASE-21577 > URL: https://issues.apache.org/jira/browse/HBASE-21577 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21577.master.001.patch, > HBASE-21577.master.002.patch > > > See HBASE-21576. DroppedSnapshot can be an FS failure; also, when WAL is > broken, some regions whose flushes are already in flight keep retrying, > resulting in minutes-long shutdown times. Since WAL will be replayed anyway > flushing regions doesn't provide much benefit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21456) Make WALFactory only used for creating WALProviders
[ https://issues.apache.org/jira/browse/HBASE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737437#comment-16737437 ] Josh Elser commented on HBASE-21456: Glanced at the WIP patch briefly: only thing that jumped out at me so far – what if we rename WALFactory to WALProviderFactory. I'm not usually one for renaming quickly, but I think that will make things more semantically correct :) > Make WALFactory only used for creating WALProviders > --- > > Key: HBASE-21456 > URL: https://issues.apache.org/jira/browse/HBASE-21456 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Josh Elser >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-21456.HBASE-20952.wip.patch > > > As a Factory, WALFactory should only have one job: creating instances of > WALProvider. > However, over the years, it has been a landing place for lots of wal-related > methods (e.g. constructing readers, WALEntryStream, and more). We want all of > this to live in the WALProvider. > We can do this in two steps: have the WALFactory methods invoke the method on > the WALProvider (handled elsewhere), then here we can replace usage of the > WALFactory "wrapper methods" with the WALProvider itself. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement
[
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20662:
---
Attachment: HBASE-20662.master.005.patch
> Increasing space quota on a violated table does not remove
> SpaceViolationPolicy.DISABLE enforcement
> ---
>
> Key: HBASE-20662
> URL: https://issues.apache.org/jira/browse/HBASE-20662
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20662.master.001.patch,
> HBASE-20662.master.002.patch, HBASE-20662.master.003.patch,
> HBASE-20662.master.004.patch, HBASE-20662.master.004.patch,
> HBASE-20662.master.005.patch
>
>
> *Steps to reproduce*
> * Create a table and set quota with {{SpaceViolationPolicy.DISABLE}} having
> limit say 2MB
> * Now put rows until space quota is violated and table gets disabled
> * Next, increase space quota with limit say 4MB on the table
> * Now try putting a row into the table
> {code:java}
> private void testSetQuotaThenViolateAndFinallyIncreaseQuota() throws
> Exception {
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, increase limit
> setQuotaLimit(tn, policy, 4L);
> // Put some row now: should not violate as quota limit increased
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*
> We should be able to put data as long as newly set quota limit is not reached
> *Actual*
> We fail to put any new row even after increasing limit
> *Root cause*
> Increasing quota on a violated table triggers the table to be enabled, but
> since the table is already in violation, the system does not allow it to be
> enabled (may be thinking that a user is trying to enable it)
> *Relevant exception trace*
> {noformat}
> 2018-05-31 00:34:27,563 INFO [regionserver/root1-ThinkPad-T440p:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenIncreaseQuotaWithDisable0
> 2018-05-31 00:34:27,571 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42525]
> ipc.CallRunner(142): callId: 11 service: MasterService methodName:
> EnableTable size: 104 connection: 127.0.0.1:38030 deadline: 1527707127568,
> exception=org.apache.hadoop.hbase.security.AccessDeniedException: Enabling
> the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to
> a violated space quota.
> 2018-05-31 00:34:27,571 ERROR [regionserver/root1-ThinkPad-T440p:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation
> policy for testSetQuotaAndThenIncreaseQuotaWithDisable0. This table will
> remain in violation.
> org.apache.hadoop.hbase.security.AccessDeniedException:
> org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table
> 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a
> violated space quota.
> at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
> at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
>
