[jira] [Commented] (HBASE-19959) How much RAM space is to be really consumed by the memstore?

[Chance Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358014#comment-16358014
 ] 

Chance Li commented on HBASE-19959:
---

In this scenario, every chunk in pool was used at least once , and the number 
of pooled chunk has reached the upper limit. That means chunk pool have take 
the physical RAM.

> How much RAM space is to be really consumed by the memstore?
> 
>
> Key: HBASE-19959
> URL: https://issues.apache.org/jira/browse/HBASE-19959
> Project: HBase
>  Issue Type: Brainstorming
>  Components: regionserver
>Reporter: Chance Li
>Priority: Minor
>
> Let's consider this scenario where memstoreLAB and ChunkPool is enable and 
> max memstore size is 10G, and after some time all pooled chunk have been 
> created, then flush all data, now memstore size is 0 but RAM actually have 
> consumed 10G, then continue writing big cell which will not use the chunk 
> pool but jvm heap, then memstore size will be increased to 10G(maybe more 
> because overhead). now we can see RAM actually consumed 20G (10G of pooled 
> chunk + 10G java objects), but the max memstore size is only 10G.
> what I say is the max memstore size not only take care about the cell "size" 
> but also RAM really used. This will be a strict memory management: the max 
> memstore size limit the RAM space which the memstore or related module can be 
> used.
> This really rarely occured. It's just for a robust memory managemant 
> semantically. 
>  What do you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19957) General framework to transit sync replication state

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358008#comment-16358008
 ] 

Duo Zhang commented on HBASE-19957:
---

Review board link:

https://reviews.apache.org/r/65582/

> General framework to transit sync replication state
> ---
>
> Key: HBASE-19957
> URL: https://issues.apache.org/jira/browse/HBASE-19957
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-19957-HBASE-19064.patch, HBASE-19957.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-19937:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0-beta-2
   Status: Resolved  (was: Patch Available)

Thanks for the patch, Xiaolin

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, 
> HBASE-19937.branch-2.006.patch, import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> 

[jira] [Updated] (HBASE-19957) General framework to transit sync replication state

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-19957:
--
Status: Patch Available  (was: Open)

> General framework to transit sync replication state
> ---
>
> Key: HBASE-19957
> URL: https://issues.apache.org/jira/browse/HBASE-19957
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-19957-HBASE-19064.patch, HBASE-19957.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19957) General framework to transit sync replication state

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358004#comment-16358004
 ] 

Duo Zhang commented on HBASE-19957:
---

Let's get this in so that we can start working on the follow-on issues.

> General framework to transit sync replication state
> ---
>
> Key: HBASE-19957
> URL: https://issues.apache.org/jira/browse/HBASE-19957
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-19957-HBASE-19064.patch, HBASE-19957.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19957) General framework to transit sync replication state

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-19957:
--
Attachment: (was: HBASE-19957-v1.patch)

> General framework to transit sync replication state
> ---
>
> Key: HBASE-19957
> URL: https://issues.apache.org/jira/browse/HBASE-19957
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-19957-HBASE-19064.patch, HBASE-19957.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19957) General framework to transit sync replication state

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-19957:
--
Attachment: HBASE-19957-HBASE-19064.patch

> General framework to transit sync replication state
> ---
>
> Key: HBASE-19957
> URL: https://issues.apache.org/jira/browse/HBASE-19957
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-19957-HBASE-19064.patch, HBASE-19957.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19957) General framework to transit sync replication state

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-19957:
--
Attachment: HBASE-19957-v1.patch

> General framework to transit sync replication state
> ---
>
> Key: HBASE-19957
> URL: https://issues.apache.org/jira/browse/HBASE-19957
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-19957-HBASE-19064.patch, HBASE-19957.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357996#comment-16357996
 ] 

Duo Zhang commented on HBASE-19965:
---

I‘m not quite familiar with the AMv2 yet...

> Fix flaky TestAsyncRegionAdminApi
> -
>
> Key: HBASE-19965
> URL: https://issues.apache.org/jira/browse/HBASE-19965
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Priority: Major
>
> See 
> [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/]
>  
> java.lang.AssertionError: expected:<2> but was:<3> at 
> org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359)
>  
> Merge regions not work. The table still have 3 regions after the 
> MergeRegionsProcedure finished.
> The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because 
> the MergeRegionsProcedure pid=138 start work first, so the balance need wait 
> for the lock. But after merge regions finished, the MoveRegionProcedure 
> pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new 
> region server. This is not right. The MoveRegionProcedure should skip to 
> assign a region which was marked as offline. Or we should clear the merged 
> regions' procedure when MergeRegionsProcedure finished.
>  
> Logs:
> 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] 
> master.HMaster(1454): balance 
> hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
> source=cd4730e3eae2,39077,1518106776411, 
> destination=cd4730e3eae2,40578,1518106776318
> 2018-02-08 16:24:44,608 DEBUG 
> [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] 
> procedure2.ProcedureExecutor(868): Stored pid=138, 
> state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure 
> table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 
> 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false
> ..
> 2018-02-08 16:24:50,111 INFO [PEWorker-13] 
> procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; 
> MergeTableRegionsProcedure table=testMergeRegions, 
> regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], 
> forcibly=false in 5.5710sec
> 2018-02-08 16:24:50,113 INFO [PEWorker-13] 
> procedure.MasterProcedureScheduler(813): pid=139, 
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure 
> hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
> source=cd4730e3eae2,39077,1518106776411, 
> destination=cd4730e3eae2,40578,1518106776318 testMergeRegions 
> testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357982#comment-16357982
 ] 

Hadoop QA commented on HBASE-19937:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
43s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 1s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
14m 22s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
11s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 8s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19937 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909890/HBASE-19937.branch-2.006.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 65e6fcade050 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | branch-2 / 4461cb7d7a |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11463/testReport/ |
| Max. process+thread count | 2700 (vs. ulimit of 1) |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11463/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> 

[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357980#comment-16357980
 ] 

Guanghao Zhang commented on HBASE-19965:


[~stack] [~Apache9] Any ideas about how to fix this?

> Fix flaky TestAsyncRegionAdminApi
> -
>
> Key: HBASE-19965
> URL: https://issues.apache.org/jira/browse/HBASE-19965
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Priority: Major
>
> See 
> [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/]
>  
> java.lang.AssertionError: expected:<2> but was:<3> at 
> org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359)
>  
> Merge regions not work. The table still have 3 regions after the 
> MergeRegionsProcedure finished.
> The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because 
> the MergeRegionsProcedure pid=138 start work first, so the balance need wait 
> for the lock. But after merge regions finished, the MoveRegionProcedure 
> pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new 
> region server. This is not right. The MoveRegionProcedure should skip to 
> assign a region which was marked as offline. Or we should clear the merged 
> regions' procedure when MergeRegionsProcedure finished.
>  
> Logs:
> 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] 
> master.HMaster(1454): balance 
> hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
> source=cd4730e3eae2,39077,1518106776411, 
> destination=cd4730e3eae2,40578,1518106776318
> 2018-02-08 16:24:44,608 DEBUG 
> [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] 
> procedure2.ProcedureExecutor(868): Stored pid=138, 
> state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure 
> table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 
> 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false
> ..
> 2018-02-08 16:24:50,111 INFO [PEWorker-13] 
> procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; 
> MergeTableRegionsProcedure table=testMergeRegions, 
> regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], 
> forcibly=false in 5.5710sec
> 2018-02-08 16:24:50,113 INFO [PEWorker-13] 
> procedure.MasterProcedureScheduler(813): pid=139, 
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure 
> hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
> source=cd4730e3eae2,39077,1518106776411, 
> destination=cd4730e3eae2,40578,1518106776318 testMergeRegions 
> testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-19965:
---
Description: 
See 
[https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/]

 

java.lang.AssertionError: expected:<2> but was:<3> at 
org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359)

 

Merge regions not work. The table still have 3 regions after the 
MergeRegionsProcedure finished.

The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because 
the MergeRegionsProcedure pid=138 start work first, so the balance need wait 
for the lock. But after merge regions finished, the MoveRegionProcedure pid=139 
start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new region server. 
This is not right. The MoveRegionProcedure should skip to assign a region which 
was marked as offline. Or we should clear the merged regions' procedure when 
MergeRegionsProcedure finished.

 

Logs:

2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] 
master.HMaster(1454): balance 
hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
source=cd4730e3eae2,39077,1518106776411, 
destination=cd4730e3eae2,40578,1518106776318

2018-02-08 16:24:44,608 DEBUG 
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] 
procedure2.ProcedureExecutor(868): Stored pid=138, 
state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure 
table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 
8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false

..

2018-02-08 16:24:50,111 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): 
Finished pid=138, state=SUCCESS; MergeTableRegionsProcedure 
table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 
8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false in 5.5710sec

2018-02-08 16:24:50,113 INFO [PEWorker-13] 
procedure.MasterProcedureScheduler(813): pid=139, 
state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure 
hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
source=cd4730e3eae2,39077,1518106776411, 
destination=cd4730e3eae2,40578,1518106776318 testMergeRegions 
testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4.

 

  was:
See 
[https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/]

 

java.lang.AssertionError: expected:<2> but was:<3> at 
org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359)

 

Merge regions not work. The table still have 3 regions after the 
MergeRegionsProcedure finished.

The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because 
the MergeRegionsProcedure pid=138 start work first, so the balance need wait 
for the lock. But after merge regions finished, the MoveRegionProcedure pid=139 
start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new region server. 
This is not right. The MoveRegionProcedure should skip to assign a region which 
was marked as offline. Or we should clear the merged regions' procedure when 
MergeRegionsProcedure finished.

 

Logs:

2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] 
master.HMaster(1454): balance 
hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
source=cd4730e3eae2,39077,1518106776411, 
destination=cd4730e3eae2,40578,1518106776318 2018-02-08 16:24:44,608 DEBUG 
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] 
procedure2.ProcedureExecutor(868): Stored pid=138, 
state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE;

MergeTableRegionsProcedure table=testMergeRegions, 
regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], 
forcibly=false

..

2018-02-08 16:24:50,111 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): 
Finished pid=138, state=SUCCESS; MergeTableRegionsProcedure 
table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 
8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false in 5.5710sec 2018-02-08 
16:24:50,113 INFO [PEWorker-13] procedure.MasterProcedureScheduler(813): 
pid=139, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure 
hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
source=cd4730e3eae2,39077,1518106776411, 
destination=cd4730e3eae2,40578,1518106776318 testMergeRegions 
testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4.

 


> Fix flaky TestAsyncRegionAdminApi
> -
>
> Key: HBASE-19965
> URL: https://issues.apache.org/jira/browse/HBASE-19965
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Priority: Major
>
> See 
> 

[jira] [Created] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi

[Guanghao Zhang (JIRA)]

Guanghao Zhang created HBASE-19965:
--

 Summary: Fix flaky TestAsyncRegionAdminApi
 Key: HBASE-19965
 URL: https://issues.apache.org/jira/browse/HBASE-19965
 Project: HBase
  Issue Type: Sub-task
Reporter: Guanghao Zhang


See 
[https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/]

 

java.lang.AssertionError: expected:<2> but was:<3> at 
org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359)

 

Merge regions not work. The table still have 3 regions after the 
MergeRegionsProcedure finished.

The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because 
the MergeRegionsProcedure pid=138 start work first, so the balance need wait 
for the lock. But after merge regions finished, the MoveRegionProcedure pid=139 
start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new region server. 
This is not right. The MoveRegionProcedure should skip to assign a region which 
was marked as offline. Or we should clear the merged regions' procedure when 
MergeRegionsProcedure finished.

 

Logs:

2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] 
master.HMaster(1454): balance 
hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
source=cd4730e3eae2,39077,1518106776411, 
destination=cd4730e3eae2,40578,1518106776318 2018-02-08 16:24:44,608 DEBUG 
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] 
procedure2.ProcedureExecutor(868): Stored pid=138, 
state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE;

MergeTableRegionsProcedure table=testMergeRegions, 
regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], 
forcibly=false

..

2018-02-08 16:24:50,111 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): 
Finished pid=138, state=SUCCESS; MergeTableRegionsProcedure 
table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 
8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false in 5.5710sec 2018-02-08 
16:24:50,113 INFO [PEWorker-13] procedure.MasterProcedureScheduler(813): 
pid=139, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure 
hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., 
source=cd4730e3eae2,39077,1518106776411, 
destination=cd4730e3eae2,40578,1518106776318 testMergeRegions 
testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter

[Reid Chan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357975#comment-16357975
 ] 

Reid Chan commented on HBASE-19950:
---

WDYT, [~anoop.hbase]

> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
>  Issue Type: Improvement
>  Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch, 
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when 
> specified condition is matched, this new filter will return the matched cell 
> only. There're already some discussions in HBASE-19824.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357957#comment-16357957
 ] 

Xiaolin Ha commented on HBASE-19937:


Changed orders of imports.

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, 
> HBASE-19937.branch-2.006.patch, import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Attachment: HBASE-19937.branch-2.006.patch

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, 
> HBASE-19937.branch-2.006.patch, import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> 

[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357955#comment-16357955
 ] 

stack commented on HBASE-16060:
---

So, it is interesting. Finding a few issues in here like:

+ If a cluster is restarted, and table is disabled, we don't do right thing 
(comes of our recent change where we mark new Regions as CLOSED). Fixing.
+ Also, need to figure w/ this bit of code

// migrating existent table state from zk, so splitters
// and recovery process treat states properly.
// TODO: There is a problem here. There is no guarantee tht 
tableStateManager#start has
// been called but we are using it below anyways (needs hbase:meta online 
which may not be
// the case yet). Also,
for (Map.Entry entry : ZKDataMigrator
.queryForTableStates(getZooKeeper()).entrySet()) {
  LOG.info("Converting state from zk to new states:" + entry);
  tableStateManager.setTableState(entry.getKey(), entry.getValue());
}

Its a migration from zk to hbase:meta of table state but how to tell when 
migration is done?

Will be back.

> 1.x clients cannot access table state talking to 2.0 cluster
> 
>
> Key: HBASE-16060
> URL: https://issues.apache.org/jira/browse/HBASE-16060
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0-beta-2
>
> Attachments: 
> 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, 
> HBASE-16060.branch-2.001.patch
>
>
> Since table state is migrated to meta instead of zk in 2.0, 1.x clients 
> talking to 2.0 cluster cannot access the table state. This causes some weird 
> behavior since from a client perspective, {{Admin.isTableEnabled()}} and 
> {{Admin.isTableDisabled()}} both return false. 
> One option we can do is to add code in 1.x clients so that they can access 
> the table state in meta if needed. Otherwise, we can mirror the table state 
> in zk (while keeping meta as the source of truth) during 2.x lifecycle so 
> that any 1.x client can still work correctly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357947#comment-16357947
 ] 

Guanghao Zhang commented on HBASE-19942:


Pushed addendum patch to master and branch-2.

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.branch-2.addendum.patch, HBASE-19942.master.001.patch, 
> HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357940#comment-16357940
 ] 

Hadoop QA commented on HBASE-19942:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  3m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 2s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 0s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
14m 33s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}121m 
20s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19942 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909874/HBASE-19942.branch-2.addendum.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 6e3f487b1289 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | branch-2 / bc080e7500 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11460/testReport/ |
| Max. process+thread count | 4514 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11460/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: 

[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357934#comment-16357934
 ] 

stack commented on HBASE-19942:
---

+1 Go for it. Thanks for fixing this.

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.branch-2.addendum.patch, HBASE-19942.master.001.patch, 
> HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357899#comment-16357899
 ] 

Ted Yu edited comment on HBASE-19937 at 2/9/18 5:16 AM:


The warning is about java.lang.Thread.sleep
You can drop the import and call sleep as other tests do.


was (Author: yuzhih...@gmail.com):
Duo knows what the order should be.

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, 
> import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> 

[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357919#comment-16357919
 ] 

Ted Yu commented on HBASE-19930:


It would be desirable to run IntegrationTestIngestWithMOB with the patch to see 
if it passes.

> fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
> ---
>
> Key: HBASE-19930
> URL: https://issues.apache.org/jira/browse/HBASE-19930
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0-beta-1
>Reporter: Gali Sheffi
>Assignee: Gali Sheffi
>Priority: Major
> Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, 
> HBASE-19930-V03.patch, HBASE-19930-V04.patch
>
>
> This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto.
> Following a comment in HBASE-19133 regarding a bug in 
> ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never 
> called for an ImmutableMemStoreLAB, and just throwing an 
> IllegalStateException whenever called), the forceCopyOfBigCellInto method now 
> performs the copy of big cells on the first MSLABImpl in its mslabs 
> linked-list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357912#comment-16357912
 ] 

Hadoop QA commented on HBASE-19942:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
56s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
46s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m 26s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m 
11s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 29s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19942 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909867/HBASE-19942.master.addendum.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 1cd847dccc98 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / f1a81618fd |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11458/testReport/ |
| Max. process+thread count | 4718 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11458/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: 

[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357908#comment-16357908
 ] 

Anoop Sam John commented on HBASE-19930:


Oh if that is the case yes.  But we need some fat comments for this. Why we do 
..  This is an Immutable MSLAB but we allow the force copy ti happen. And the 
normal copy op throw Unsupported exception..  Pls add enough comments.  Also 
the copy is to 0th offset MSLAB.. this looks fine as to which the copy happens, 
does not matter.  Pls add enough comments. 

> fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
> ---
>
> Key: HBASE-19930
> URL: https://issues.apache.org/jira/browse/HBASE-19930
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0-beta-1
>Reporter: Gali Sheffi
>Assignee: Gali Sheffi
>Priority: Major
> Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, 
> HBASE-19930-V03.patch, HBASE-19930-V04.patch
>
>
> This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto.
> Following a comment in HBASE-19133 regarding a bug in 
> ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never 
> called for an ImmutableMemStoreLAB, and just throwing an 
> IllegalStateException whenever called), the forceCopyOfBigCellInto method now 
> performs the copy of big cells on the first MSLABImpl in its mslabs 
> linked-list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357899#comment-16357899
 ] 

Ted Yu commented on HBASE-19937:


Duo knows what the order should be.

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, 
> import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> 

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Attachment: import-order.png

Can you tell me the right order of imports? My order now is as follows:

​




> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, 
> import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357888#comment-16357888
 ] 

Ted Yu commented on HBASE-19937:


Can you address checkstyle warning ?

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> 

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357879#comment-16357879
 ] 

Hadoop QA commented on HBASE-19937:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
1s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
40s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} hbase-rsgroup: The patch generated 1 new + 0 unchanged 
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
30s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
12m 55s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
13s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 9s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 55s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19937 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909877/HBASE-19937.branch-2.005.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 3e47709372b2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | branch-2 / bc080e7500 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11461/artifact/patchprocess/diff-checkstyle-hbase-rsgroup.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11461/testReport/ |
| Max. process+thread count | 2783 (vs. ulimit of 1) |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11461/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357871#comment-16357871
 ] 

Hadoop QA commented on HBASE-19937:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
41s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} hbase-rsgroup: The patch generated 1 new + 0 unchanged 
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
28s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
12m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
16s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 7s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19937 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909876/HBASE-19937.branch-2.004.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux b37626f9f86c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | branch-2 / bc080e7500 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11459/artifact/patchprocess/diff-checkstyle-hbase-rsgroup.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11459/testReport/ |
| Max. process+thread count | 2864 (vs. ulimit of 1) |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11459/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This 

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Attachment: HBASE-19937.branch-2.005.patch

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> 

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357858#comment-16357858
 ] 

Xiaolin Ha commented on HBASE-19937:


I have combined the two patches to one, and made some changes in the test case.

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> 

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Attachment: HBASE-19937.branch-2.004.patch

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, 
> HBASE-19937.branch-2.004.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup 

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Attachment: HBASE-19937.branch-2.003.patch

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, 

[jira] [Updated] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-19942:
---
Attachment: HBASE-19942.branch-2.addendum.patch

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.branch-2.addendum.patch, HBASE-19942.master.001.patch, 
> HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-19964) TestWriteHeavyIncrementObserver fails

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-19964:
-

 Summary: TestWriteHeavyIncrementObserver fails
 Key: HBASE-19964
 URL: https://issues.apache.org/jira/browse/HBASE-19964
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Reporter: Duo Zhang
 Fix For: 2.0.0-beta-2


https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/

I could reproduce it locally. Let me dig more.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15381) Implement a distributed MOB compaction by procedure

[Jingcheng Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357845#comment-16357845
 ] 

Jingcheng Du commented on HBASE-15381:
--

Please go ahead [~huaxiang], thanks a lot.

> Implement a distributed MOB compaction by procedure
> ---
>
> Key: HBASE-15381
> URL: https://issues.apache.org/jira/browse/HBASE-15381
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
>Priority: Major
> Attachments: HBASE-15381-v2.patch, HBASE-15381-v3.patch, 
> HBASE-15381-v4.patch, HBASE-15381-v5.patch, HBASE-15381-v6.patch, 
> HBASE-15381.patch, mob distributed compaction design-v2.pdf, mob distributed 
> compaction design.pdf
>
>
> In MOB, there is a periodical compaction which runs in HMaster (It can be 
> disabled by configuration), some small mob files are merged into bigger ones. 
> Now the compaction only runs in HMaster which is not efficient and might 
> impact the running of HMaster. In this JIRA, a distributed MOB compaction is 
> introduced, it is triggered by HMaster, but all the compaction jobs are 
> distributed to HRegionServers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-19942:
---
Status: Patch Available  (was: Reopened)

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch, HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-19942:
---
Attachment: HBASE-19923.master.003.patch

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch, HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-19942:
---
Attachment: (was: HBASE-19923.master.003.patch)

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch, HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang reopened HBASE-19942:


> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch, HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-19942:
---
Attachment: HBASE-19942.master.addendum.patch

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch, HBASE-19942.master.addendum.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357813#comment-16357813
 ] 

Guanghao Zhang commented on HBASE-19942:


Not a same problem, let me dig more.

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-19920:
--
Attachment: HBASE-19920.v8.patch

> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, 
> HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, 
> HBASE-19920.v6.patch, HBASE-19920.v7.patch, HBASE-19920.v8.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil 
> which in its static block initializes DynamicClassLoader and that creates the 
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class 
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when 
> one accesses hbase as a client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357768#comment-16357768
 ] 

Hadoop QA commented on HBASE-19920:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
19s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  
5s{color} | {color:red} hbase-server: The patch generated 1 new + 5 unchanged - 
0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
43s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 37s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
1s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 97m 
59s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19920 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909843/HBASE-19920.v7.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 06a6f053dfcd 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / f1a81618fd |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 

[jira] [Commented] (HBASE-19835) Make explicit casting of atleast one operand to final type

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357752#comment-16357752
 ] 

Hudson commented on HBASE-19835:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4551 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4551/])
HBASE-19835 Use explicit casting to avoid side effects (tedyu: rev 
f1a81618fdd6318df5edded64fbb07e085e10853)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
* (edit) 
hbase-common/src/main/java/org/apache/hadoop/hbase/IndividualBytesFieldCell.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/PrivateCellUtil.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanInfo.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java


> Make explicit casting of atleast one operand to final type
> --
>
> Key: HBASE-19835
> URL: https://issues.apache.org/jira/browse/HBASE-19835
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Minor
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19835.master.01.patch, HBASE-19835.master.02.patch
>
>
> We have used 
> _long = int + int_ 
> at many places mostly wherever ClassSize.java variables are used for 
> calculation.
> Need to cast explicitly at-least one operand to final type(i.e. type the 
> result is intended to be casted).
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-19920:
--
Attachment: HBASE-19920.v7.patch

> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, 
> HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, 
> HBASE-19920.v6.patch, HBASE-19920.v7.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil 
> which in its static block initializes DynamicClassLoader and that creates the 
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class 
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when 
> one accesses hbase as a client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357650#comment-16357650
 ] 

Mike Drob commented on HBASE-19920:
---

Ugh. v7, try again with checkstyle.

> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, 
> HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, 
> HBASE-19920.v6.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil 
> which in its static block initializes DynamicClassLoader and that creates the 
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class 
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when 
> one accesses hbase as a client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17825) Backup: further optimizations

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357641#comment-16357641
 ] 

Josh Elser commented on HBASE-17825:


bq. I think it is fine, this is work in progress and it has not been released 
yet to public.

Yeah, not a big deal at this point. We can include a warning as a release note, 
but not a big problem if this is a necessary change as you've outlined.

> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
>  Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17825) Backup: further optimizations

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357637#comment-16357637
 ] 

Vladimir Rodionov commented on HBASE-17825:
---

{quote}
This is just "in-flight" backups, so there's no concern with any existing 
backup data, right?
{quote}

No, this changes directory structure in backup FS. I think it is fine, this is 
work in progress and it has not been released yet to public.

> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
>  Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19391) Calling HRegion#initializeRegionInternals from a region replica can still re-create a region directory

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357604#comment-16357604
 ] 

huaxiang sun commented on HBASE-19391:
--

Going to commit if no objection.

> Calling HRegion#initializeRegionInternals from a region replica can still 
> re-create a region directory
> --
>
> Key: HBASE-19391
> URL: https://issues.apache.org/jira/browse/HBASE-19391
> Project: HBase
>  Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Major
> Attachments: HBASE-19391.master.v0.patch
>
>
> This is a follow up from HBASE-18024. There stills a chance that attempting 
> to open a region that is not the default region replica can still create a 
> GC'd region directory by the CatalogJanitor causing inconsistencies with hbck.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357602#comment-16357602
 ] 

huaxiang sun commented on HBASE-19553:
--

I am going to commit if no objection.

> Old replica regions should be cleared from AM memory after primary region 
> split or merge
> 
>
> Key: HBASE-19553
> URL: https://issues.apache.org/jira/browse/HBASE-19553
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: huaxiang sun
>Assignee: Pankaj Kumar
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-19553-branch-1-v2.patch, 
> HBASE-19553-branch-1-v3.patch, HBASE-19553-branch-1-v4.patch, 
> HBASE-19553-branch-1-v4.patch, HBASE-19553-branch-1.patch
>
>
> Similar to HBASE-18025, the replica parent's info is not removed from master. 
> Actually I think it can be removed after replica region is split or merged, I 
> will check the logic and apply one patch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357593#comment-16357593
 ] 

stack commented on HBASE-16060:
---


.001 adds mirroring of table state to zk.

> 1.x clients cannot access table state talking to 2.0 cluster
> 
>
> Key: HBASE-16060
> URL: https://issues.apache.org/jira/browse/HBASE-16060
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0-beta-2
>
> Attachments: 
> 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, 
> HBASE-16060.branch-2.001.patch
>
>
> Since table state is migrated to meta instead of zk in 2.0, 1.x clients 
> talking to 2.0 cluster cannot access the table state. This causes some weird 
> behavior since from a client perspective, {{Admin.isTableEnabled()}} and 
> {{Admin.isTableDisabled()}} both return false. 
> One option we can do is to add code in 1.x clients so that they can access 
> the table state in meta if needed. Otherwise, we can mirror the table state 
> in zk (while keeping meta as the source of truth) during 2.x lifecycle so 
> that any 1.x client can still work correctly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-16060:
--
Attachment: HBASE-16060.branch-2.001.patch

> 1.x clients cannot access table state talking to 2.0 cluster
> 
>
> Key: HBASE-16060
> URL: https://issues.apache.org/jira/browse/HBASE-16060
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0-beta-2
>
> Attachments: 
> 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, 
> HBASE-16060.branch-2.001.patch
>
>
> Since table state is migrated to meta instead of zk in 2.0, 1.x clients 
> talking to 2.0 cluster cannot access the table state. This causes some weird 
> behavior since from a client perspective, {{Admin.isTableEnabled()}} and 
> {{Admin.isTableDisabled()}} both return false. 
> One option we can do is to add code in 1.x clients so that they can access 
> the table state in meta if needed. Otherwise, we can mirror the table state 
> in zk (while keeping meta as the source of truth) during 2.x lifecycle so 
> that any 1.x client can still work correctly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner

[Chia-Ping Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated HBASE-19876:
---
Status: Open  (was: Patch Available)

Trying to write the test for branch-1.2 and branch-1.3...

> The exception happening in converting pb mutation to hbase.mutation messes up 
> the CellScanner
> -
>
> Key: HBASE-19876
> URL: https://issues.apache.org/jira/browse/HBASE-19876
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19876.master.001.patch, HBASE-19876.v0.patch, 
> HBASE-19876.v1.patch, HBASE-19876.v2.patch, HBASE-19876.v3.patch, 
> HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch, 
> HBASE-19876.v4.patch, HBASE-19876.v5.patch
>
>
> {code:java}
> 2018-01-27 22:51:43,794 INFO  [hconnection-0x3291b443-shared-pool11-t6] 
> client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, 
> attempt=6/16 failed=20ops, last 
> exception=org.apache.hadoop.hbase.client.WrongRowIOException: 
> org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't 
> match the original one aaa
>   at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
>   at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code}
> I noticed this bug when testing the table space quota.
> When rs are converting pb mutation to hbase.mutation, the quota exception or 
> cell exception may be thrown.
> {code}
> Unable to find source-code formatter for language: 
> rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, 
> applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, 
> java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, 
> rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml  for 
> (ClientProtos.Action action: mutations) {
> MutationProto m = action.getMutation();
> Mutation mutation;
> if (m.getMutateType() == MutationType.PUT) {
>   mutation = ProtobufUtil.toPut(m, cells);
>   batchContainsPuts = true;
> } else {
>   mutation = ProtobufUtil.toDelete(m, cells);
>   batchContainsDelete = true;
> }
> mutationActionMap.put(mutation, action);
> mArray[i++] = mutation;
> checkCellSizeLimit(region, mutation);
> // Check if a space quota disallows this mutation
> spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
> quota.addMutation(mutation);
>   }
> {code}
> rs has caught the exception but it doesn't have the cellscanner skip the 
> failed cells.
> {code:java}
> } catch (IOException ie) {
>   if (atomic) {
> throw ie;
>   }
>   for (Action mutation : mutations) {
> builder.addResultOrException(getResultOrException(ie, 
> mutation.getIndex()));
>   }
> }
> {code}
> The bug results in the WrongRowIOException to remaining mutations since they 
> refer to invalid cells.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17825) Backup: further optimizations

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357561#comment-16357561
 ] 

Josh Elser commented on HBASE-17825:


bq. I do not remember exactly, but it was necessary to remove this data 
subdirectory to make the patch work. Anyway, I simplified the structure of 
backup file system.

Is this documented somewhere that should be updated to reflect the new 
structure? It doesn't ring any bells to me.

This is just "in-flight" backups, so there's no concern with any existing 
backup data, right?

> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
>  Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357552#comment-16357552
 ] 

Hadoop QA commented on HBASE-19876:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
49s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
42s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 15s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}115m  
6s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19876 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909826/HBASE-19876.v5.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux e3a1d3476825 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 277ce3d8a3 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11455/testReport/ |
| Max. process+thread count | 5748 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11455/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> The exception happening in converting pb mutation to hbase.mutation messes up 
> the CellScanner
> 

[jira] [Updated] (HBASE-19835) Make explicit casting of atleast one operand to final type

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-19835:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0-beta-2
   Status: Resolved  (was: Patch Available)

Test failure was not related.

Pushed to branch-2 and master

Thanks, Aman

> Make explicit casting of atleast one operand to final type
> --
>
> Key: HBASE-19835
> URL: https://issues.apache.org/jira/browse/HBASE-19835
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Minor
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19835.master.01.patch, HBASE-19835.master.02.patch
>
>
> We have used 
> _long = int + int_ 
> at many places mostly wherever ClassSize.java variables are used for 
> calculation.
> Need to cast explicitly at-least one operand to final type(i.e. type the 
> result is intended to be casted).
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19835) Make explicit casting of atleast one operand to final type

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357523#comment-16357523
 ] 

Hadoop QA commented on HBASE-19835:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
51s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
43s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
19s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
58s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 43s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  3m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 58s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19835 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909740/HBASE-19835.master.02.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux df8aaf58db35 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (HBASE-17825) Backup: further optimizations

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357516#comment-16357516
 ] 

Vladimir Rodionov commented on HBASE-17825:
---

{quote}
Why change directory ?
{quote}

I do not remember exactly, but it was necessary to remove this *data* 
subdirectory to make the patch work. Anyway, I simplified the structure of 
backup file system.

 

> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
>  Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357509#comment-16357509
 ] 

Hadoop QA commented on HBASE-19920:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  7m 
46s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
13s{color} | {color:red} hbase-server: The patch generated 2 new + 5 unchanged 
- 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
11s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
21m 58s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
10s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m  
0s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
42s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19920 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909825/HBASE-19920.v6.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux d515c4d769a6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / 277ce3d8a3 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 

[jira] [Resolved] (HBASE-19962) Regression : Unable to use the variable to which a table is assigned

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-19962.

Resolution: Duplicate

HBASE-19770

> Regression : Unable to use the variable to which a table is assigned
> 
>
> Key: HBASE-19962
> URL: https://issues.apache.org/jira/browse/HBASE-19962
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Reporter: Sharmadha Sainath
>Priority: Major
>
> Getting the table using variable t and using it to do HBase operations fails :
> {code}
> hbase(main):010:0> t = get_table 't'
> Took 0.0015 seconds
> hbase(main):011:0> t.scan
> NoMethodError: undefined method `command' for nil:NilClass
> {code}
> HBase version : 2.0.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19898) Canary should choose RegionStdOutSink automatically when write sniffing is specified

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-19898:
---
Status: Open  (was: Patch Available)

> Canary should choose RegionStdOutSink automatically when write sniffing is 
> specified
> 
>
> Key: HBASE-19898
> URL: https://issues.apache.org/jira/browse/HBASE-19898
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Trivial
> Attachments: 19898.v1.txt, 19898.v1.txt
>
>
> Currently RegionServerStdOutSink is instantiated by default, even if user 
> specifies -writeSniffing on the command line.
> Write sniffing would be ignored since Sink instance is of 
> RegionServerStdOutSink class:
> {code}
> if (this.sink instanceof RegionServerStdOutSink || this.regionServerMode) 
> {
>   monitor =
>   new RegionServerMonitor(connection, monitorTargets, this.useRegExp,
>   (StdOutSink) this.sink, this.executor, 
> this.regionServerAllRegions,
>   this.treatFailureAsError);
> {code}
> I discovered this during testing Canary tool on cluster.
> The downside of current behavior is that when user runs daemon mode with 
> write sniffing, he / she would easily miss that Canary tool doesn't actually 
> perform write sniffing, leading to false negative.
> RegionStdOutSink should be used for write sniffing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19952) Find tests which are declared with wrong category

[Appy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357423#comment-16357423
 ] 

Appy commented on HBASE-19952:
--

Tweaking forkCount and jvm reuse sure looks like something we can evaluate to 
make our tests faster, but at the same time, changing these high level 
properties might be too big of perturbations for time like this when our 
priority is test stabilization rather then test speed.
Who knows...making it to 5 or 10 reduces cpu resources available to per 
test/minicluster so low that we start seeing new issues; resource constraint 
might not seem as apparent cause, and a lot of time and energy will be spent 
hunting for them.
If it's not *needed*, since we already made change in HBASE-19956, let's skip 
it for now?

> Find tests which are declared with wrong category
> -
>
> Key: HBASE-19952
> URL: https://issues.apache.org/jira/browse/HBASE-19952
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19735) Create a minimal "client" tarball installation

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357415#comment-16357415
 ] 

Hadoop QA commented on HBASE-19735:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 13m 
43s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
47s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 55s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m  2s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19735 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909817/HBASE-19735.004.patch 
|
| Optional Tests |  asflicense  javac  javadoc  unit  shadedjars  hadoopcheck  
xml  compile  |
| uname | Linux 6e49ec354e5d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 277ce3d8a3 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11451/artifact/patchprocess/patch-unit-root.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11451/testReport/ |
| Max. process+thread count | 5323 (vs. ulimit of 1) |
| modules | C: hbase-procedure hbase-spark hbase-assembly . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11451/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Create a minimal "client" tarball installation
> 

[jira] [Commented] (HBASE-17825) Backup: further optimizations

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357357#comment-16357357
 ] 

Josh Elser commented on HBASE-17825:


{code}
+  if (fileName.indexOf(".tabledesc") > 0 || 
+  fileName.indexOf(".regioninfo") > 0 ) {
{code}

Do we have constants elsewhere we can use instead of the literal strings?

{code}
   TableMapReduceUtil.addDependencyJarsForClasses(job.getConfiguration(),
-  
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.class);
+
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.class);
{code}

{{TableMapReduceUtil.addHBaseDependencyJars}} should be doing this already for 
you. I think you should be able to drop this right now, but maybe it's good to 
preserve to prevent any breakage in the future? Not sure..

Also curious about the change in data directory that Ted pointed out.

> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
>  Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner

[Chia-Ping Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated HBASE-19876:
---
Attachment: HBASE-19876.v5.patch

> The exception happening in converting pb mutation to hbase.mutation messes up 
> the CellScanner
> -
>
> Key: HBASE-19876
> URL: https://issues.apache.org/jira/browse/HBASE-19876
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19876.master.001.patch, HBASE-19876.v0.patch, 
> HBASE-19876.v1.patch, HBASE-19876.v2.patch, HBASE-19876.v3.patch, 
> HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch, 
> HBASE-19876.v4.patch, HBASE-19876.v5.patch
>
>
> {code:java}
> 2018-01-27 22:51:43,794 INFO  [hconnection-0x3291b443-shared-pool11-t6] 
> client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, 
> attempt=6/16 failed=20ops, last 
> exception=org.apache.hadoop.hbase.client.WrongRowIOException: 
> org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't 
> match the original one aaa
>   at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
>   at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code}
> I noticed this bug when testing the table space quota.
> When rs are converting pb mutation to hbase.mutation, the quota exception or 
> cell exception may be thrown.
> {code}
> Unable to find source-code formatter for language: 
> rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, 
> applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, 
> java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, 
> rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml  for 
> (ClientProtos.Action action: mutations) {
> MutationProto m = action.getMutation();
> Mutation mutation;
> if (m.getMutateType() == MutationType.PUT) {
>   mutation = ProtobufUtil.toPut(m, cells);
>   batchContainsPuts = true;
> } else {
>   mutation = ProtobufUtil.toDelete(m, cells);
>   batchContainsDelete = true;
> }
> mutationActionMap.put(mutation, action);
> mArray[i++] = mutation;
> checkCellSizeLimit(region, mutation);
> // Check if a space quota disallows this mutation
> spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
> quota.addMutation(mutation);
>   }
> {code}
> rs has caught the exception but it doesn't have the cellscanner skip the 
> failed cells.
> {code:java}
> } catch (IOException ie) {
>   if (atomic) {
> throw ie;
>   }
>   for (Action mutation : mutations) {
> builder.addResultOrException(getResultOrException(ie, 
> mutation.getIndex()));
>   }
> }
> {code}
> The bug results in the WrongRowIOException to remaining mutations since they 
> refer to invalid cells.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto

[Chia-Ping Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357308#comment-16357308
 ] 

Chia-Ping Tsai commented on HBASE-19930:


{quote}When the in memory flush happens to a CCM, then itself all cells 
(including any big cells) should get copied to MSLAB
{quote}
When the in memory flush happens, the first ImmutableSegment added to pipeline 
is CSLMImmutableSegment. Hence, it has a good chance that a 
CSLMImmutableSegment exists in the merge group when the count of segments 
reaches the threshold (the action is changed from flatten to merge).

> fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
> ---
>
> Key: HBASE-19930
> URL: https://issues.apache.org/jira/browse/HBASE-19930
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0-beta-1
>Reporter: Gali Sheffi
>Assignee: Gali Sheffi
>Priority: Major
> Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, 
> HBASE-19930-V03.patch, HBASE-19930-V04.patch
>
>
> This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto.
> Following a comment in HBASE-19133 regarding a bug in 
> ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never 
> called for an ImmutableMemStoreLAB, and just throwing an 
> IllegalStateException whenever called), the forceCopyOfBigCellInto method now 
> performs the copy of big cells on the first MSLABImpl in its mslabs 
> linked-list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357270#comment-16357270
 ] 

Mike Drob commented on HBASE-19920:
---

v6: checkstyle

> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, 
> HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, 
> HBASE-19920.v6.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil 
> which in its static block initializes DynamicClassLoader and that creates the 
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class 
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when 
> one accesses hbase as a client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-19920:
--
Attachment: HBASE-19920.v6.patch

> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, 
> HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, 
> HBASE-19920.v6.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil 
> which in its static block initializes DynamicClassLoader and that creates the 
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class 
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when 
> one accesses hbase as a client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357267#comment-16357267
 ] 

Ted Yu commented on HBASE-19954:


Here is related test output with exception (the first line was added for 
debugging):
{code}
2018-02-08 17:22:23,382 DEBUG [RS:0;cn012:38253] 
util.ShutdownHookManager$ShutdownHookManagerV2(94): removing hook 
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92): 
Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/cn012:0, 
corePoolSize=5, maxPoolSize=5
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92): 
Starting executor service name=M_LOG_REPLAY_OPS-master/cn012:0, 
corePoolSize=10, maxPoolSize=10
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92): 
Starting executor service name=MASTER_TABLE_OPERATIONS-master/cn012:0, 
corePoolSize=1, maxPoolSize=1
2018-02-08 17:22:23,389 ERROR [RS:0;cn012:38253] 
helpers.MarkerIgnoringBase(159): * ABORTING region server 
cn012.l42scl.hortonworks.com,38253,1518110542271: Unhandled: Failed suppression 
of fs shutdown hook: 
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e *
java.lang.RuntimeException: Failed suppression of fs shutdown hook: 
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e
  at 
org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:207)
  at 
org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:85)
  at 
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:927)
  at 
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187)
  at 
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:133)
  at 
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171)
  at java.security.AccessController.doPrivileged(Native Method)
{code}
I also added a debug log in 
ShutdownHookManager#ShutdownHookManagerV2#addShutdownHook() which didn't show 
up in the test output.
Since there was no check that hadoop tracks the ClientFinalizer before calling 
#removeShutdownHook, the false return from hadoop resulted in the 
RuntimeException to be thrown.


> ShutdownHook should check whether shutdown hook is tracked by 
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
>   synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled = 
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled && 
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook, 
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins 
> job).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19835) Make explicit casting of atleast one operand to final type

[Aman Poonia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aman Poonia updated HBASE-19835:

Status: Patch Available  (was: In Progress)

> Make explicit casting of atleast one operand to final type
> --
>
> Key: HBASE-19835
> URL: https://issues.apache.org/jira/browse/HBASE-19835
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Minor
> Attachments: HBASE-19835.master.01.patch, HBASE-19835.master.02.patch
>
>
> We have used 
> _long = int + int_ 
> at many places mostly wherever ClassSize.java variables are used for 
> calculation.
> Need to cast explicitly at-least one operand to final type(i.e. type the 
> result is intended to be casted).
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15381) Implement a distributed MOB compaction by procedure

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357244#comment-16357244
 ] 

huaxiang sun commented on HBASE-15381:
--

Hi [~jingcheng...@intel.com], I am planning to spend time on this feature and 
finish it. Do you mind if I take it over from here? Thanks

> Implement a distributed MOB compaction by procedure
> ---
>
> Key: HBASE-15381
> URL: https://issues.apache.org/jira/browse/HBASE-15381
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
>Priority: Major
> Attachments: HBASE-15381-v2.patch, HBASE-15381-v3.patch, 
> HBASE-15381-v4.patch, HBASE-15381-v5.patch, HBASE-15381-v6.patch, 
> HBASE-15381.patch, mob distributed compaction design-v2.pdf, mob distributed 
> compaction design.pdf
>
>
> In MOB, there is a periodical compaction which runs in HMaster (It can be 
> disabled by configuration), some small mob files are merged into bigger ones. 
> Now the compaction only runs in HMaster which is not efficient and might 
> impact the running of HMaster. In this JIRA, a distributed MOB compaction is 
> introduced, it is triggered by HMaster, but all the compaction jobs are 
> distributed to HRegionServers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19824) SingleColumnValueFilter returns wrong result when used in shell command

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357223#comment-16357223
 ] 

Haibo Chen commented on HBASE-19824:


Thanks [~mdrob]

> SingleColumnValueFilter returns wrong result when used in shell command
> ---
>
> Key: HBASE-19824
> URL: https://issues.apache.org/jira/browse/HBASE-19824
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Ted Yu
>Priority: Major
> Attachments: HBASE-19824.master.001.patch, 
> HBASE-19824.master.002.patch
>
>
> There are two cells in table t1:
> {code}
> ROW COLUMN+CELL
>  r1 column=f1:a1, 
> timestamp=1516313683984, value=a2
>  r1 column=f1:b1, 
> timestamp=1516313700744, value=b2
> {code}
> When SingleColumnValueFilter is used in shell command, no filtering was done:
> {code}
> hbase(main):022:0> scan 't1', {FILTER => "SingleColumnValueFilter('f1', 'a1', 
> =, 'binary:a2')"}
> ROW COLUMN+CELL
>  r1 column=f1:a1, 
> timestamp=1516313683984, value=a2
>  r1 column=f1:b1, 
> timestamp=1516313700744, value=b2
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357219#comment-16357219
 ] 

Hadoop QA commented on HBASE-19937:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
10s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} hbase-rsgroup: The patch generated 1 new + 0 unchanged 
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 1s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
14m 17s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 15s{color} 
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m  
9s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.rsgroup.TestEnableRSGroup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19937 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909781/HBASE-19937.branch-2.002.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 6befb01ab0de 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | branch-2 / c93ba37dfc |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11452/artifact/patchprocess/diff-checkstyle-hbase-rsgroup.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11452/artifact/patchprocess/patch-unit-hbase-rsgroup.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11452/testReport/ |
| asflicense | 

[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357212#comment-16357212
 ] 

stack commented on HBASE-19954:
---

What is the exception seen? Why does it happen? Why would the manager not be 
tracking the shutdown hook still? Has it been removed? If so, by whom? Thanks.

> ShutdownHook should check whether shutdown hook is tracked by 
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
>   synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled = 
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled && 
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook, 
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins 
> job).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19929) Call RS.stop on a session expired RS may hang

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357204#comment-16357204
 ] 

stack commented on HBASE-19929:
---

Ok. Still some timeouts on nightlies. Digging. Thanks for heads-up.

> Call RS.stop on a session expired RS may hang
> -
>
> Key: HBASE-19929
> URL: https://issues.apache.org/jira/browse/HBASE-19929
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19929-v1.patch, HBASE-19929.patch
>
>
> See the discussion in HBASE-19927. The problem is that, for a normal stop we 
> will try to close all the regions and wait until they are all closed. But if 
> the RS has already session expired, master will start the failover work which 
> will move the WAL directory, and then we will be stuck in writing flush 
> marker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357175#comment-16357175
 ] 

Hadoop QA commented on HBASE-19950:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
45s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle 
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} The patch hbase-protocol passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} hbase-client: The patch generated 0 new + 71 
unchanged - 2 fixed = 71 total (was 73) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
41s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 39s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}  
1m 36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
58s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}101m 
36s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 

[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357170#comment-16357170
 ] 

Ted Yu commented on HBASE-19937:


Combine the two patches into one.
Add license header to the test.


> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup 

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-19937:
---
Status: Patch Available  (was: Open)

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> 

[jira] [Commented] (HBASE-19735) Create a minimal "client" tarball installation

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357138#comment-16357138
 ] 

Josh Elser commented on HBASE-19735:


The hbase-spark issue was a problem with that module not directly listing its 
test-dependency on hbase-common:tests.

> Create a minimal "client" tarball installation
> --
>
> Key: HBASE-19735
> URL: https://issues.apache.org/jira/browse/HBASE-19735
> Project: HBase
>  Issue Type: New Feature
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19735.000.patch, HBASE-19735.001.branch-2.patch, 
> HBASE-19735.002.branch-2.patch, HBASE-19735.003.patch, HBASE-19735.004.patch
>
>
> We're moving ourselves towards more controlled dependencies. A logical next 
> step is to try to do the same for our "binary" artifacts that we create 
> during releases.
> There is code (our's and our dependency's) which the HMaster and RegionServer 
> require which, obviously, clients do not need.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19735) Create a minimal "client" tarball installation

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-19735:
---
Attachment: HBASE-19735.004.patch

> Create a minimal "client" tarball installation
> --
>
> Key: HBASE-19735
> URL: https://issues.apache.org/jira/browse/HBASE-19735
> Project: HBase
>  Issue Type: New Feature
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19735.000.patch, HBASE-19735.001.branch-2.patch, 
> HBASE-19735.002.branch-2.patch, HBASE-19735.003.patch, HBASE-19735.004.patch
>
>
> We're moving ourselves towards more controlled dependencies. A logical next 
> step is to try to do the same for our "binary" artifacts that we create 
> during releases.
> There is code (our's and our dependency's) which the HMaster and RegionServer 
> require which, obviously, clients do not need.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-19963) TestFSHDFSUtils assumes wrong default port for Hadoop 3.0.1+

[Mike Drob (JIRA)]

Mike Drob created HBASE-19963:
-

 Summary: TestFSHDFSUtils assumes wrong default port for Hadoop 
3.0.1+
 Key: HBASE-19963
 URL: https://issues.apache.org/jira/browse/HBASE-19963
 Project: HBase
  Issue Type: Task
  Components: test
Reporter: Mike Drob


We try to accommodate HDFS changing ports when testing if it is the same FS in 
our tests:
https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSHDFSUtils.java#L156-L162

{code}
if (isHadoop3) {
  // Hadoop 3.0.0 alpha1+ change default nn port to 9820. See HDFS-9427
  testIsSameHdfs(9820);
} else {
  // pre hadoop 3.0.0 defaults to port 8020
  testIsSameHdfs(8020);
}
{code}

But in Hadoop 3.0.1, they decided to go back to the old port - see HDFS-12990.

So our tests will fail against the snapshot and against future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter

[Reid Chan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357085#comment-16357085
 ] 

Reid Chan commented on HBASE-19950:
---

I got your mind, and thought over this Q myself.
When query comes to more complicated, combinations of filters needed, 
{{Get/Scan #addColumn(byte[] family, byte[] qualifier))}} may be too a strong 
constraint.
It is more like a complementation to SCVF, IMO.

> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
>  Issue Type: Improvement
>  Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch, 
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when 
> specified condition is matched, this new filter will return the matched cell 
> only. There're already some discussions in HBASE-19824.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19824) SingleColumnValueFilter returns wrong result when used in shell command

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357048#comment-16357048
 ] 

Mike Drob commented on HBASE-19824:
---

[~haibochen] - linked to HBASE-19950 for you.

> SingleColumnValueFilter returns wrong result when used in shell command
> ---
>
> Key: HBASE-19824
> URL: https://issues.apache.org/jira/browse/HBASE-19824
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Ted Yu
>Priority: Major
> Attachments: HBASE-19824.master.001.patch, 
> HBASE-19824.master.002.patch
>
>
> There are two cells in table t1:
> {code}
> ROW COLUMN+CELL
>  r1 column=f1:a1, 
> timestamp=1516313683984, value=a2
>  r1 column=f1:b1, 
> timestamp=1516313700744, value=b2
> {code}
> When SingleColumnValueFilter is used in shell command, no filtering was done:
> {code}
> hbase(main):022:0> scan 't1', {FILTER => "SingleColumnValueFilter('f1', 'a1', 
> =, 'binary:a2')"}
> ROW COLUMN+CELL
>  r1 column=f1:a1, 
> timestamp=1516313683984, value=a2
>  r1 column=f1:b1, 
> timestamp=1516313700744, value=b2
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356833#comment-16356833
 ] 

Xiaolin Ha edited comment on HBASE-19937 at 2/8/18 3:03 PM:


The patch 002 is with a UT which can make the bug appear. Please ensure the 
createRSGroupTable is called in a run.

You can notice the log: 
{code:java}
2018-02-08 19:16:47,398 INFO  
[org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130]
 master.HMaster(1791): Client=null/null create 'hbase:rsgroup', 
{TABLE_ATTRIBUTES => {coprocessor$1 => 
'|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|', 
METADATA => {'SPLIT_POLICY' => 
'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 
'm', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 
'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', 
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', 
REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 
'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', 
PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 
'true', BLOCKSIZE => '65536'}
{code}
 

The patch 001 has fixed this problem.

 

 


was (Author: xiaolin ha):
The patch 002 is add a UT which can make the bug appear. Please ensure the 
createRSGroupTable is called in a run.

You can notice the log: 
{code:java}
2018-02-08 19:16:47,398 INFO  
[org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130]
 master.HMaster(1791): Client=null/null create 'hbase:rsgroup', 
{TABLE_ATTRIBUTES => {coprocessor$1 => 
'|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|', 
METADATA => {'SPLIT_POLICY' => 
'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 
'm', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 
'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', 
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', 
REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 
'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', 
PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 
'true', BLOCKSIZE => '65536'}
{code}
 

The patch 001 has fixed this problem.

 

 

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> -
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 2.0.0-beta-2
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch, 
> HBASE-19937.branch-2.002.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master 
> load system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounter Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure 

[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357037#comment-16357037
 ] 

Anoop Sam John commented on HBASE-19950:


One Q.  In such a requirement, why we can not project the required column alone 
in Get/Scan (  addColumn(byte [] family, byte [] qualifier) ) and add a SCVF 
also with it?  Am I missing some thing?

> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
>  Issue Type: Improvement
>  Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch, 
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when 
> specified condition is matched, this new filter will return the matched cell 
> only. There're already some discussions in HBASE-19824.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Description: 
hbase:rsgroup table will be created by calling createRSGroupTable when master 
load system coprocessors in 

 
{code:java}
844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
will encounter Exception as follows,

 
{noformat}
Exception in thread 
"org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
 java.lang.IllegalArgumentException
at 
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
while ProcedureExecutor  initialized by calling
{code:java}
848  startServiceThreads();{code}
And LoadBalancer is initialized by calling 
{code:java}
868  this.balancer.initialize();{code}
When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
encounters Exception as follows,

2018-02-02,16:12:45,688 ERROR 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
CreateTableProcedure table=hbase:rsgroup
 java.lang.NullPointerException
 at 
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
 at 
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
 at 
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
 at 
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
 at 
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
 at 
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
 at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)

 

As a result of CreateTableProcedure.rollbackState, it may then print logs 
warning TableExistsException as follows,

2018-02-02,16:12:55,503 WARN 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
Failed to perform check
 java.io.IOException: Failed to create group table. 
org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
 at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)

 

After some auto-retries, it loops running the thread RSGroupStartupWorker, will 
print logs as follows, 

2018-02-02,16:23:17,626 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true
 2018-02-02,16:23:17,730 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true
 2018-02-02,16:23:17,834 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true
 2018-02-02,16:23:17,937 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true

 

And using shells of rsgroup, it will tips that currently is in "offline mode".

 

The reason of this problem is that the order of createRSGroupTable and 
initializing of ProcedureExecutor and LoadBalancer is out of control. If the 
former is excuted earlier, it will encounter Exception mentioned before.

 

  was:
rsgroup table will be created by calling 

[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

[Xiaolin Ha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
---
Description: 
rsgroup table will be created by calling createRSGroupTable when master load 
system coprocessors in 

 
{code:java}
844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
will encounter Exception as follows,

 
{noformat}
Exception in thread 
"org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
 java.lang.IllegalArgumentException
at 
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
while ProcedureExecutor  initialized by calling
{code:java}
848  startServiceThreads();{code}
And LoadBalancer is initialized by calling 
{code:java}
868  this.balancer.initialize();{code}
When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
encounters Exception as follows,

2018-02-02,16:12:45,688 ERROR 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
CreateTableProcedure table=hbase:rsgroup
 java.lang.NullPointerException
 at 
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
 at 
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
 at 
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
 at 
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
 at 
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
 at 
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
 at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
 at 
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)

 

As a result of CreateTableProcedure.rollbackState, it may then print logs 
warning TableExistsException as follows,

2018-02-02,16:12:55,503 WARN 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
Failed to perform check
 java.io.IOException: Failed to create group table. 
org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
 at 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)

 

After some auto-retries, it loops running the thread RSGroupStartupWorker, will 
print logs as follows, 

2018-02-02,16:23:17,626 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true
 2018-02-02,16:23:17,730 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true
 2018-02-02,16:23:17,834 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true
 2018-02-02,16:23:17,937 INFO 
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
rootMetaFound=true

 

And using shells of rsgroup, it will tips that currently is in "offline mode".

 

The reason of this problem is that the order of createRSGroupTable and 
initializing of ProcedureExecutor and LoadBalancer is out of control. If the 
former is excuted earlier, it will encounter Exception mentioned before.

 

  was:
rsgroup table will be created by calling createRSGroupTable 

[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357028#comment-16357028
 ] 

Anoop Sam John commented on HBASE-19930:


Same Q as Ram's.  Not sure how we will reach to such a call.  When the in 
memory flush happens to a CCM, then itself all cells (including any big cells) 
should get copied to MSLAB.  This ImmutableMemStoreLAB comes into picture when 
many Immutable segments are merged into one.  So then only we need this copy of 
big cells?  Or is it the case with non basic type? (data compaction not just 
index compaction?)

> fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
> ---
>
> Key: HBASE-19930
> URL: https://issues.apache.org/jira/browse/HBASE-19930
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0-beta-1
>Reporter: Gali Sheffi
>Assignee: Gali Sheffi
>Priority: Major
> Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, 
> HBASE-19930-V03.patch, HBASE-19930-V04.patch
>
>
> This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto.
> Following a comment in HBASE-19133 regarding a bug in 
> ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never 
> called for an ImmutableMemStoreLAB, and just throwing an 
> IllegalStateException whenever called), the forceCopyOfBigCellInto method now 
> performs the copy of big cells on the first MSLABImpl in its mslabs 
> linked-list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19506) Support variable sized chunks from ChunkCreator

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357016#comment-16357016
 ] 

Hadoop QA commented on HBASE-19506:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
58s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
46s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} hbase-server: The patch generated 0 new + 10 
unchanged - 3 fixed = 10 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
40s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 56s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m  1s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 26s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestKeepDeletes |
|   | hadoop.hbase.filter.TestFilter |
|   | hadoop.hbase.coprocessor.TestCoprocessorInterface |
|   | hadoop.hbase.regionserver.TestScanWithBloomError |
|   | hadoop.hbase.regionserver.TestScanner |
|   | hadoop.hbase.io.encoding.TestSeekBeforeWithReverseScan |
|   | hadoop.hbase.regionserver.TestBlocksScanned |
|   | hadoop.hbase.regionserver.TestStoreFileRefresherChore |
|   | hadoop.hbase.filter.TestMultipleColumnPrefixFilter |
|   | hadoop.hbase.filter.TestColumnPrefixFilter |
|   | hadoop.hbase.coprocessor.TestRegionObserverStacking |
|   | hadoop.hbase.filter.TestFilterFromRegionSide |
|   | hadoop.hbase.filter.TestInvocationRecordFilter |
|   | hadoop.hbase.regionserver.TestWideScanner |
|   | hadoop.hbase.regionserver.TestMinVersions |
|   | hadoop.hbase.filter.TestDependentColumnFilter |
|   | hadoop.hbase.client.TestIntraRowPagination |
|   | hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19506 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12909802/HBASE-19506-V04.patch 
|
| Optional Tests |  asflicense  javac  

[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter

[Peter Somogyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356979#comment-16356979
 ] 

Peter Somogyi commented on HBASE-19950:
---

+1

Thanks for addressing my review comments!

> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
>  Issue Type: Improvement
>  Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch, 
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when 
> specified condition is matched, this new filter will return the matched cell 
> only. There're already some discussions in HBASE-19824.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19959) How much RAM space is to be really consumed by the memstore?

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356980#comment-16356980
 ] 

Anastasia Braginsky commented on HBASE-19959:
-

Pay attention that when something is allocated (using new or malloc) like 
chunks in pool, but never written (as you explain), than it only takes part of 
virtual memory (virtual address space). No physical RAM is used/taken.

> How much RAM space is to be really consumed by the memstore?
> 
>
> Key: HBASE-19959
> URL: https://issues.apache.org/jira/browse/HBASE-19959
> Project: HBase
>  Issue Type: Brainstorming
>  Components: regionserver
>Reporter: Chance Li
>Priority: Minor
>
> Let's consider this scenario where memstoreLAB and ChunkPool is enable and 
> max memstore size is 10G, and after some time all pooled chunk have been 
> created, then flush all data, now memstore size is 0 but RAM actually have 
> consumed 10G, then continue writing big cell which will not use the chunk 
> pool but jvm heap, then memstore size will be increased to 10G(maybe more 
> because overhead). now we can see RAM actually consumed 20G (10G of pooled 
> chunk + 10G java objects), but the max memstore size is only 10G.
> what I say is the max memstore size not only take care about the cell "size" 
> but also RAM really used. This will be a strict memory management: the max 
> memstore size limit the RAM space which the memstore or related module can be 
> used.
> This really rarely occured. It's just for a robust memory managemant 
> semantically. 
>  What do you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter

[Reid Chan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reid Chan updated HBASE-19950:
--
Attachment: HBASE-19950.master.003.patch

> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
>  Issue Type: Improvement
>  Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch, 
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when 
> specified condition is matched, this new filter will return the matched cell 
> only. There're already some discussions in HBASE-19824.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19506) Support variable sized chunks from ChunkCreator

[Gali Sheffi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gali Sheffi updated HBASE-19506:

Attachment: HBASE-19506-V04.patch

> Support variable sized chunks from ChunkCreator
> ---
>
> Key: HBASE-19506
> URL: https://issues.apache.org/jira/browse/HBASE-19506
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Priority: Major
> Attachments: HBASE-19506-V01.patch, HBASE-19506-V02.patch, 
> HBASE-19506-V03.patch, HBASE-19506-V04.patch
>
>
> When CellChunkMap is created it allocates a special index chunk (or chunks) 
> where array of cell-representations is stored. When the number of 
> cell-representations is small, it is preferable to allocate a chunk smaller 
> than a default value which is 2MB.
> On the other hand, those "non-standard size" chunks can not be used in pool. 
> On-demand allocations in off-heap are costly. So this JIRA is about to 
> investigate the trade of between memory usage and the final performance. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19929) Call RS.stop on a session expired RS may hang

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356906#comment-16356906
 ] 

Duo Zhang commented on HBASE-19929:
---

[~stack] When implementing the test for this issue, I found that if I kill the 
RS which holds meta region, then it will be likely to timeout.

In the test, I setup a cluster with 2 RSes, and kill one RS, and then waiting 
for 30 seconds until all the regions are onlined on the other RS. If I kill the 
one with meta region, then usually I will get a timeout when waiting for the 
regions to online... I guess there maybe some problems...

Thanks.

> Call RS.stop on a session expired RS may hang
> -
>
> Key: HBASE-19929
> URL: https://issues.apache.org/jira/browse/HBASE-19929
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19929-v1.patch, HBASE-19929.patch
>
>
> See the discussion in HBASE-19927. The problem is that, for a normal stop we 
> will try to close all the regions and wait until they are all closed. But if 
> the RS has already session expired, master will start the failover work which 
> will move the WAL directory, and then we will be stuck in writing flush 
> marker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19929) Call RS.stop on a session expired RS may hang

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-19929:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

No big problems. Resolve.

> Call RS.stop on a session expired RS may hang
> -
>
> Key: HBASE-19929
> URL: https://issues.apache.org/jira/browse/HBASE-19929
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19929-v1.patch, HBASE-19929.patch
>
>
> See the discussion in HBASE-19927. The problem is that, for a normal stop we 
> will try to close all the regions and wait until they are all closed. But if 
> the RS has already session expired, master will start the failover work which 
> will move the WAL directory, and then we will be stuck in writing flush 
> marker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19929) Call RS.stop on a session expired RS may hang

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-19929:
--
Component/s: wal

> Call RS.stop on a session expired RS may hang
> -
>
> Key: HBASE-19929
> URL: https://issues.apache.org/jira/browse/HBASE-19929
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19929-v1.patch, HBASE-19929.patch
>
>
> See the discussion in HBASE-19927. The problem is that, for a normal stop we 
> will try to close all the regions and wait until they are all closed. But if 
> the RS has already session expired, master will start the failover work which 
> will move the WAL directory, and then we will be stuck in writing flush 
> marker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356900#comment-16356900
 ] 

Duo Zhang commented on HBASE-19942:
---

Maybe another bug?

https://builds.apache.org/job/HBASE-Flaky-Tests/25692/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testHandlerIsolation/

> Fix flaky TestSimpleRpcScheduler
> 
>
> Key: HBASE-19942
> URL: https://issues.apache.org/jira/browse/HBASE-19942
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19942.branch-2.001.patch, 
> HBASE-19942.master.001.patch
>
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html]
>  
> https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/
>  
> h3. Stacktrace
> java.lang.AssertionError at 
> org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19956) Remove category as a consideration timing out tests; set all test to timeout at 10minutes regardless

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356897#comment-16356897
 ] 

Hudson commented on HBASE-19956:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4548 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4548/])
HBASE-19956 Remove category as a consideration timing out tests; set all 
(stack: rev 277ce3d8a3d68f87e82c98fdeb3a3cb3dee33036)
* (edit) hbase-server/pom.xml
* (edit) 
hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseClassTestRule.java


> Remove category as a consideration timing out tests; set all test to timeout 
> at 10minutes regardless
> 
>
> Key: HBASE-19956
> URL: https://issues.apache.org/jira/browse/HBASE-19956
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19956.branch-2.001.patch
>
>
> Appy suggestion from parent issue.
> Notion is that during transition measuring timeout on a class-basis instead 
> of on a test method basis, categories only mess us up. The only important 
> regard when it comes to timeout is that our timeout cuts in before the 
> surefire one does; it just does a savage kill w/o regard for who is about. If 
> we kill, there'll be info on culprit. Thats what we want.
> We used to have timeout per test method. It was changed for hbase2 to be 
> class based. Easier for devs to grok. For now, we purge changing test class 
> timeout based off category and just paint all tests w/ the one 10-minute max 
> brush.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356879#comment-16356879
 ] 

Ted Yu commented on HBASE-19954:


[~chia7712]:
Can you take a look ?

> ShutdownHook should check whether shutdown hook is tracked by 
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
>   synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled = 
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled && 
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook, 
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins 
> job).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)