[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913320#comment-16913320
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2
[build #2184 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for 
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//artifact/output-integration/hadoop-3.log].
 (note that this means we didn't check the Hadoop 3 shaded client)


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913291#comment-16913291
 ] 

Hudson commented on HBASE-22810:


Results for branch master
[build #1352 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1352/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1352//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1352//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1352//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913226#comment-16913226
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2.1
[build #1509 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913223#comment-16913223
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2.2
[build #533 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913211#comment-16913211
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-1
[build #1021 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021//JDK7_Nightly_Build_Report/]


(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913207#comment-16913207
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-1.4
[build #971 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971//JDK7_Nightly_Build_Report/]


(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913202#comment-16913202
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-1.3
[build #931 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931//General_Nightly_Build_Report/]


(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-22 Thread Hudson (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913042#comment-16913042
 ] 

Hudson commented on HBASE-22810:


SUCCESS: Integrated in Jenkins build HBase-1.3-IT #623 (See 
[https://builds.apache.org/job/HBase-1.3-IT/623/])
HBASE-22810 Initialize an separate ThreadPoolExecutor for (openinx: rev 
40cf771a825cf657f5f7cb0f7ef8c986e5b1fe4c)
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestConcurrentFlushSnapshotFromClient.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-21 Thread Zheng Hu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912993#comment-16912993
 ] 

Zheng Hu commented on HBASE-22810:
--

Committed the addendum to all branches, Thanks all.

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-21 Thread Zheng Hu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912278#comment-16912278
 ] 

Zheng Hu commented on HBASE-22810:
--

Created a PR to address the above uniform & add a UT.

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-21 Thread Zheng Hu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912256#comment-16912256
 ] 

Zheng Hu commented on HBASE-22810:
--

Thanks [~stack] for the fix.  Read the UT code,  it's indeed  a test which easy 
to be flaky.  For example, all snapshot request are submitted but the snapshot 
is a bit slow,  none are completed  when the assert begin: 
{code}
+assertTrue("We expect at least 1 request to be rejected because of we 
concurrently" +
+" issued many requests", takenSize < ssNum && takenSize > 0);
{code}
Then, the assert will be failure.  so +1 for me to remove it (I guess after 
increasing the 'hbase.master.executor.snapshot.threads',   it's easy to happen 
now).

[~an...@apache.org], Thanks for the reminding .  It's true, there are two 
different config keys for the snapshot threads size,  but I think they have 
different meanings:
1.  hbase.master.executor.snapshot.threads :   means how many snapshot requests 
from client we can handle at master side the same time; 
2. hbase.snapshot.master.threads:   how many snapshot procedure we can 
coordinator with region server. 
The config key#1 limit the all the snapshot request, while the key#2 only limit 
the snapshot procedure with RS ( it's a part of the snapshot request).Maybe 
we can uniform the two config keys into one ?  although we will initialize two 
different thread pools with the same thread size for different purpose.



> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-20 Thread Ankit Singhal (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911898#comment-16911898
 ] 

Ankit Singhal commented on HBASE-22810:
---

[~openinx], It seems SnapshotManager configures pools separately which allow 
only 1 thread and that's why 
TestFlushSnapshotFromClient#testConcurrentSnapshottingAttempts is failing, so 
you may also need to update the below code with your new config to keep the 
threads consistent.

SnapshotManager#initialize
{code}
int opThreads = conf.getInt(SNAPSHOT_POOL_THREADS_KEY, 
SNAPSHOT_POOL_THREADS_DEFAULT);
{code}

to 

{code}
int opThreads = conf.getInt(MASTER_SNAPSHOT_OPERATIONS_THREADS, 
MASTER_SNAPSHOT_OPERATIONS_THREADS_DEFAULT);
{code}


And, committing improvements which change the defaults value in patch 
releases(2.[0-2].x), doesn't it affect our operational compatibility?


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-19 Thread stack (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910973#comment-16910973
 ] 

stack commented on HBASE-22810:
---

Ok. This did not make it into 2.0.6RC1 which just became 2.0.6. I moved fix 
version to 2.0.7 though there will be no more releases on branch-2.0.

I just saw this issue and how it breaks TestFlushSnapshotFromClient. I looked 
at the TestFlushSnapshotFromClient failures independently and to me the test 
looked flakey anyways and was seemingly written flakey in the first place so 
just removed it with the below commit. Did not go back to branch-1. After 
seeing this issue, I was a mite reactionary it seems.

{code}
commit 05b88af057d35e247bf37b725f2735292f944387 (HEAD -> 2.0, origin/branch-2.0)
Author: stack 
Date:   Mon Aug 19 14:25:21 2019 -0700

 HBASE-22882 TestFlushSnapshotFromClient#testConcurrentSnapshottingAttempts 
is flakey (was written flakey)
 Addendum; just remove the test altogether
{code}

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-18 Thread Zheng Hu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909963#comment-16909963
 ] 

Zheng Hu commented on HBASE-22810:
--

Fine.  Sorry , I think I forgot this issue before, Let me dig this .

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-18 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909956#comment-16909956
 ] 

Duo Zhang commented on HBASE-22810:
---

OK, seems it also breaks the tests on branch-2.1...

https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html

Could you please quickly identify whether this is a test issue or not? If not, 
we need to sink all the pending RCs and prepare new ones.

Thanks. [~openinx]

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908631#comment-16908631
 ] 

Duo Zhang commented on HBASE-22810:
---

It also breaks the test on master.

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Zheng Hu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908626#comment-16908626
 ] 

Zheng Hu commented on HBASE-22810:
--

OK, Thanks [~apurtell], Let me take a look, will provide the addendum soon if 
necessary.

> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908554#comment-16908554
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2.2
[build #517 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for 
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//artifact/output-integration/hadoop-2.log].
 (note that this means we didn't run on Hadoop 3)


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908431#comment-16908431
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2
[build #2169 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908347#comment-16908347
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-1.3
[build #920 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908345#comment-16908345
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-1.4
[build #959 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908234#comment-16908234
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2.0
[build #1867 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908225#comment-16908225
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-2.1
[build #1486 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908058#comment-16908058
 ] 

Hudson commented on HBASE-22810:


Results for branch master
[build #1335 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1335/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1335//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1335//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1335//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907988#comment-16907988
 ] 

Hudson commented on HBASE-22810:


Results for branch branch-1
[build #1009 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

2019-08-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907867#comment-16907867
 ] 

Hudson commented on HBASE-22810:


SUCCESS: Integrated in Jenkins build HBase-1.3-IT #618 (See 
[https://builds.apache.org/job/HBase-1.3-IT/618/])
HBASE-22810 Initialize an separate ThreadPoolExecutor for (openinx: rev 
b97621b30216dc39ad605e06738a070a69a47da3)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot 
> 
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means  taking snapshot 
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS  Executor now. 
> {code}
>   /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
>   C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
>   /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
>   C_M_RESTORE_SNAPSHOT  (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I 
> see : 
> {code}
>   private void startServiceThreads() throws IOException{
>// ...  some other code initializing  
>// We depend on there being only one instance of this executor running
>// at a time.  To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
>  1);
>startProcedureExecutor();
> {code}
> That's to say,  for CPs  enable or disable table sequencely,  we will create 
> a ThreadPoolExecutor with threadPoolSize=1.   Then we actually cann't 
> accomplish the snapshoting  concurrence even if they are total difference 
> tables, says if there are two table snapshoting request, and the Table A cost 
>  5min for snapshoting, then the Table B need to wait 5min and once Table A 
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for 
> table B snapshoting .   Actually,  we can create a separate thead pool for 
> snapshot operations only.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)