[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913320#comment-16913320
]
Hudson commented on HBASE-22810:
Results for branch branch-2
[build #2184 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2184//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913291#comment-16913291
]
Hudson commented on HBASE-22810:
Results for branch master
[build #1352 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1352/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1352//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1352//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1352//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913226#comment-16913226
]
Hudson commented on HBASE-22810:
Results for branch branch-2.1
[build #1509 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1509//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913223#comment-16913223
]
Hudson commented on HBASE-22810:
Results for branch branch-2.2
[build #533 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/533//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913211#comment-16913211
]
Hudson commented on HBASE-22810:
Results for branch branch-1
[build #1021 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1021//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913207#comment-16913207
]
Hudson commented on HBASE-22810:
Results for branch branch-1.4
[build #971 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/971//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913202#comment-16913202
]
Hudson commented on HBASE-22810:
Results for branch branch-1.3
[build #931 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/931//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913042#comment-16913042
]
Hudson commented on HBASE-22810:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #623 (See
[https://builds.apache.org/job/HBase-1.3-IT/623/])
HBASE-22810 Initialize an separate ThreadPoolExecutor for (openinx: rev
40cf771a825cf657f5f7cb0f7ef8c986e5b1fe4c)
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestConcurrentFlushSnapshotFromClient.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912993#comment-16912993
]
Zheng Hu commented on HBASE-22810:
--
Committed the addendum to all branches, Thanks all.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912278#comment-16912278
]
Zheng Hu commented on HBASE-22810:
--
Created a PR to address the above uniform & add a UT.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912256#comment-16912256
]
Zheng Hu commented on HBASE-22810:
--
Thanks [~stack] for the fix. Read the UT code, it's indeed a test which easy
to be flaky. For example, all snapshot request are submitted but the snapshot
is a bit slow, none are completed when the assert begin:
{code}
+assertTrue("We expect at least 1 request to be rejected because of we
concurrently" +
+" issued many requests", takenSize < ssNum && takenSize > 0);
{code}
Then, the assert will be failure. so +1 for me to remove it (I guess after
increasing the 'hbase.master.executor.snapshot.threads', it's easy to happen
now).
[~an...@apache.org], Thanks for the reminding . It's true, there are two
different config keys for the snapshot threads size, but I think they have
different meanings:
1. hbase.master.executor.snapshot.threads : means how many snapshot requests
from client we can handle at master side the same time;
2. hbase.snapshot.master.threads: how many snapshot procedure we can
coordinator with region server.
The config key#1 limit the all the snapshot request, while the key#2 only limit
the snapshot procedure with RS ( it's a part of the snapshot request).Maybe
we can uniform the two config keys into one ? although we will initialize two
different thread pools with the same thread size for different purpose.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911898#comment-16911898
]
Ankit Singhal commented on HBASE-22810:
---
[~openinx], It seems SnapshotManager configures pools separately which allow
only 1 thread and that's why
TestFlushSnapshotFromClient#testConcurrentSnapshottingAttempts is failing, so
you may also need to update the below code with your new config to keep the
threads consistent.
SnapshotManager#initialize
{code}
int opThreads = conf.getInt(SNAPSHOT_POOL_THREADS_KEY,
SNAPSHOT_POOL_THREADS_DEFAULT);
{code}
to
{code}
int opThreads = conf.getInt(MASTER_SNAPSHOT_OPERATIONS_THREADS,
MASTER_SNAPSHOT_OPERATIONS_THREADS_DEFAULT);
{code}
And, committing improvements which change the defaults value in patch
releases(2.[0-2].x), doesn't it affect our operational compatibility?
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910973#comment-16910973
]
stack commented on HBASE-22810:
---
Ok. This did not make it into 2.0.6RC1 which just became 2.0.6. I moved fix
version to 2.0.7 though there will be no more releases on branch-2.0.
I just saw this issue and how it breaks TestFlushSnapshotFromClient. I looked
at the TestFlushSnapshotFromClient failures independently and to me the test
looked flakey anyways and was seemingly written flakey in the first place so
just removed it with the below commit. Did not go back to branch-1. After
seeing this issue, I was a mite reactionary it seems.
{code}
commit 05b88af057d35e247bf37b725f2735292f944387 (HEAD -> 2.0, origin/branch-2.0)
Author: stack
Date: Mon Aug 19 14:25:21 2019 -0700
HBASE-22882 TestFlushSnapshotFromClient#testConcurrentSnapshottingAttempts
is flakey (was written flakey)
Addendum; just remove the test altogether
{code}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.2.1, 2.1.6, 1.3.6, 1.4.11, 2.0.7
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909963#comment-16909963
]
Zheng Hu commented on HBASE-22810:
--
Fine. Sorry , I think I forgot this issue before, Let me dig this .
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909956#comment-16909956
]
Duo Zhang commented on HBASE-22810:
---
OK, seems it also breaks the tests on branch-2.1...
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2.1/lastSuccessfulBuild/artifact/dashboard.html
Could you please quickly identify whether this is a test issue or not? If not,
we need to sink all the pending RCs and prepare new ones.
Thanks. [~openinx]
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908631#comment-16908631
]
Duo Zhang commented on HBASE-22810:
---
It also breaks the test on master.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908626#comment-16908626
]
Zheng Hu commented on HBASE-22810:
--
OK, Thanks [~apurtell], Let me take a look, will provide the addendum soon if
necessary.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908554#comment-16908554
]
Hudson commented on HBASE-22810:
Results for branch branch-2.2
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/517//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908431#comment-16908431
]
Hudson commented on HBASE-22810:
Results for branch branch-2
[build #2169 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2169//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908347#comment-16908347
]
Hudson commented on HBASE-22810:
Results for branch branch-1.3
[build #920 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/920//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908345#comment-16908345
]
Hudson commented on HBASE-22810:
Results for branch branch-1.4
[build #959 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/959//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908234#comment-16908234
]
Hudson commented on HBASE-22810:
Results for branch branch-2.0
[build #1867 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1867//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908225#comment-16908225
]
Hudson commented on HBASE-22810:
Results for branch branch-2.1
[build #1486 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1486//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908058#comment-16908058
]
Hudson commented on HBASE-22810:
Results for branch master
[build #1335 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1335/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1335//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1335//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1335//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907988#comment-16907988
]
Hudson commented on HBASE-22810:
Results for branch branch-1
[build #1009 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1009//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22810) Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
[
https://issues.apache.org/jira/browse/HBASE-22810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907867#comment-16907867
]
Hudson commented on HBASE-22810:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #618 (See
[https://builds.apache.org/job/HBase-1.3-IT/618/])
HBASE-22810 Initialize an separate ThreadPoolExecutor for (openinx: rev
b97621b30216dc39ad605e06738a070a69a47da3)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> Initialize an separate ThreadPoolExecutor for taking/restoring snapshot
>
>
> Key: HBASE-22810
> URL: https://issues.apache.org/jira/browse/HBASE-22810
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6, 1.3.6, 1.4.11
>
>
> In EventType class, we have the following definition, means taking snapshot
> & restoring snapshot are use the MASTER_TABLE_OPERATIONS Executor now.
> {code}
> /**
>* Messages originating from Client to Master.
>* C_M_SNAPSHOT_TABLE
>* Client asking Master to snapshot an offline table.
>*/
> C_M_SNAPSHOT_TABLE(48, ExecutorType.MASTER_TABLE_OPERATIONS),
> /**
>* Messages originating from Client to Master.
>* C_M_RESTORE_SNAPSHOT
>* Client asking Master to restore a snapshot.
>*/
> C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_TABLE_OPERATIONS),
> {code}
> But when I checked the MASTER_TABLE_OPERATIONS thread pool initialization, I
> see :
> {code}
> private void startServiceThreads() throws IOException{
>// ... some other code initializing
>// We depend on there being only one instance of this executor running
>// at a time. To do concurrency, would need fencing of enable/disable of
>// tables.
>// Any time changing this maxThreads to > 1, pls see the comment at
>// AccessController#postCompletedCreateTableAction
>
> this.executorService.startExecutorService(ExecutorType.MASTER_TABLE_OPERATIONS,
> 1);
>startProcedureExecutor();
> {code}
> That's to say, for CPs enable or disable table sequencely, we will create
> a ThreadPoolExecutor with threadPoolSize=1. Then we actually cann't
> accomplish the snapshoting concurrence even if they are total difference
> tables, says if there are two table snapshoting request, and the Table A cost
> 5min for snapshoting, then the Table B need to wait 5min and once Table A
> finish its snapshot , then Table B will start the snapshot.
> While we've setting the snapshot timeout, so it will be easy to timeout for
> table B snapshoting . Actually, we can create a separate thead pool for
> snapshot operations only.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
