[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580783#comment-16580783 ] ASF GitHub Bot commented on FLINK-10123: asfgit closed pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/_includes/generated/akka_configuration.html b/docs/_includes/generated/akka_configuration.html index 352c656488e..f5a2a5d77f9 100644 --- a/docs/_includes/generated/akka_configuration.html +++ b/docs/_includes/generated/akka_configuration.html @@ -12,11 +12,41 @@ "10 s" Timeout used for all futures and blocking Akka calls. If Flink fails due to timeouts then you should try to increase this value. Timeouts can be caused by slow machines or a congested network. The timeout value requires a time-unit specifier (ms/s/min/h/d). + +akka.client-socket-worker-pool.pool-size-factor +1.0 +The pool size factor is used to determine thread pool size using the following formula: ceil(available processors * factor). Resulting size is then bounded by the pool-size-min and pool-size-max values. + + +akka.client-socket-worker-pool.pool-size-max +2 +Max number of threads to cap factor-based number to. + + +akka.client-socket-worker-pool.pool-size-min +1 +Min number of threads to cap factor-based number to. + akka.client.timeout "60 s" Timeout for all blocking calls on the client side. + +akka.fork-join-executor.parallelism-factor +2.0 +The parallelism factor is used to determine thread pool size using the following formula: ceil(available processors * factor). Resulting size is then bounded by the parallelism-min and parallelism-max values. + + +akka.fork-join-executor.parallelism-max +64 +Max number of threads to cap factor-based parallelism number to. + + +akka.fork-join-executor.parallelism-min +8 +Min number of threads to cap factor-based parallelism number to. + akka.framesize "10485760b" @@ -42,6 +72,21 @@ 50 Milliseconds a gate should be closed for after a remote connection was disconnected. + +akka.server-socket-worker-pool.pool-size-factor +1.0 +The pool size factor is used to determine thread pool size using the following formula: ceil(available processors * factor). Resulting size is then bounded by the pool-size-min and pool-size-max values. + + +akka.server-socket-worker-pool.pool-size-max +2 +Max number of threads to cap factor-based number to. + + +akka.server-socket-worker-pool.pool-size-min +1 +Min number of threads to cap factor-based number to. + akka.ssl.enabled true diff --git a/flink-core/src/main/java/org/apache/flink/configuration/AkkaOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/AkkaOptions.java index 43c7876fbd8..02234b9d259 100644 --- a/flink-core/src/main/java/org/apache/flink/configuration/AkkaOptions.java +++ b/flink-core/src/main/java/org/apache/flink/configuration/AkkaOptions.java @@ -194,4 +194,81 @@ .key("akka.retry-gate-closed-for") .defaultValue(50L) .withDescription("Milliseconds a gate should be closed for after a remote connection was disconnected."); + + // == + // Configurations for fork-join-executor. + // == + + public static final ConfigOption FORK_JOIN_EXECUTOR_PARALLELISM_FACTOR = ConfigOptions + .key("akka.fork-join-executor.parallelism-factor") + .defaultValue(2.0) + .withDescription(Description.builder() + .text("The parallelism factor is used to determine thread pool size using the" + + " following formula: ceil(available processors * factor). Resulting size" + + " is then bounded by the parallelism-min and parallelism-max values." + ).build()); + + public
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579568#comment-16579568 ] ASF GitHub Bot commented on FLINK-10123: tillrohrmann commented on issue #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#issuecomment-412817474 Thanks @yanghua. Fixed the PR. Let's see what Travis says this time :-) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579304#comment-16579304 ] ASF GitHub Bot commented on FLINK-10123: yanghua commented on issue #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#issuecomment-412765241 checkstyle error : ``` 08:33:28.933 [ERROR] src/test/java/org/apache/flink/test/recovery/JobManagerHACheckpointRecoveryITCase.java:[22] (imports) ImportOrder: Import org.apache.flink.api.common.functions.RichFlatMapFunction appears after other imports that it should precede ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577988#comment-16577988 ] ASF GitHub Bot commented on FLINK-10123: tillrohrmann commented on issue #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#issuecomment-412454181 Thanks for the review @zentol and @yanghua. Waiting for Travis to become stable again and after having green light merging this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577911#comment-16577911 ] ASF GitHub Bot commented on FLINK-10123: tillrohrmann commented on a change in pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#discussion_r209512211 ## File path: flink-runtime/src/main/scala/akka/actor/RobustActorSystem.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package akka.actor Review comment: We need to use the same package as the Akka classes since `ActorSystemImpl` is package private `private[akka]`. It is a bit ugly but it should work. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577420#comment-16577420 ] ASF GitHub Bot commented on FLINK-10123: yanghua commented on a change in pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#discussion_r209443308 ## File path: flink-runtime/src/main/scala/akka/actor/RobustActorSystem.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package akka.actor Review comment: @zentol here we should use this package. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577419#comment-16577419 ] ASF GitHub Bot commented on FLINK-10123: zentol commented on a change in pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#discussion_r209443286 ## File path: flink-runtime/src/main/scala/akka/actor/RobustActorSystem.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package akka.actor Review comment: was this copied from akka? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577019#comment-16577019 ] ASF GitHub Bot commented on FLINK-10123: yanghua commented on issue #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#issuecomment-412244963 Till, scala checkstyle error : ``` error file=/home/travis/build/apache/flink/flink-runtime/src/main/scala/akka/actor/RobustActorSystem.scala message=File line length exceeds 100 characters line=88 ``` other travis build task also failed, because of some connection timeout. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576554#comment-16576554 ] ASF GitHub Bot commented on FLINK-10123: yanghua commented on issue #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539#issuecomment-412139875 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10123) Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client
[ https://issues.apache.org/jira/browse/FLINK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576433#comment-16576433 ] ASF GitHub Bot commented on FLINK-10123: tillrohrmann opened a new pull request #6539: [FLINK-10123] Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client URL: https://github.com/apache/flink/pull/6539 ## What is the purpose of the change Using the ExecutorThreadFactory hardens the system because it uses the FatalExitExceptionHandler as UncaughtExceptionHandler which terminates the JVM in case of an exception. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use ExecutorThreadFactory instead of DefaultThreadFactory in RestServer/Client > -- > > Key: FLINK-10123 > URL: https://issues.apache.org/jira/browse/FLINK-10123 > Project: Flink > Issue Type: Improvement > Components: REST >Affects Versions: 1.6.0, 1.7.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > Instead of using the {{DefaultThreadFactory}} in the > {{RestServerEndpoint}}/{{RestClient}} we should use the > {{ExecutorThreadFactory}} because it uses the {{FatalExitExceptionHandler}} > per default as the uncaught exception handler. This should guard against > uncaught exceptions by simply terminating the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)