[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493402#comment-16493402 ] ASF GitHub Bot commented on FLINK-9326: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/6041 > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > Fix For: 1.6.0, 1.5.1 > > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493187#comment-16493187 ] Chesnay Schepler commented on FLINK-9326: - 1.5: cbf2c8f785b6a8b620d9aeeb702fdb81b7c8bbfa > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486997#comment-16486997 ] ASF GitHub Bot commented on FLINK-9326: --- Github user yanghua commented on a diff in the pull request: https://github.com/apache/flink/pull/6041#discussion_r190187802 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/LocalStreamEnvironment.java --- @@ -103,9 +103,16 @@ public JobExecutionResult execute(String jobName) throws Exception { configuration.setInteger(RestOptions.PORT, 0); } + int numSlotsPerTaskManager; + if (configuration.contains(TaskManagerOptions.NUM_TASK_SLOTS)) { --- End diff -- cc @zentol > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483779#comment-16483779 ] ASF GitHub Bot commented on FLINK-9326: --- Github user yanghua commented on a diff in the pull request: https://github.com/apache/flink/pull/6041#discussion_r189856924 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/LocalStreamEnvironment.java --- @@ -103,9 +103,16 @@ public JobExecutionResult execute(String jobName) throws Exception { configuration.setInteger(RestOptions.PORT, 0); } + int numSlotsPerTaskManager; + if (configuration.contains(TaskManagerOptions.NUM_TASK_SLOTS)) { --- End diff -- @zentol does it need a new commit? > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483716#comment-16483716 ] ASF GitHub Bot commented on FLINK-9326: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6041#discussion_r189836110 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/LocalStreamEnvironment.java --- @@ -103,9 +103,16 @@ public JobExecutionResult execute(String jobName) throws Exception { configuration.setInteger(RestOptions.PORT, 0); } + int numSlotsPerTaskManager; + if (configuration.contains(TaskManagerOptions.NUM_TASK_SLOTS)) { --- End diff -- this block can be simplified to: ``` numSlotsPerTaskManager = configuration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, jobGraph.getMaximumParallelism()); ``` > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483653#comment-16483653 ] ASF GitHub Bot commented on FLINK-9326: --- Github user yanghua commented on the issue: https://github.com/apache/flink/pull/6041 cc @zentol @tzulitai if you have time, please review this PR, thanks~ > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480307#comment-16480307 ] ASF GitHub Bot commented on FLINK-9326: --- Github user yanghua commented on the issue: https://github.com/apache/flink/pull/6041 cc @tillrohrmann > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480228#comment-16480228 ] ASF GitHub Bot commented on FLINK-9326: --- GitHub user yanghua opened a pull request: https://github.com/apache/flink/pull/6041 [FLINK-9326] TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode ## What is the purpose of the change *This pull request fixed the num slot config value override error.* ## Brief change log - *fixed the num slot config value override error.* ## Verifying this change This change is already covered by existing tests*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) You can merge this pull request into a Git repository by running: $ git pull https://github.com/yanghua/flink FLINK-9326 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6041.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6041 commit 16fb81beaa4983a04e9929267ced36a3121f8188 Author: yanghuaDate: 2018-05-18T06:36:19Z [FLINK-9326] TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470243#comment-16470243 ] Till Rohrmann commented on FLINK-9326: -- You're right [~Sam_D]. That way the user experience out of the box is a bit better, because it won't fail. I agree, though, that this is not very intuitive and confusing. Maybe we could do it that we use the {{TaskManagerOptions#NUM_TASK_SLOTS}} if explicitly set and otherwise we default to the {{jobGraph.getMaximumParallelism}}. > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Assignee: vinoyang >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9326) TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode
[ https://issues.apache.org/jira/browse/FLINK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469326#comment-16469326 ] Samuel Doyle commented on FLINK-9326: - I'm guessing this is because it appears whatever number of slots that was set in the configuration is later overridden ? In the *LocalStreamEnvironment* *execute* method {code:java} ... // add (and override) the settings with what the user defined configuration.addAll(this.configuration); if (!configuration.contains(RestOptions.PORT)) { configuration.setInteger(RestOptions.PORT, 0); } MiniClusterConfiguration cfg = new MiniClusterConfiguration.Builder() .setConfiguration(configuration) .setNumSlotsPerTaskManager(jobGraph.getMaximumParallelism()) // <-- HERE overrides the slots .build(); ...{code} > TaskManagerOptions.NUM_TASK_SLOTS does not work for local/embedded mode > --- > > Key: FLINK-9326 > URL: https://issues.apache.org/jira/browse/FLINK-9326 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.5.0 > Environment: Linux 64bit > Flink branch release-1.5 >Reporter: Samuel Doyle >Priority: Major > > When attempting to set the number of task slots via the api such ash > {code:java} > configuration = new Configuration(); > configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 16); > configuration.setInteger(CoreOptions.DEFAULT_PARALLELISM, 1); > {code} > I will always end up with the default slot setting based on the number of > cores I have where my standalone instance is running, it doesn't matter what > I set the the NUM_TASK_SLOTS to, it has no effect -- This message was sent by Atlassian JIRA (v7.6.3#76005)