[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Flink Jira Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336826#comment-17336826
 ] 

Flink Jira Bot commented on FLINK-5253:
---

This issue was labeled "stale-major" 7 ago and has not received any updates so 
it is being deprioritized. If this ticket is actually Major, please raise the 
priority and ask a committer to assign you the issue or revive the public 
discussion.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>Priority: Major
>  Labels: flip-6, pull-request-available, stale-major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Flink Jira Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328859#comment-17328859
 ] 

Flink Jira Bot commented on FLINK-5253:
---

This major issue is unassigned and itself and all of its Sub-Tasks have not 
been updated for 30 days. So, it has been labeled "stale-major". If this ticket 
is indeed "major", please either assign yourself or give an update. Afterwards, 
please remove the label. In 7 days the issue will be deprioritized.

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>Priority: Major
>  Labels: flip-6, pull-request-available, stale-major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134818#comment-16134818
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/3356
  
I think this PR is still relevant but needs some rebasing onto the latest 
master @zentol. What we mainly have to do is to use the methods introduced with 
#4415 to load a configuration with dynamic properties parsed from the command 
line.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134521#comment-16134521
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3356
  
@tillrohrmann Is this issue still valid after FLINK-7269? (#4415)


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100220#comment-16100220
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/3356
  
I think the actual problem is that we use 
`GlobalConfiguration.loadConfiguration` at far too many places where we should 
rather pass in a `Configuration` object.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Mariusz Wojakowski (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080495#comment-16080495
 ] 

Mariusz Wojakowski commented on FLINK-5253:
---

Thanks for the comments! I'll take a look and update the PR.

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075050#comment-16075050
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/3356
  
@tillrohrmann the `dynamicProperties` static field is unfortunate, yes.   
To eliminate it, we must evaluate the use of the zero-arg 
`GlobalConfiguration.loadConfiguration()` that is used in many places 
(`FileInputFormat`, `HadoopUtils`).   The field ensures that all those sites 
see the same configuration.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074914#comment-16074914
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3356#discussion_r125669802
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -319,13 +306,7 @@ public AbstractYarnClusterDescriptor 
createDescriptor(String defaultApplicationN
yarnClusterDescriptor.setTaskManagerSlots(slots);
}
 
-   String[] dynamicProperties = null;
-   if (cmd.hasOption(DYNAMIC_PROPERTIES.getOpt())) {
-   dynamicProperties = 
cmd.getOptionValues(DYNAMIC_PROPERTIES.getOpt());
-   }
-   String dynamicPropertiesEncoded = 
StringUtils.join(dynamicProperties, YARN_DYNAMIC_PROPERTIES_SEPARATOR);
-
-   
yarnClusterDescriptor.setDynamicPropertiesEncoded(dynamicPropertiesEncoded);
+   
GlobalConfiguration.setDynamicProperties(retrieveDynamicProperties(cmd, 
DYNAMIC_PROPERTIES));
--- End diff --

Can we get around setting a static field in `GlobalConfiguration`? Why not 
parsing the dynamic properties and then passing them to the `ClusterDescriptor` 
at creation time. I guess it would also make sense to do the configuration 
loading outside of the `ClusterDescriptor`.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074913#comment-16074913
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3356#discussion_r125670830
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -711,8 +701,18 @@ public FileVisitResult 
preVisitDirectory(java.nio.file.Path dir, BasicFileAttrib
LocalResource flinkConf = 
Records.newRecord(LocalResource.class);
Path remotePathJar =
Utils.setupLocalResource(fs, appId.toString(), 
flinkJarPath, appMasterJar, fs.getHomeDirectory());
-   Path remotePathConf =
-   Utils.setupLocalResource(fs, appId.toString(), 
flinkConfigurationPath, flinkConf, fs.getHomeDirectory());
+
+   final org.apache.flink.configuration.Configuration 
dynamicProperties = GlobalConfiguration.getDynamicProperties();
--- End diff --

Can't we pass this configuration into the `ClusterDescriptor` differently? 
What about reading it when creating the descriptor and then passing it as a 
constructor parameter? Even better, we could read the global configuration, 
then parse the dynamic properties, adding them to the read config and then 
creating the descriptor.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074915#comment-16074915
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3356#discussion_r125670308
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnCLI.java ---
@@ -149,13 +148,7 @@ public YarnClusterDescriptorV2 createDescriptor(String 
defaultApplicationName, C
yarnClusterDescriptor.setJobManagerMemory(jmMemory);
}
 
-   String[] dynamicProperties = null;
-   if (cmd.hasOption(DYNAMIC_PROPERTIES.getOpt())) {
-   dynamicProperties = 
cmd.getOptionValues(DYNAMIC_PROPERTIES.getOpt());
-   }
-   String dynamicPropertiesEncoded = 
StringUtils.join(dynamicProperties, YARN_DYNAMIC_PROPERTIES_SEPARATOR);
-
-   
yarnClusterDescriptor.setDynamicPropertiesEncoded(dynamicPropertiesEncoded);
+   
GlobalConfiguration.setDynamicProperties(retrieveDynamicProperties(cmd, 
DYNAMIC_PROPERTIES));
--- End diff --

Can we avoid the static dynamic properties field?


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893108#comment-15893108
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user mariusz89016 commented on the issue:

https://github.com/apache/flink/pull/3356
  
Thanks for the review! 
Currently, dynamic properties are set by 
`GlobalConfiguration.dynamicProperties` instead of saving in cluster descriptor.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889993#comment-15889993
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3356
  
I agree, the CLI should accept dynamic options that are logically added to 
the configuration. But we still do not need the extra dynamic properties 
mechanism when setting up the TaskManager and JobManager via YARN.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888735#comment-15888735
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/3356
  
@StephanEwen I agree that the YARN AM/TM has no need for dynamic properties 
since the entire config file is generated per container.   But there's a case 
to be made that dynamic properties on the CLI makes sense for some cluster 
managers.  Imagine a Kubernetes deployment using a docker image with Flink 
pre-installed & (mostly) pre-configured.   It would make sense to pass the JM 
RPC port (which Kubernetes generates on-the-fly) as a dynamic property, e.g:
```
bin/flink-jobmanager.sh -Djobmanager.rpc.port=$PORT0
```

The mesos jobmanager and taskmanager use this approach for similar reasons.



> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888704#comment-15888704
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/3356
  
Would you mind having a look at `GlobalConfiguration.dynamicProperties`, 
which is set by the main method of various Flink processes based on CLI 
arguments.   The `GlobalConfiguration.loadConfiguration` method (which loads 
the YAML config) treats the dynamic properties as overrides to the config 
entries.This feature is necessary because `loadConfiguration` is used all 
over the place.   

Just bringing it to your attention in case there's some interplay.



> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888694#comment-15888694
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/3356
  
Consider using an instance of `Configuration` in place of 
`Map` to store configuration data.  `Configuration` provides 
handy methods for dealing with typed properties and for merging with other 
collections.


> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873805#comment-15873805
 ] 

ASF GitHub Bot commented on FLINK-5253:
---

GitHub user mariusz89016 opened a pull request:

https://github.com/apache/flink/pull/3356

[FLINK-5253] Remove special treatment of "dynamic properties"

This PR removes special treatment of 'dynamic properties' (aka special way 
of encoding them as environment variable). Instead, these properties are 
appended to _flink-conf.yaml_ file.

Link to the issue: https://issues.apache.org/jira/browse/FLINK-5253

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariusz89016/flink 
flink-5253-dynamic-properties

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3356


commit 1467b86525d4b2ce07ba9dbe42a5fae7fff1bf22
Author: Mariusz Wojakowski 
Date:   2017-02-18T14:15:07Z

[FLINK-5253] Remove special treatment of "dynamic properties"




> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866673#comment-15866673
 ] 

Stephan Ewen commented on FLINK-5253:
-

That means we can drop it for all modes that do that. Not sure if the YARN 
session does that as well.

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866672#comment-15866672
 ] 

Stephan Ewen commented on FLINK-5253:
-

When the {{flink-conf.yaml}} is anyways generated, we can simply put those 
options in there.

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Mariusz Wojakowski (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866340#comment-15866340
 ] 

Mariusz Wojakowski commented on FLINK-5253:
---

I have 2 questions:
* should we store these dynamic properties in _flink-conf.yaml_ and completely 
remove that information from cluster descriptor? (Correct me if I'm wrong)
* do we want to completely remove that special handling? I’m thinking about 
existing implementation that setup Flink on Yarn (’yarn-sessions’).

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735390#comment-15735390
 ] 

Stephan Ewen commented on FLINK-5253:
-

This refers to the new Flink-on-YARN implementation in FLIP-6.

It should be based on the pull request https://github.com/apache/flink/pull/2935

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5253) Remove special treatment of "dynamic properties"

[Mariusz Wojakowski (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727188#comment-15727188
 ] 

Mariusz Wojakowski commented on FLINK-5253:
---

I'd like to implement that (if you don't mind).
I'm not experienced developer but it looks feasible for me.

> Remove special treatment of "dynamic properties"
> 
>
> Key: FLINK-5253
> URL: https://issues.apache.org/jira/browse/FLINK-5253
> Project: Flink
>  Issue Type: Sub-task
>  Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>  Labels: flip-6
>
> The YARN client accepts configuration keys as command line parameters.
> Currently these are send to the AppMaster and TaskManager as "dynamic 
> properties", encoded in a special way via environment variables.
> The mechanism is quite fragile. We should simplify it:
>   - The YARN client takes the local {{flink-conf.yaml}} as the base.
>   - It overwrite config entries with command line properties when preparing 
> the configuration to be shipped to YARN container processes (JM / TM)
>   - No additional handling neccessary



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)