[jira] [Commented] (BEAM-6058) Support flink config directory for flink runner.
[ https://issues.apache.org/jira/browse/BEAM-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689270#comment-16689270 ] Maximilian Michels commented on BEAM-6058: -- AFAIK _all_ job-related configuration is assembled upon JobGraph creation. The flink-conf.yaml on the cluster is only relevant for configuring the cluster itself and for reading environment information (e.g. default parallelism). I agree that this distinction is not always clear. [~thw] [~angoenka] How about adding the config to the Jobserver parameters? That would make the configuration explicit and avoid confusion with where the config is located or loaded. We clearly need a way to specify the config and I'm not sure whether the environment variable is the way to go because that is primarily a way to set it from the bash scripts of the Flink CLI. > Support flink config directory for flink runner. > > > Key: BEAM-6058 > URL: https://issues.apache.org/jira/browse/BEAM-6058 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Give user option to use flink configuration from a config directory. > Earlier we only used jars to submit flink jobs which by default picks the > cluster flink properties. > Portability uses remote environment to submit flink job. In Remote > execution, flink uses the client config instead of the cluster config. This > require us to submit the custom flink properties via flink client. > Suggestion is to read the flink config from a flink configuration folder and > set them to the environment. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6058) Support flink config directory for flink runner.
[ https://issues.apache.org/jira/browse/BEAM-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688638#comment-16688638 ] Ankur Goenka commented on BEAM-6058: Also, from the code I observed that the jar based job submission workflow picks the right properties from the deployment. > Support flink config directory for flink runner. > > > Key: BEAM-6058 > URL: https://issues.apache.org/jira/browse/BEAM-6058 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Give user option to use flink configuration from a config directory. > Earlier we only used jars to submit flink jobs which by default picks the > cluster flink properties. > Portability uses remote environment to submit flink job. In Remote > execution, flink uses the client config instead of the cluster config. This > require us to submit the custom flink properties via flink client. > Suggestion is to read the flink config from a flink configuration folder and > set them to the environment. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6058) Support flink config directory for flink runner.
[ https://issues.apache.org/jira/browse/BEAM-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688618#comment-16688618 ] Ankur Goenka commented on BEAM-6058: I agree, that runtime properties should be managed by deployment. However, for the remote execution environment, many of the properties are taken from the job itself and are not read from the cluster config. This might be a bug in Flink but this is the only work around that I can see for now. Example property taskmanager.runtime.large-record-handler > Support flink config directory for flink runner. > > > Key: BEAM-6058 > URL: https://issues.apache.org/jira/browse/BEAM-6058 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Give user option to use flink configuration from a config directory. > Earlier we only used jars to submit flink jobs which by default picks the > cluster flink properties. > Portability uses remote environment to submit flink job. In Remote > execution, flink uses the client config instead of the cluster config. This > require us to submit the custom flink properties via flink client. > Suggestion is to read the flink config from a flink configuration folder and > set them to the environment. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6058) Support flink config directory for flink runner.
[ https://issues.apache.org/jira/browse/BEAM-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688521#comment-16688521 ] Maximilian Michels commented on BEAM-6058: -- I agree, if at all, the file should be shipped. Using the local file system doesn't make sense. > Support flink config directory for flink runner. > > > Key: BEAM-6058 > URL: https://issues.apache.org/jira/browse/BEAM-6058 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Give user option to use flink configuration from a config directory. > Earlier we only used jars to submit flink jobs which by default picks the > cluster flink properties. > Portability uses remote environment to submit flink job. In Remote > execution, flink uses the client config instead of the cluster config. This > require us to submit the custom flink properties via flink client. > Suggestion is to read the flink config from a flink configuration folder and > set them to the environment. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6058) Support flink config directory for flink runner.
[ https://issues.apache.org/jira/browse/BEAM-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688253#comment-16688253 ] Thomas Weise commented on BEAM-6058: I don't see why old Flink runner and job server server should behave differently. Both see the same environment and can load any configuration present. Flink has an environment variable to identify the config dir: [https://github.com/apache/flink/blob/1a9675d54fda7c6d7c519935dde05f47eb449401/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L66] That could be set in either case and should cause the desired configuration to be loaded. Passing a local file reference as pipeline option seems odd, since it implies the user having knowledge about the job server environment. > Support flink config directory for flink runner. > > > Key: BEAM-6058 > URL: https://issues.apache.org/jira/browse/BEAM-6058 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Give user option to use flink configuration from a config directory. > Earlier we only used jars to submit flink jobs which by default picks the > cluster flink properties. > Portability uses remote environment to submit flink job. In Remote > execution, flink uses the client config instead of the cluster config. This > require us to submit the custom flink properties via flink client. > Suggestion is to read the flink config from a flink configuration folder and > set them to the environment. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
