[
https://issues.apache.org/jira/browse/FLINK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504767#comment-17504767
]
Fil Karnicki edited comment on FLINK-26570 at 3/11/22, 7:48 AM:
----------------------------------------------------------------
Hi [~trohrmann] sounds like a good idea to me.
I recently opened a pull request, which among other things, uses job entrypoint
args in the creation of the streaming execution environment for statefun jobs.
This was so that jobs running from an uber jar could set various settings (like
checkpointing interval) per job [1]. I wonder if we should include system and
env properties there too (either before both the env and
StatefulFunctionsConfig get created, or maybe after the env gets created, but
before StatefulFunctionsConfig does). Having those settings be part of
StatefulFunctionsConfig's globalConfig would automatically allow for the string
interpolation inside RemodeModule#bindComponent.
I'm not so familiar with the project as to be able to foresee if loading
system+env properties into StatefulFunctionsConfig isn't going to cause
problems for existing users, but I hope Igal is going shed some light on this
as part of the PR review.
[https://github.com/apache/flink-statefun/pull/307/files#diff-6249db934ffac5890c899612fc2daa31e7e85768faee779f1ccf17ddbbee23f7]
[1] - I know that the preferred way of launching statefun is via docker, but we
need to make do with a shared cloudera cluster.
was (Author: JIRAUSER284249):
Hi [~trohrmann] sounds like a good idea to me.
I recently opened a pull request to use job entrypoint args in the creation of
the streaming execution environment for statefun jobs. This was so that jobs
running from an uber jar could set various settings (like checkpointing
interval) per job [1]. I wonder if we should include system and env properties
there too (either before both the env and StatefulFunctionsConfig get created,
or maybe after the env gets created, but before StatefulFunctionsConfig does).
Having those settings be part of StatefulFunctionsConfig's globalConfig would
automatically allow for the string interpolation inside
RemodeModule#bindComponent.
I'm not so familiar with the project as to be able to foresee if loading
system+env properties into StatefulFunctionsConfig isn't going to cause
problems for existing users, but I hope Igal is going shed some light on this
as part of the PR review.
[https://github.com/apache/flink-statefun/pull/307/files#diff-6249db934ffac5890c899612fc2daa31e7e85768faee779f1ccf17ddbbee23f7]
[1] - I know that the preferred way of launching statefun is via docker, but we
need to make do with a shared cloudera cluster.
> Remote module configuration interpolation
> -----------------------------------------
>
> Key: FLINK-26570
> URL: https://issues.apache.org/jira/browse/FLINK-26570
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
> Reporter: Fil Karnicki
> Priority: Major
>
> Add the ability for users to provide placeholders in module.yaml, e.g.
> {code:java}
> kind: com.foo.bar/test
> spec:
> something: ${REPLACE_ME}
> transport:
> password: ${REPLACE_ME_WITH_A_SECRET}
> array:
> - ${REPLACE_ME}
> - sthElse {code}
> These placeholders would be resolved in
> org.apache.flink.statefun.flink.core.jsonmodule.RemoteModule#bindComponent
> using
> {code:java}
> ParameterTool.fromSystemProperties().mergeWith(ParameterTool.fromMap(globalConfiguration)
> {code}
> by traversing the ComponentJsonObject.specJsonNode() and replacing values
> that contain placeholders with values from the combined system+globalConfig
> map
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
