[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655112#comment-17655112
]
Arseniy Tashoyan commented on FLINK-21383:
--
One more problem caused by this issue: it is impossible to use environment
variables in _flink-conf.yaml_
For example:
{code:yaml}
metrics.reporter.custom_reporter.username: ${reporter_username}
metrics.reporter.custom_reporter.password: ${reporter_password}
{code}
The script _docker-entrypoints.sh_ makes an attempt to rewrite
_flink-conf.yaml_ with resolved environment variables:
{code}
Unable to find source-code formatter for language: shell. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlenvsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp"
"${CONF_FILE}"
{code}
This attempt fails because the ConfigMap is mounted readonly:
{code:none}
/docker-entrypoint.sh: line 89: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
{code}
Normally, Flink should resolve environment variables when reading values from
{_}flink-conf.yaml{_}. A proper way could be to use a template engine like
Apache Velocity.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.6, 1.12.7, 1.13.5, 1.14.3
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427119#comment-17427119
]
David Morávek commented on FLINK-21383:
---
I guess we should either do that or have a special entrypoint that wouldn't
print these "warnings / errors" (we are confident in the native case that we've
provided the correct values).
It probably depends on whether this dynamic log4j configuration thing should be
a concern.
Just for reference, this issue has been introduced in 1.9.6 k8s version. [1]
[1] https://github.com/kubernetes/kubernetes/issues/62099
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427110#comment-17427110
]
Yang Wang commented on FLINK-21383:
---
I agree with you that Flink cluster may not have up-to-date flink-conf.yaml if
not used carefully. But compared to setting the environment
{{FLINK_PROPERTIES}}, using the ConfigMap is an easier way to have plenty of
user-specified Flink configurations.
BTW, do you think we also need to create a mutable copy of the mounted
ConfigMap for native K8s deployment?
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427025#comment-17427025
]
David Morávek commented on FLINK-21383:
---
[~wangyang0918] IMO mounting config-maps this way is a bad practice anyway,
because change to the config-map doesn't restart pods, so you're not actually
sure if your configuration is up-to-date. What we usually did was adding a
unique suffix to the config-map name, which has been changed with each update
(deployments also need to be updated accordingly, but this kind of goes
hand-in-hand with the declarative nature of the Kubernetes API).
I agree that log4j configuration is a bit different story as we log4j supports
dynamic re-loading of the configuration. I'm not sure if that's something we
officially want to support (by documenting it) though, because than it may have
other side-effects, like not having up-to-date flink-conf.yaml if not used
carefully.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427013#comment-17427013
]
Till Rohrmann commented on FLINK-21383:
---
Maybe we could/should split the log4j configuration from the Flink
configuration. The Flink configuration does not support dynamic updates anyway.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426473#comment-17426473
]
Yang Wang commented on FLINK-21383:
---
[~dmvk] I am afraid this solution might also have some side effects. For
example, if we change the flink configuration ConfigMap, they will not be
updated automatically. A possible use case is changing the log level of some
specific package.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426252#comment-17426252
]
David Morávek commented on FLINK-21383:
---
Possible workaround is to tweak the Kubernetes deployment manifest to create a
mutable copy of the mounted ConfigMap in InitContainer. I think this may
actually not be a workaround but the correct approach. Example:
{code:java}
kind: Deployment
metadata:
name: flink-taskmanager
spec:
replicas: 2
selector:
matchLabels:
app: flink
component: taskmanager
template:
metadata:
labels:
app: flink
component: taskmanager
spec:
containers:
- name: taskmanager
image: apache/flink:1.14.0-scala_2.12-java8
env:
- name: JOB_MANAGER_RPC_ADDRESS
value: flink-jobmanager
- name: TASK_MANAGER_NUMBER_OF_TASK_SLOTS
value: "2"
args: ["taskmanager"]
ports:
- containerPort: 6122
name: rpc
- containerPort: 6125
name: query-state
livenessProbe:
tcpSocket:
port: 6122
initialDelaySeconds: 30
periodSeconds: 60
volumeMounts:
- name: flink-config-volume
mountPath: /opt/flink/conf
- name: job-artifacts-volume
mountPath: /opt/flink/usrlib
securityContext:
runAsUser:
initContainers:
- name: init-conf-directory
image: busybox:stable
command: ['sh', '-c', 'cp -L /opt/flink/conf-readonly/* /opt/flink/conf
&& chown -R : /opt/flink/conf']
volumeMounts:
- name: flink-config-volume
mountPath: /opt/flink/conf
- name: flink-config-readonly-volume
mountPath: /opt/flink/conf-readonly
volumes:
- name: flink-config-volume
emptyDir: {}
- name: flink-config-readonly-volume
configMap:
name: flink-config
items:
- key: flink-conf.yaml
path: flink-conf.yaml
- key: log4j-console.properties
path: log4j-console.properties
- name: job-artifacts-volume
hostPath:
path: /opt/usrlib
{code}
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408691#comment-17408691
]
Robert Metzger commented on FLINK-21383:
+1 for fixing this issue, as it might confuse users
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350207#comment-17350207
]
Yang Wang commented on FLINK-21383:
---
The error of {{sed}} does not affect using the native K8s application mode. It
is just a little annoying.
For the second error "bash: kubernetes-jobmanager.sh: command not found", I
believe the Flink client you are using is not same with the image version. For
example, the client is release-1.13 and the image is release-1.12.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350055#comment-17350055
]
HYUNHOO KWON commented on FLINK-21383:
--
Any updates?
On native k8s application mode, I found the following error in the console logs.
{code:java}
sed: couldn't open temporary file /opt/flink/conf/sedFktect: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sed63dRsw: Read-only file
system
/docker-entrypoint.sh: line 75: /opt/flink/conf/flink-conf.yaml: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedrIpLLv: Read-only file
system
/docker-entrypoint.sh: line 90: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
bash: kubernetes-jobmanager.sh: command not found
{code}
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335905#comment-17335905
]
Flink Jira Bot commented on FLINK-21383:
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: stale-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327332#comment-17327332
]
Flink Jira Bot commented on FLINK-21383:
This major issue is unassigned and itself and all of its Sub-Tasks have not
been updated for 30 days. So, it has been labeled "stale-major". If this ticket
is indeed "major", please either assign yourself or give an update. Afterwards,
please remove the label. In 7 days the issue will be deprioritized.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: stale-major, usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297920#comment-17297920
]
Peng Zhang commented on FLINK-21383:
It would be nice to address this issue. For us, we prefer to use
`flink-config.yaml` to configure all possible flink properties since it is more
visible and easier to maintain and update.
Now, we are trying to set up Flink HA on K8S. it is a bit confusing whether we
could configure all parameters via `flink-config.yaml`. Or, what would go wrong
if we configure Flink properties in flink-config.yaml, but not via the docker
image command line? Thanks!
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286450#comment-17286450
]
Till Rohrmann commented on FLINK-21383:
---
The failure messages are unfortunately not very descriptive. Maybe we can
improve this a bit.
Yes making the configuration configurable within the Flink process should solve
the process. One way would be to use the dynamic properties for that. A related
idea is to implement
[FLIP-161|https://cwiki.apache.org/confluence/display/FLINK/FLIP-161%3A+Configuration+through+envrionment+variables?focusedCommentId=173083588].
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286253#comment-17286253
]
Yang Wang commented on FLINK-21383:
---
ConfigMap volume is always read-only. So updating the flink-conf.yaml in
JobManager/TaskManager pods will not take effect. We could find the following
error in the console logs.
{code:java}
/docker-entrypoint.sh: line 76: /opt/flink/conf/flink-conf.yaml: Permission
denied
sed: couldn't open temporary file /opt/flink/conf/sedaGciYX: Read-only file
system
/docker-entrypoint.sh: line 76: /opt/flink/conf/flink-conf.yaml: Permission
denied
/docker-entrypoint.sh: line 95: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
{code}
I am not sure whether the users will set the {{FLINK_PROPERTIES}} and
{{JOB_MANAGER_RPC_ADDRESS}} to override the config options in Kubernetes world.
Personally, I prefer to do it via {{-D}} dynamic properties. If we make both
{{jobmanager.sh}} and {{standalone-job.sh}} could support dynamic properties,
then maybe we could give up updating the flink-conf.yaml when it is read-only.
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls
[
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285323#comment-17285323
]
Till Rohrmann commented on FLINK-21383:
---
cc [~chesnay], [~fly_in_gis].
> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: usability
>
> Flink's Docker image does not play well together with ConfigMap based
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}},
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}}
> scripts don't support to pass in dynamic configuration properties into the
> processes.
> I think the problem is that our assumption that we can modify the
> {{flink-conf.yaml}} does not always hold true. If we updated the final
> configuration from within the Flink process (dynamic properties and env
> variables), then this problem could be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
