> several failures to write into $FLINK_HOME/conf/. I'm working on <https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/858356/> building Flink and flink-kubernetes-operator images for the Wikimedia Foundation, and I found this strange as well. It makes sense in a docker / docker-compose only environment, but in k8s where you have ConfigMap responsible for flink-conf.yaml, and (also logs all going to the console, not FLINK_HOME/log), I'd prefer if the image was not modified by the ENTRYPOINT.
I believe that for flink-kubernetes-operator, the docker-entrypoint.sh <https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/docker-entrypoint.sh> provided by flink-docker is not really needed. It seems to be written more for deployments outside of kubernetes. flink-kubernetes-operator never calls the built in subcommands (e.g. standalone-job), and always runs in 'pass-through' mode, just execing the args passed to it. At WMF we build <https://doc.wikimedia.org/docker-pkg/> our own images, so I'm planning on removing all of the stuff in ENTRYPOINTs that mangles the image. Anything that I might want to keep from docker-entrypoint.sh (like enabling jemoalloc <https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/858356/6/images/flink/Dockerfile.template#73>) I should be able to do in the Dockerfile at image creation time. > want to set an API key as part of the flink-conf.yaml file, but we don't want it to be persisted in Kubernetes or in our version control I personally am still pretty green at k8s, but would using kubernetes Secrets <https://kubernetes.io/docs/concepts/configuration/secret/#use-case-secret-visible-to-one-container-in-a-pod> work for your use case? I know we use them at WMF, but from a quick glance I'm not sure how to combine them in flink-kubernetes-operator's ConfigMap that renders flink-conf.yaml, but I feel like there should be a way. On Wed, Nov 30, 2022 at 4:59 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > Hi Lucas! > > The Flink kubernetes integration itself is responsible for mounting the > configmap and overwriting the entrypoint not the operator. Therefore this > is not something we can easily change from the operator side. However I > think we are looking at the problem from the wrong side and there may be a > solution already :) > > Ideally what you want is ENV replacement in Flink configuration. This is > not something that the Flink community has added yet unfortunately but we > have it on our radar for the operator at least ( > https://issues.apache.org/jira/browse/FLINK-27491). It will probably be > added in the next 1.4.0 version. > > This will be possible from Flink 1.16 which introduced a small feature > that allows us to inject parameters to the kubernetes entrypoints: > https://issues.apache.org/jira/browse/FLINK-29123 > > https://github.com/apache/flink/commit/c37643031dca2e6d4c299c0d704081a8bffece1d > > While it's not implemented in the operator yet, you could try setting the > following config in Flink 1.16.0: > kubernetes.jobmanager.entrypoint.args: -D > datadog.secret.conf=$MY_SECRET_ENV > kubernetes.taskmanager.entrypoint.args: -D > datadog.secret.conf=$MY_SECRET_ENV > > If you use this configuration together with the default native mode in the > operator, it should work I believe. > > Please try and let me know! > Gyula > > On Wed, Nov 30, 2022 at 10:36 PM Lucas Caparelli < > lucas.capare...@gympass.com> wrote: > >> Hello folks, >> >> Not sure if this is the best list for this, sorry if it isn't. I'd >> appreciate some pointers :-) >> >> When using flink-kubernetes-operator [1], docker-entrypoint.sh [2] goes >> through several failures to write into $FLINK_HOME/conf/. We believe this >> is due to this volume being mounted from a ConfigMap, which means it's >> read-only. >> >> This has been reported in the past in GCP's operator, but I was unable to >> find any kind of resolution for it: >> https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/issues/213 >> >> In our use case, we want to set an API key as part of the flink-conf.yaml >> file, but we don't want it to be persisted in Kubernetes or in our version >> control, since it's sensitive data. This API Key is used by Flink to report >> metrics to Datadog [3]. >> >> We have automation in place which allows us to accomplish this by setting >> environment variables pointing to a path in our secret manager, which only >> gets injected during runtime. That part is working fine. >> >> However, we're trying to inject this secret using the FLINK_PROPERTIES >> variable, which is appended [4] to the flink-conf.yaml file in the >> docker-entrypoint script, which fails due to the filesystem where the file >> is being read-only. >> >> We attempted working around this in 2 different ways: >> >> - providing our own .spec.containers[0].command, where we copied over >> /opt/flink to /tmp/flink and set FLINK_HOME=/tmp/flink. This did not work >> because the operator overwrote it and replaced it with its original >> command/args; >> - providing an initContainer sharing the volumes so it could make the >> copy without being overridden by the operator's command/args. This did not >> work because the initContainer present in the spec never makes it to the >> resulting Deployment, it seems the operator ignores it. >> >> We have some questions: >> >> 1. Is this overriding of the pod template present in FlinkDeployment >> intentional? That is, should our custom command/args and initContainers >> have been overwritten? If so, I find it a bit confusing that these fields >> are present and available for use at all. >> 2. Since the ConfigMap volume will always be mounted as read-only, it >> seems to me there's some adjustments to be made in order for this script to >> work correctly. Do you think it would make sense for the script to copy >> over contents from the ConfigMap volume to a writable directory during >> initialization, and then use this copy for any subsequent operation? >> Perhaps copying over to $FLINK_HOME, which the user could set themselves, >> maybe even with a sane default which wouldn't fail on writes (eg >> /tmp/flink). >> >> Thanks in advance for your attention and hard work on the project! >> >> [1]: https://github.com/apache/flink-kubernetes-operator >> [2]: >> https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/docker-entrypoint.sh >> [3]: https://docs.datadoghq.com/integrations/flink/ >> [4]: >> https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/docker-entrypoint.sh#L86-L88 >> >
