Kristof-Mattei edited a comment on issue #10027: URL: https://github.com/apache/airflow/issues/10027#issuecomment-665990279
`pod_mutation_hook` works perfectly, I am stating that defining `pod_mutation_hook` inside of `airflow_local_settings.py` mounted INSIDE of a K8s container via `AIRFLOW__KUBERNETES__AIRFLOW_LOCAL_SETTINGS_CONFIGMAP ` has 0 effect. Let's look at the documentation: https://airflow.apache.org/docs/stable/configurations-ref.html#airflow-local-settings-configmap The example config file looks as follows: ``` --- apiVersion: v1 kind: ConfigMap metadata: name: airflow-configmap data: airflow_local_settings.py: | def pod_mutation_hook(pod): ... airflow.cfg: | ... ``` Let's try that out: We set `AIRFLOW__KUBERNETES__AIRFLOW_LOCAL_SETTINGS_CONFIGMAP` to `airflow-k8s-configmap` and create the following configmap: ``` --- apiVersion: v1 kind: ConfigMap metadata: name: airflow-k8s-configmap data: airflow_local_settings.py: | import pprint import logging logging.warning("Imported airflow-k8s-configmap") def pod_mutation_hook(pod): pod.annotations["foo"] = "bar" logging.warning(pprint.pprint(pod)) ``` One would expect this logging to show up in the scheduler, and one would expect newly created pods to have the `"foo": "bar"` annotation. I have tried this, and I can assure you, it does not work. I have debugged the code line by line, and found out that this config-map is mounted INSIDE of the newly created pod, and thus AFTER the `KubernetesExecutor` has LAUNCHED the pod, ergo AFTER `KubernetesExecutor` has done any modificatons through `pod_mutation_hook(pod)`. In order of execution: 1. Read contents of `AIRFLOW_LOCAL_SETTINGS_CONFIGMAP` into variable: https://github.com/apache/airflow/blob/master/airflow/executors/kubernetes_executor.py#L218 1. Define the volume mount that mounts `airflow_local_settings.py` in this config map to `config_path = '{}/config/airflow_local_settings.py'.format(self.worker_airflow_home)`: https://github.com/apache/airflow/blob/master/airflow/kubernetes/worker_configuration.py#L298 1. Create the pod framework to be launched: https://github.com/apache/airflow/blob/master/airflow/kubernetes/worker_configuration.py#L426 1. Pass on this pod: https://github.com/apache/airflow/blob/master/airflow/executors/kubernetes_executor.py#L424 -> https://github.com/apache/airflow/blob/master/airflow/executors/kubernetes_executor.py#L462 -> https://github.com/apache/airflow/blob/master/airflow/executors/kubernetes_executor.py#L480 1. Mutate the pod, with `pod_mutation_hook` and launch it: https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_launcher.py#L69 All of this happened in a context where we do not have that particular `airflow_local_settings.py` mounted, ergo our `pod_mutation_hook` is never used! The newly launched pod, in 5. will have said file mounted, but the new pod is not responsible for launching new pods, it's responsible for executing a task! In order to get `pod_mutation_hook` one needs to mount a `airflow_local_settings.py` inside of the scheduler, which is what we did, and how we fixed it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
