barrywhart opened a new issue, #25827: URL: https://github.com/apache/airflow/issues/25827
### Apache Airflow version main (development) ### What happened The KubernetesPodOperator `env_vars` field is [documented to be templated](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L99), but it doesn't work. If I pass a JSON string to the constructor, it throws an exception: ``` File "/opt/airflow/airflow/models/baseoperator.py", line 410, in apply_defaults result = func(self, **kwargs, default_args=default_args) File "/opt/airflow/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 236, in __init__ self.env_vars = convert_env_vars(env_vars) if env_vars else [] File "/opt/airflow/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py", line 105, in convert_env_vars raise AirflowException(f"Expected dict or list, got {type(env_vars)}") airflow.exceptions.AirflowException: Expected dict or list, got <class 'str'> ``` ### What you think should happen instead If it receives a string, the constructor should convert it to Python (probably using `ast.literal_eval()`). ### How to reproduce I've written a test to demonstrate the behavior. It could be added to `kubernetes_tests/test_kubernetes_pod_operator.py`. ``` def test_env_vars_are_templatized(self): # WHEN env_vars = """ [ { "name": "ENV1", "value": "val1", "value_from": null }, { "name": "ENV2", "value": "val2", "value_from": null }, { "name": "ENV3", "value": null, "value_from": { "config_map_key_ref": null, "field_ref": { "api_version": null, "field_path": "status.podIP" }, "resource_field_ref": null, "secret_key_ref": null } } ] """ k = KubernetesPodOperator( namespace='default', image="ubuntu:16.04", cmds=["bash", "-cx"], arguments=["echo 10"], env_vars=env_vars, labels={"foo": "bar"}, name="test-" + str(random.randint(0, 1000000)), task_id="task" + self.get_current_task_name(), in_cluster=False, do_xcom_push=False, ) # THEN context = create_context(k) actual_pod = self.api_client.sanitize_for_serialization(k.build_pod_request_obj(context)) self.expected_pod['spec']['containers'][0]['env'] = [ {'name': 'ENV1', 'value': 'val1'}, {'name': 'ENV2', 'value': 'val2'}, {'name': 'ENV3', 'valueFrom': {'fieldRef': {'fieldPath': 'status.podIP'}}}, ] assert self.expected_pod == actual_pod ``` ### Operating System Debian GNU/Linux 11 (bullseye) (I'm running the Breeze dev environment) ### Versions of Apache Airflow Providers I'm using the Breeze development environment with the latest `main` code as of August 19, 2022: ``` commit 0254f30a5a90f0c3104782525fabdcfdc6d3b7df (HEAD -> main, origin/main, origin/HEAD) Author: Ash Berlin-Taylor <[email protected]> Date: Fri Aug 19 14:15:43 2022 +0100 Let timetables control generated run_ids. (#25795) ``` ### Deployment Other ### Deployment details Breeze development environment ### Anything else The problem happens every time. I would like to submit a PR to address this issue. I would also like (**if it sounds reasonable -- please let me know**) to add similar templating support for these other fields: * `resources` * `tolerations` ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
