tirkarthi commented on issue #27147: URL: https://github.com/apache/airflow/issues/27147#issuecomment-1285733237
I receive below error on trying to reproduce this with the posted dag file. The config file should be relative to the dag or config file should be present as under the value template_searchpath. The application_file should be replaced with file content in below code part using `resolve_template_files` function where all extensions .yaml, .json and .yml are checked. If I add a trailing space after ".yaml " extension then I can reproduce the error as the extension doesn't match and just sets application_file as "/opt/airflow/kubernetes/templates/airflow-worker-template.yaml" instead of the contents of the yaml file. * Can you please add if I am missing some configuration? * Can you please try with airflow-worker-template.yaml relative to the dag file and have `application_file="airflow-worker-template.yaml"` without any spaces in the end? You can also try breakpoint in the below file to see what `resolve_template_files` does. https://github.com/apache/airflow/blob/b9e133e40c2848b0d555051a99bf8d2816fd28a7/airflow/models/abstractoperator.py#L164-L185 ``` airflow dags test spark_pi [2022-10-20 15:06:45,774] {dagbag.py:537} INFO - Filling up the DagBag from /files/dags [2022-10-20 15:06:45,792] {abstractoperator.py:176} ERROR - Failed to resolve template field '\x1b[01mapplication_file\x1b[22m' Traceback (most recent call last): File "/opt/airflow/airflow/models/abstractoperator.py", line 174, in resolve_template_files setattr(self, field, env.loader.get_source(env, content)[0]) # type: ignore File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 218, in get_source raise TemplateNotFound(template) jinja2.exceptions.TemplateNotFound: /opt/airflow/kubernetes/templates/airflow-worker-template.yaml [2022-10-20 15:06:45,873] {dag.py:3654} INFO - dagrun id: spark_pi /opt/airflow/airflow/models/dag.py:3669 RemovedInAirflow3Warning: Calling `DAG.create_dagrun()` without an explicit data interval is deprecated [2022-10-20 15:06:45,903] {dag.py:3671} INFO - created dagrun <DagRun spark_pi @ 2022-10-20T15:06:45.774111+00:00: manual__2022-10-20T15:06:45.774111+00:00, state:running, queued_at: None. externally triggered: False> [2022-10-20 15:06:45,925] {dag.py:3621} INFO - ***************************************************** [2022-10-20 15:06:45,925] {dag.py:3625} INFO - Running task airflow_health_check [2022-10-20 15:06:46,334] {taskinstance.py:1587} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=spark_pi AIRFLOW_CTX_TASK_ID=airflow_health_check AIRFLOW_CTX_EXECUTION_DATE=2022-10-20T15:06:45.774111+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2022-10-20T15:06:45.774111+00:00 [2022-10-20 15:06:46,334] {taskinstance.py:1587} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=spark_pi AIRFLOW_CTX_TASK_ID=airflow_health_check AIRFLOW_CTX_EXECUTION_DATE=2022-10-20T15:06:45.774111+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2022-10-20T15:06:45.774111+00:00 [2022-10-20 15:06:46,341] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=spark_pi, task_id=airflow_health_check, execution_date=20221020T150645, start_date=, end_date=20221020T150646 [2022-10-20 15:06:46,341] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=spark_pi, task_id=airflow_health_check, execution_date=20221020T150645, start_date=, end_date=20221020T150646 [2022-10-20 15:06:46,360] {dag.py:3629} INFO - airflow_health_check ran successfully! [2022-10-20 15:06:46,360] {dag.py:3632} INFO - ***************************************************** [2022-10-20 15:06:46,369] {dag.py:3621} INFO - ***************************************************** [2022-10-20 15:06:46,370] {dag.py:3625} INFO - Running task main_app [2022-10-20 15:06:46,390] {abstractoperator.py:422} ERROR - Exception rendering Jinja template for task 'main_app', field 'application_file'. Template: '/opt/airflow/kubernetes/templates/airflow-worker-template.yaml' Traceback (most recent call last): File "/opt/airflow/airflow/models/abstractoperator.py", line 415, in _do_render_template_fields seen_oids, File "/opt/airflow/airflow/models/abstractoperator.py", line 466, in render_template template = jinja_env.get_template(value) File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1010, in get_template return self._load_template(name, globals) File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 969, in _load_template template = self.loader.load(self, name, self.make_globals(globals)) File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 126, in load source, filename, uptodate = self.get_source(environment, name) File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 218, in get_source raise TemplateNotFound(template) jinja2.exceptions.TemplateNotFound: /opt/airflow/kubernetes/templates/airflow-worker-template.yaml ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
