maryampashmi opened a new issue #17203:
URL: https://github.com/apache/airflow/issues/17203
Hello.
I have created a Variable but I am receiving an error for the final task. I
have posted the corresponding code. You just has to import the Json file for
the variable definition. And you should be able to re produce the same error. I
am using airflow 2.1.2
This is Json file, which is loaded as variable!
{
"example_variables_config": {
"var1": "value1",
"var2": [1, 2, 3],
"var3": {
"k": "value3"
}
}
}
Here is Dag:
```
import airflow
from datetime import datetime
from airflow import DAG
from airflow.models import Variable
from airflow.operators.dummy import DummyOperator
from airflow.operators.bash import BashOperator
default_args = {
'owner': 'airflow',
'start_date': datetime(2021, 2, 15),
'end_date': datetime(2021, 2, 15)
}
dag = DAG('example_variables',
schedule_interval="@once",
# Access like a dictionary
dag_config = Variable.get("example_variables_config", deserialize_json=True)
var1 = dag_config["var1"]
var2 = dag_config["var2"]
var3 = dag_config["var3"]
start = DummyOperator(
task_id="start",
dag=dag
)
# To test this task, run this command:
t1 = BashOperator(
task_id="get_dag_config",
bash_command='echo "{0}"'.format(dag_config),
dag=dag,
)
# You can directly use a variable from a jinja template
## {{ var.value.<variable_name> }}
t2 = BashOperator(
task_id="get_variable_value",
bash_command='echo {{ var.value.var3 }} ',
dag=dag,
)
## {{ var.json.<variable_name> }}
t3 = BashOperator(
task_id="get_variable_json",
bash_command='echo {{ var.json.example_variables_config.var3 }} ',
dag=dag,
)
start >> [t1, t2, t3]
```
**we recently uploaded the helm
https://github.com/airflow-helm/charts/blob/main/charts/airflow/CHANGELOG.md
error
```
[2021-07-23 14:46:32,330] {taskinstance.py:1107} INFO - Executing
<Task(BashOperator): get_variable_value> on 2021-02-15T00:00:00+00:00
[2021-07-23 14:46:32,333] {standard_task_runner.py:52} INFO - Started
process 1031 to run task
[2021-07-23 14:46:32,336] {standard_task_runner.py:76} INFO - Running:
['airflow', 'tasks', 'run', 'example_variables', 'get_variable_value',
'2021-02-15T00:00:00+00:00', '--job-id', '4976', '--pool', 'default_pool',
'--raw', '--subdir', 'DAGS_FOLDER/example_variables_dag.py', '--cfg-path',
'/tmp/tmppjwkr5bj', '--error-file', '/tmp/tmp3kalpsvq']
[2021-07-23 14:46:32,337] {standard_task_runner.py:77} INFO - Job 4976:
Subtask get_variable_value
[2021-07-23 14:46:32,503] {logging_mixin.py:104} INFO - Running
<TaskInstance: example_variables.get_variable_value 2021-02-15T00:00:00+00:00
[running]> on host airflow-worker-0.airflow-worker.airflow.svc.cluster.local
[2021-07-23 14:46:32,590] {taskinstance.py:1501} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1294, in _prepare_and_execute_task_with_callbacks
self.render_templates(context=context)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1793, in render_templates
self.task.render_template_fields(context)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py",
line 992, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context,
jinja_env, set())
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py",
line 1005, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env,
seen_oids)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py",
line 1042, in render_template
return jinja_env.from_string(content).render(**context)
File
"/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line
1090, in render
self.environment.handle_exception()
File
"/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line
832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/_compat.py",
line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/sandbox.py",
line 407, in getattr
value = getattr(obj, attribute)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1667, in __getattr__
self.var = Variable.get(item)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/variable.py",
line 140, in get
raise KeyError(f'Variable {key} does not exist')
KeyError: 'Variable var3 does not exist'
[2021-07-23 14:46:32,592] {taskinstance.py:1544} INFO - Marking task as
FAILED. dag_id=example_variables, task_id=get_variable_value,
execution_date=20210215T000000, start_date=20210723T144632,
end_date=20210723T144632
[2021-07-23 14:46:32,668] {local_task_job.py:149} INFO - Task exited with
return code 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
