madison-ookla opened a new issue #13350:
URL: https://github.com/apache/airflow/issues/13350
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**: X
- **OS** (e.g. from /etc/os-release): X
- **Kernel** (e.g. `uname -a`): X
- **Install tools**: X
- **Others**: X
**What happened**:
The `UndefinedJinjaVariablesRule` appears to expect non-standard python
objects to have a `template_fields` attribute, when many (`pathlib.Path`,
`None`) do not.
<details><summary>`Path` example</summary>
```python
==========================================================================================================================================================
STATUS
==========================================================================================================================================================
Check for latest versions of apache-airflow and
checker...........................................................................................................................................................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin
class...........................................................................................................................................................................................................................................................................SUCCESS
Chain between DAG and operator not
allowed........................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_id is not
unique..................................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_type is not
nullable..............................................................................................................................................................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom
operators.................................................................................................................................................................................................................................................SUCCESS
Fernet is enabled by
default......................................................................................................................................................................................................................................................................................FAIL
GCP service account key
deprecation...............................................................................................................................................................................................................................................................................SUCCESS
Unify hostname_callable option in core
section....................................................................................................................................................................................................................................................................FAIL
Changes in import paths of hooks, operators, sensors and
others...................................................................................................................................................................................................................................................SUCCESS
Users must delete deprecated configs for
KubernetesExecutor.......................................................................................................................................................................................................................................................FAIL
Legacy UI is deprecated by
default................................................................................................................................................................................................................................................................................SUCCESS
Logging configuration has been moved to new
section...............................................................................................................................................................................................................................................................FAIL
Removal of Mesos
Executor.........................................................................................................................................................................................................................................................................................SUCCESS
No additional argument allowed in
BaseOperator....................................................................................................................................................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file
value...............................................................................................................................................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib
module....................................................................................................................................................................................................................................................................SUCCESS
Changes in import path of remote task
handlers....................................................................................................................................................................................................................................................................SUCCESS
Traceback (most recent call last):
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow",
line 37, in <module>
args.func(args)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py",
line 88, in run
all_problems = check_upgrade(formatter, rules)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py",
line 37, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py",
line 44, in from_rule
result = rule.check()
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 141, in check
dag_messages = self.iterate_over_dag_tasks(dag)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 128, in iterate_over_dag_tasks
error_messages = self.iterate_over_template_fields(task)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 117, in iterate_over_template_fields
self._check_rendered_content(rendered_content, set())
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 63, in _check_rendered_content
debug_error_messages.update(self._check_rendered_content(value))
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 69, in _check_rendered_content
return self._nested_check_rendered(rendered_content, seen_oids)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 75, in _nested_check_rendered
nested_template_fields = rendered_content.template_fields
AttributeError: 'PosixPath' object has no attribute 'template_fields'
```
</details>
<details><summary>`None` example</summary>
```python
==========================================================================================================================================================
STATUS
==========================================================================================================================================================
Check for latest versions of apache-airflow and
checker...........................................................................................................................................................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin
class...........................................................................................................................................................................................................................................................................SUCCESS
Chain between DAG and operator not
allowed........................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_id is not
unique..................................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_type is not
nullable..............................................................................................................................................................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom
operators.................................................................................................................................................................................................................................................SUCCESS
Fernet is enabled by
default......................................................................................................................................................................................................................................................................................FAIL
GCP service account key
deprecation...............................................................................................................................................................................................................................................................................SUCCESS
Unify hostname_callable option in core
section....................................................................................................................................................................................................................................................................FAIL
Changes in import paths of hooks, operators, sensors and
others...................................................................................................................................................................................................................................................SUCCESS
Users must delete deprecated configs for
KubernetesExecutor.......................................................................................................................................................................................................................................................FAIL
Legacy UI is deprecated by
default................................................................................................................................................................................................................................................................................SUCCESS
Logging configuration has been moved to new
section...............................................................................................................................................................................................................................................................FAIL
Removal of Mesos
Executor.........................................................................................................................................................................................................................................................................................SUCCESS
No additional argument allowed in
BaseOperator....................................................................................................................................................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file
value...............................................................................................................................................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib
module....................................................................................................................................................................................................................................................................SUCCESS
Changes in import path of remote task
handlers....................................................................................................................................................................................................................................................................SUCCESS
Error with <Task(PythonOperator): ts_manual_edits>: 'NoneType' object has no
attribute 'template_fields'
Traceback (most recent call last):
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow",
line 37, in <module>
args.func(args)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py",
line 88, in run
all_problems = check_upgrade(formatter, rules)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py",
line 37, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py",
line 44, in from_rule
result = rule.check()
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 146, in check
dag_messages = self.iterate_over_dag_tasks(dag)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 130, in iterate_over_dag_tasks
error_messages = self.iterate_over_template_fields(task)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 118, in iterate_over_template_fields
self._check_rendered_content(rendered_content, set())
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 64, in _check_rendered_content
debug_error_messages.update(self._check_rendered_content(value))
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 70, in _check_rendered_content
return self._nested_check_rendered(rendered_content, seen_oids)
File
"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py",
line 76, in _nested_check_rendered
nested_template_fields = rendered_content.template_fields
AttributeError: 'NoneType' object has no attribute 'template_fields'
```
</details>
**What you expected to happen**:
I expect these types to be handled appropriately by the upgrade rule and not
assume that a `template_fields` attribute exists on them.
**How to reproduce it**:
Create a simple DAG with a `PythonOperator` task that has `op_kwargs:
{"foo": None}` and run the upgrade check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
