11010cy opened a new issue #22638:
URL: https://github.com/apache/airflow/issues/22638
### Apache Airflow version
2.2.2
### What happened
get 500 while request 'api/v1/dags/{dag_id}/tasks'. It happens randomly.
```
AttributeError: 'dict' object has no attribute 'dump'
File "flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "flask/_compat.py", line 39, in reraise
raise value
File "flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "airflow/_vendor/connexion/decorators/decorator.py", line 48, in
wrapper
response = function(request)
File "airflow/_vendor/connexion/decorators/uri_parsing.py", line 144, in
wrapper
response = function(request)
File "airflow/_vendor/connexion/decorators/validation.py", line 384, in
wrapper
return function(request)
File "airflow/_vendor/connexion/decorators/response.py", line 103, in
wrapper
response = function(request)
File "airflow/_vendor/connexion/decorators/parameter.py", line 121, in
wrapper
return function(**kwargs)
File "airflow/api_connexion/security.py", line 47, in decorated
return func(*args, **kwargs)
File "airflow/api_connexion/endpoints/task_endpoint.py", line 71, in
get_tasks
return task_collection_schema.dump(task_collection)
File "marshmallow/schema.py", line 552, in dump
result = self._serialize(processed_obj, many=many)
File "marshmallow/schema.py", line 520, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "marshmallow/fields.py", line 338, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "marshmallow/fields.py", line 765, in _serialize
return [self.inner._serialize(each, attr, obj, **kwargs) for each in
value]
File "marshmallow/fields.py", line 765, in <listcomp>
return [self.inner._serialize(each, attr, obj, **kwargs) for each in
value]
File "marshmallow/fields.py", line 634, in _serialize
return schema.dump(nested_obj, many=many)
File "marshmallow/schema.py", line 552, in dump
result = self._serialize(processed_obj, many=many)
File "marshmallow/schema.py", line 520, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "marshmallow/fields.py", line 338, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "marshmallow/fields.py", line 1870, in _serialize
return self._serialize_method(obj)
File "airflow/api_connexion/schemas/task_schema.py", line 70, in get_params
return {k: v.dump() for k, v in params.items()}
File "airflow/api_connexion/schemas/task_schema.py", line 70, in <dictcomp>
return {k: v.dump() for k, v in params.items()}
```
After some inspection, I have located the cause of `task.params`.
In some cases, `current_app.dag_bag.get_dag(dag_id)` will return a dag with
one of task.params is type `dict` instead of type `ParamsDict`. It happens in
the same gunicorn worker process `15994`, while the `current_app.dag_bag.dags`
supposed to be the same instance. Before the error happened, there was nothing
wrong. But 2 minutes after, the error happend.
I have print the pid and the id of task.params before and when the above
error happend.
BEFORE
```[params] dagbag min_serialized_dag_fetch_secs=0:00:10
[params] check_updated sd_last_updated_datetime=2022-03-30
04:18:07.180249+00:00
[params] pid=159994 thread=MainThread id=139950227846976
[params] get_tasks 139944558692128 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f476175d040>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476175d130>, 'business': <airflow.models.param.Param object at
0x7f476175dcd0>}
[params] get_tasks 139944558781120 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f47617bbbb0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617bbf70>, 'business': <airflow.models.param.Param object at
0x7f476174c9a0>}
[params] get_tasks 139944558691008 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761704520>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761704550>, 'business': <airflow.models.param.Param object at
0x7f4761704250>}
[params] get_tasks 139944558691488 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761751d00>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476184bc10>, 'business': <airflow.models.param.Param object at
0x7f47617bbe80>}
[params] get_tasks 139944558782080 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761742550>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761742c70>, 'business': <airflow.models.param.Param object at
0x7f4761742fa0>}
[params] get_tasks 139944558781360 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f47617041f0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617045e0>, 'business': <airflow.models.param.Param object at
0x7f47617044c0>}
[params] get_tasks 139944558782720 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761723460>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617234c0>, 'business': <airflow.models.param.Param object at
0x7f4761723550>}
[params] get_tasks 139949715461600 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761723fa0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761723fd0>, 'business': <airflow.models.param.Param object at
0x7f4761723eb0>}
[params] get_tasks 139949730158048 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f47617238e0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617239d0>, 'business': <airflow.models.param.Param object at
0x7f4761723790>}
[params] get_tasks 139944558961024 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761704e20>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761704430>, 'business': <airflow.models.param.Param object at
0x7f4761704850>}
[params] get_tasks 139949716343280 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f476175d970>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476175d3a0>, 'business': <airflow.models.param.Param object at
0x7f476175dac0>}
[params] get_tasks 139944559033456 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f476174c5b0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476174cb80>, 'business': <airflow.models.param.Param object at
0x7f476174cd60>}
127.0.0.1 - airflow [30/Mar/2022:14:04:38 +0800] "GET /api/v1/dags/xxx/tasks
HTTP/1.0" 200 64063 "-" "OpenAPI-Generator/2.2.0/python"
```
WHEN
```params] dagbag min_serialized_dag_fetch_secs=0:00:10
[params] check_updated sd_last_updated_datetime=2022-03-30
04:18:07.180249+00:00
[params] pid=159994 thread=MainThread id=139950227846976
[params] get_tasks 139944558692128 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f476175d040>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476175d130>, 'business': <airflow.models.param.Param object at
0x7f476175dcd0>}
[params] get_tasks 139944558781120 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f47617bbbb0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617bbf70>, 'business': <airflow.models.param.Param object at
0x7f476174c9a0>}
[params] get_tasks 139944558691008 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761704520>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761704550>, 'business': <airflow.models.param.Param object at
0x7f4761704250>}
[params] get_tasks 139944558691488 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761751d00>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476184bc10>, 'business': <airflow.models.param.Param object at
0x7f47617bbe80>}
[params] get_tasks 139944558782080 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761742550>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761742c70>, 'business': <airflow.models.param.Param object at
0x7f4761742fa0>}
[params] get_tasks 139944558781360 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f47617041f0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617045e0>, 'business': <airflow.models.param.Param object at
0x7f47617044c0>}
[params] get_tasks 139944558782720 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761723460>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617234c0>, 'business': <airflow.models.param.Param object at
0x7f4761723550>}
[params] get_tasks 139949715461600 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761723fa0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761723fd0>, 'business': <airflow.models.param.Param object at
0x7f4761723eb0>}
[params] get_tasks 139949730158048 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f47617238e0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f47617239d0>, 'business': <airflow.models.param.Param object at
0x7f4761723790>}
[params] get_tasks 139944558961024 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f4761704e20>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f4761704430>, 'business': <airflow.models.param.Param object at
0x7f4761704850>}
[params] get_tasks 139944557266624 <class 'dict'> {xxx}
[params] get_tasks 139944559033456 <class 'airflow.models.param.ParamsDict'>
{'dag': <airflow.models.param.Param object at 0x7f476174c5b0>, 'tianji':
<airflow.models.param.Param objec
t at 0x7f476174cb80>, 'business': <airflow.models.param.Param object at
0x7f476174cd60>}
[^[[34m2022-03-30 14:06:30,706^[[0m] {^[[34mapp.py:^[[0m1891} ERROR^[[0m -
Exception on /api/v1/dags/troubleshooter_15/tasks [GET]^[[0m
```
We could see that task `139949716343280` with type `ParamsDict` changed to
`139944557266624` with type `dict`, while the other task object hasn't changed.
Does anyone knows: when and where could the `task.params` in a dag of
`current_app.dag_bag.dags` be changed from `ParamsDict` to `dict`
### What you think should happen instead
There's maybe a bug related to `current_app.dag_bag.dags`.
### How to reproduce
I have no idea, it happened occasionally.
### Operating System
Linux Server 7.2 (Paladin)
### Versions of Apache Airflow Providers
apache-airflow-providers-alibaba==1.0.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-sqlite==2.0.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
