tirkarthi opened a new issue, #27084: URL: https://github.com/apache/airflow/issues/27084
### Apache Airflow version main (development) ### What happened Opening issue as per comment : https://github.com/apache/airflow/issues/23727#issuecomment-1207530417 . We have also noticed this issue where we have a 2.1.x setup using Kubernetes executor and on upgrade the new task instances created post upgrade are working fine in task instance endpoint. The old objects fail with similar traceback as below. We also tried main branch (2.4.1 as of writing) and it also has same issue. It seems a fix similar to https://github.com/apache/airflow/pull/24117 has to be made with a custom String field for `executor_config` in task instance schema that calls the `_serialize` method and on error returns empty dict as string. We have a fix internally though a test case might not be possible since it needs an older value of executor_config that we can't export due to private data. I will be happy to make a PR with the fix though and opened this issue for discussion, ``` File "/home/airflow/.local/lib/python3.9/site-packages/airflow/api_connexion/endpoints/task_instance_endpoint.py", line 412, in get_task_instances_batch return task_instance_collection_schema.dump( File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 557, in dump result = self._serialize(processed_obj, many=many) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 525, in _serialize value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 342, in serialize return self._serialize(value, attr, obj, **kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 774, in _serialize return [self.inner._serialize(each, attr, obj, **kwargs) for each in value] File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 774, in <listcomp> return [self.inner._serialize(each, attr, obj, **kwargs) for each in value] File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 643, in _serialize return schema.dump(nested_obj, many=many) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 557, in dump result = self._serialize(processed_obj, many=many) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 525, in _serialize value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 342, in serialize return self._serialize(value, attr, obj, **kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 893, in _serialize return utils.ensure_text_type(value) File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/utils.py", line 212, in ensure_text_type return str(val) File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 214, in __repr__ return self.to_str() File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 210, in to_str return pprint.pformat(self.to_dict()) File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 196, in to_dict result[attr] = value.to_dict() File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1058, in to_dict result[attr] = list(map( File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1059, in <lambda> lambda x: x.to_dict() if hasattr(x, "to_dict") else x, File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 660, in to_dict value = getattr(self, attr) File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 458, in startup_probe return self._startup_probe AttributeError: 'V1Container' object has no attribute '_startup_probe' ``` cc: @dstandish @joshzana ### What you think should happen instead Older task instances that cannot be serialized should possibly return empty dict instead of failing completely. ### How to reproduce 1. Upgrade to main branch from an older version like 2.1.x with task instances and executor_config using Kubernetes as executor. 2. Hit task instance endpoint of old dagrun to fetch task instances with executor_config serialized. The traceback is thrown. ### Operating System Redhat ### Versions of Apache Airflow Providers _No response_ ### Deployment Other Docker-based deployment ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
