saurasingh opened a new issue #16836:
URL: https://github.com/apache/airflow/issues/16836
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the
following questions.
Don't worry if they're not all applicable; just try to include what you can
:-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the
context.
-->
**Apache Airflow version**: 2.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
AWS EKS, AWS ECS
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
I have a python operator where python_callable just returns a data frame.
And I was getting this error
```
[2021-07-06 15:02:08,889] {xcom.py:229} ERROR - Could not serialize the XCom
value into JSON. If you are using pickle instead of JSON for XCom, then you
need to enable pickle support for XCom in your airflow config.
[2021-07-06 15:02:08,890] {taskinstance.py:1481} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1137, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1344, in _execute_task
self.xcom_push(key=XCOM_RETURN_KEY, value=result)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 70, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1925, in xcom_push
session=session,
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 67, in wrapper
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line
79, in set
value = XCom.serialize_value(value)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line
226, in serialize_value
return json.dumps(value).encode('UTF-8')
File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DataFrame is not JSON serializable
```
Then I set `enable_xcom_pickling: True` in airflow.cfg as per suggestion in
the error. It did work and DAG was successful but then my XCOM's UI broke with
the following error.
```
Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.
Python version: 3.7.10
Airflow version: 2.1.0
Node: airflow-webserver-7d7f64fbc4-zl8zk
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line
2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line
1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line
1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py",
line 39, in reraise
raise value
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line
1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line
1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py",
line 109, in wraps
return f(self, *args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/views.py",
line 553, in list
self.list_template, title=self.list_title, widgets=widgets
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py",
line 288, in render_template
template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
File
"/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line
140, in render_template
ctx.app,
File
"/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line
120, in _render
rv = template.render(context)
File
"/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line
1090, in render
self.environment.handle_exception()
File
"/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line
832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/_compat.py",
line 28, in reraise
raise value.with_traceback(tb)
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/list.html",
line 2, in top-level template code
{% import 'appbuilder/general/lib.html' as lib %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/base.html",
line 1, in top-level template code
{% extends base_template %}
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/www/templates/airflow/main.html",
line 20, in top-level template code
{% extends 'appbuilder/baselayout.html' %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html",
line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html",
line 37, in top-level template code
{% block body %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html",
line 19, in block "body"
{% block content %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/list.html",
line 13, in block "content"
{% block list_list scoped %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/model/list.html",
line 15, in block "list_list"
{{ widgets.get('list')()|safe }}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/widgets.py",
line 37, in __call__
return template.render(args)
File
"/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line
1090, in render
self.environment.handle_exception()
File
"/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line
832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/_compat.py",
line 28, in reraise
raise value.with_traceback(tb)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/www/templates/airflow/model_list.html",
line 21, in top-level template code
{% extends 'appbuilder/general/widgets/base_list.html' %}
File
"/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/general/widgets/base_list.html",
line 23, in top-level template code
{% block begin_loop_values %}
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/www/templates/airflow/model_list.html",
line 80, in block "begin_loop_values"
{% elif item[value] != None %}
File
"/home/airflow/.local/lib/python3.7/site-packages/pandas/core/generic.py", line
1443, in __nonzero__
f"The truth value of a {type(self).__name__} is ambiguous. "
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty,
a.bool(), a.item(), a.any() or a.all().
```
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
I expected to see the XCOM return_value in the xcom list as it use to in
previous versions.
<!-- What do you think went wrong? -->
**How to reproduce it**:
<!---
As minimally and precisely as possible. Keep in mind we do not have access
to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using
minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a
youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an
unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
