ecerulm opened a new issue #15325:
URL: https://github.com/apache/airflow/issues/15325
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the
following questions.
Don't worry if they're not all applicable; just try to include what you can
:-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the
context.
-->
**Apache Airflow version**: 2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
While running `airflow connections list` with a `xcom_backend =
xcom.MyXComBackend` in `airflow.cfg` I will get a
ImportError: cannot import name 'BaseHook' from partially initialized
module 'airflow.hooks.base' (most likely due to a circular import)
(/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
(see below for complete traceback).
The issue seems to be that `airflow/cli/commands/connection_command.py`
imports `BaseHook` directly and `BaseHook -> Connection -> BaseOperator ->
TaskInstance -> XCom -> MyXComBackend -> S3Hook -> BaseHook ` (again see
complete traceback below)
```
airflow connections list
[2021-04-12 10:51:34,020] {configuration.py:459} ERROR - cannot import name
'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely
due to a circular import)
(/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
Traceback (most recent call last):
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/configuration.py",
line 457, in getimport
return import_string(full_qualified_path)
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py",
line 32, in import_string
module = import_module(module_path)
File
"/Users/rubelagu/.pyenv/versions/3.8.8/lib/python3.8/importlib/__init__.py",
line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File "/Users/rubelagu/tmp/airflow-xcom-backend/plugins/xcom.py", line 4,
in <module>
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 37, in <module>
from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 41, in <module>
from airflow.hooks.base import BaseHook
ImportError: cannot import name 'BaseHook' from partially initialized module
'airflow.hooks.base' (most likely due to a circular import)
(/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/bin/airflow", line 8,
in <module>
sys.exit(main())
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/__main__.py",
line 40, in main
args.func(args)
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py",
line 47, in command
func = import_string(import_path)
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py",
line 32, in import_string
module = import_module(module_path)
File
"/Users/rubelagu/.pyenv/versions/3.8.8/lib/python3.8/importlib/__init__.py",
line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/cli/commands/connection_command.py",
line 30, in <module>
from airflow.hooks.base import BaseHook
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py",
line 23, in <module>
from airflow.models.connection import Connection
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/__init__.py",
line 20, in <module>
from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/baseoperator.py",
line 55, in <module>
from airflow.models.taskinstance import Context, TaskInstance,
clear_task_instances
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 58, in <module>
from airflow.models.xcom import XCOM_RETURN_KEY, XCom
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/xcom.py",
line 289, in <module>
XCom = resolve_xcom_backend()
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/xcom.py",
line 279, in resolve_xcom_backend
clazz = conf.getimport("core", "xcom_backend",
fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
File
"/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/configuration.py",
line 460, in getimport
raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: The object could not be loaded.
Please check "xcom_backend" key in "core" section. Current value:
"xcom.MyXComBackend".
```
**What you expected to happen**:
I expected to be able to use S3Hook or GCSHook from a cusom XCom backedn
following the example in what I though was the canonical example at
https://medium.com/apache-airflow/airflow-2-0-dag-authoring-redesigned-651edc397178
**How to reproduce it**:
* In `airflow.cfg` set `xcom_backend = xcom.MyXComBackend`
* Create file `plugins/xcom.py` with the following contents
```
from airflow.models.xcom import BaseXCom
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
class MyXComBackend(BaseXCom):
pass
```
**Anything else we need to know**:
This can be workaround by not importing the hooks on the module level but at
the method level, but that is
* really ugly
* not evident for anybody trying to create a custom xcom backend for the
first time.
```
from airflow.models.xcom import BaseXCom
class MyXComBackend(BaseXCom):
@staticmethod
def serialize_value(value):
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook()
pass
@staticmethod
def deserialize_value(value):
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook()
pass
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
