Nedko Hristov created AIRFLOW-4367:
--------------------------------------
Summary: When extending BaseSensorOperator getting exception in
python 3
Key: AIRFLOW-4367
URL: https://issues.apache.org/jira/browse/AIRFLOW-4367
Project: Apache Airflow
Issue Type: Bug
Components: plugins
Affects Versions: 1.10.3
Environment: python 3, airflow 1.10.3
Reporter: Nedko Hristov
When creating a custom sensor in 1.10.3 (EMRTaskSensor) I extend
BaseSensorOperator (e.g. from airflow.operators.sensors import
BaseSensorOperator) which in turn initializes all sensors including HdfsSensor
which uses snakebite (not python 3 compatible) I get the following stacktrace
due to the 1L implicits in Python 3. The same works as expected in 1.10.1 as
I've tested it and have been running it for a while. Now, I know that snakebite
is not python 3 compatible. I also know that I'm not using HDFS so not sure why
the need to initialize it by default. A workaround will be to remove snakebite
and use snakebite-py3 post install in my docker container but I'd prefer if
there was a smarter way to not break my instance with features I don't use.
Possible change the modules.py where things get loaded...
File "/git/airflow/dag/dp_etl_flow.py", line 8, in <module>
from plugins.sc_data_runner.sensors.emr_task_sensor import EMRTaskSensor
File "/git/airflow/plugins/sc_data_runner/sensors/emr_task_sensor.py", line 3,
in <module>
from airflow.operators.sensors import BaseSensorOperator
File
"/usr/local/airflow/.local/lib/python3.6/site-packages/airflow/operators/sensors.py",
line 35, in <module>
from airflow.sensors.hdfs_sensor import HdfsSensor as HdfsSensorImp
File
"/usr/local/airflow/.local/lib/python3.6/site-packages/airflow/sensors/hdfs_sensor.py",
line 25, in <module>
from airflow.hooks.hdfs_hook import HDFSHook
File
"/usr/local/airflow/.local/lib/python3.6/site-packages/airflow/hooks/hdfs_hook.py",
line 24, in <module>
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
File "/usr/local/lib/python3.6/site-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
