Here is a sample of what I am seeing.
I am running on ubuntu, python 3.5. This is a subset of the full message.
Once it went into the snakebite package, checked and the library wasn’t
supported in python3.
….
File "/usr/local/airflow/dags/dim_date.py", line 5, in
from airflow.operators.sensors import S3KeySensor
File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py",
line 34, in
from airflow.hooks.hdfs_hook import HDFSHook
File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py",
line 20, in
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
^
> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <cindyrottingh...@gmail.com>
> wrote:
>
> Hi,
>
> Yes, it looks like all of the other sensors will work under Python3, but not
> hdfs. I am planning to use the s3 sensor, which is wrapped up in the
> sensors.py module. My issue is that when I test my dag or airflow
> installation under python3, I get errors about the hdfs_hook, which I’m not
> using. Other then creating my own version of the sensors.py file and
> removing the hdfs related functions/libraries, is there any thing else I can
> do to work around this?
>
>
>
>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>>
>> Hi Cindy,
>>
>> The other sensors should work under Python3. We try to support Python3 as
>> much as possible, but sometimes libraries are used that are not compatible.
>> Could you describe what you are running into?
>>
>> Cheers, Fokko
>>
>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <cindyrottingh...@gmail.com>:
>>
>>> Hi,
>>>
>>> Are there any plans to update the HDFS_hook.py script to remove the
>>> reference to the snakebite python library? I’d like to run airflow on
>>> python3, and this is causing some issues. The hdfs_hook script is
>>> referenced in the sensors module.
>>>
>>> Any suggestions?
>>>
>>> Thanks,
>>> Cindy
>
