Re: Python3 and sensors module

[Joy Gao]

Not a full patch to replace snakebite with libhdfs, but should temporarily
unblock ci/cd: https://github.com/apache/incubator-airflow/pull/3365

On Wed, May 16, 2018 at 10:00 AM, Bolke de Bruin  wrote:

> Snakebite is not py3 compatible. We should move to libhdfs. Happy to take
> patches!
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis <
> cindyrottingh...@gmail.com> het volgende geschreven:
> >
> > Here is a sample of what I am seeing.
> > I am running on ubuntu, python 3.5.   This is a subset of the full
> message.  Once it went into the snakebite package, checked and the library
> wasn’t supported in python3.
> > ….
> > File "/usr/local/airflow/dags/dim_date.py", line 5, in 
> >from airflow.operators.sensors import S3KeySensor
> >  File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py",
> line 34, in 
> >from airflow.hooks.hdfs_hook import HDFSHook
> >  File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py",
> line 20, in 
> >from snakebite.client import Client, HAClient, Namenode,
> AutoConfigClient
> >  File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py",
> line 1473
> >baseTime = min(time * (1L << retries), cap);
> >^
> >
> >
> >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <
> cindyrottingh...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> Yes, it looks like all of the other sensors will work under Python3,
> but not hdfs.  I am planning to use the s3 sensor, which is wrapped up in
> the sensors.py module.  My issue is that when I test my dag or airflow
> installation under python3, I get errors about the hdfs_hook, which I’m not
> using.   Other then creating my own version of the sensors.py file and
> removing the hdfs related functions/libraries, is there any thing else I
> can do to work around this?
> >>
> >>
> >>
> >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko 
> wrote:
> >>>
> >>> Hi Cindy,
> >>>
> >>> The other sensors should work under Python3. We try to support Python3
> as
> >>> much as possible, but sometimes libraries are used that are not
> compatible.
> >>> Could you describe what you are running into?
> >>>
> >>> Cheers, Fokko
> >>>
> >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <
> cindyrottingh...@gmail.com>:
> >>>
>  Hi,
> 
>  Are there any plans to update the HDFS_hook.py script to remove the
>  reference to the snakebite python library? I’d like to run airflow on
>  python3, and this is causing some issues.   The hdfs_hook script is
>  referenced in the sensors module.
> 
>  Any suggestions?
> 
>  Thanks,
>  Cindy
> >>
> >
>
>

Re: Python3 and sensors module

[Bolke de Bruin]

Snakebite is not py3 compatible. We should move to libhdfs. Happy to take 
patches!

B.

Verstuurd vanaf mijn iPad

> Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis  
> het volgende geschreven:
> 
> Here is a sample of what I am seeing.   
> I am running on ubuntu, python 3.5.   This is a subset of the full message.  
> Once it went into the snakebite package, checked and the library wasn’t 
> supported in python3.
> ….
> File "/usr/local/airflow/dags/dim_date.py", line 5, in 
>from airflow.operators.sensors import S3KeySensor
>  File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", 
> line 34, in 
>from airflow.hooks.hdfs_hook import HDFSHook
>  File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", 
> line 20, in 
>from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
>  File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473
>baseTime = min(time * (1L << retries), cap);
>^
> 
> 
>> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis  
>> wrote:
>> 
>> Hi,
>> 
>> Yes, it looks like all of the other sensors will work under Python3, but not 
>> hdfs.  I am planning to use the s3 sensor, which is wrapped up in the 
>> sensors.py module.  My issue is that when I test my dag or airflow 
>> installation under python3, I get errors about the hdfs_hook, which I’m not 
>> using.   Other then creating my own version of the sensors.py file and 
>> removing the hdfs related functions/libraries, is there any thing else I can 
>> do to work around this?
>> 
>> 
>> 
>>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko  wrote:
>>> 
>>> Hi Cindy,
>>> 
>>> The other sensors should work under Python3. We try to support Python3 as
>>> much as possible, but sometimes libraries are used that are not compatible.
>>> Could you describe what you are running into?
>>> 
>>> Cheers, Fokko
>>> 
>>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis :
>>> 
 Hi,
 
 Are there any plans to update the HDFS_hook.py script to remove the
 reference to the snakebite python library? I’d like to run airflow on
 python3, and this is causing some issues.   The hdfs_hook script is
 referenced in the sensors module.
 
 Any suggestions?
 
 Thanks,
 Cindy
>> 
> 

Re: Python3 and sensors module

[Cindy Rottinghuis]

Here is a sample of what I am seeing.   
I am running on ubuntu, python 3.5.   This is a subset of the full message.  
Once it went into the snakebite package, checked and the library wasn’t 
supported in python3.
….
 File "/usr/local/airflow/dags/dim_date.py", line 5, in 
from airflow.operators.sensors import S3KeySensor
  File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", 
line 34, in 
from airflow.hooks.hdfs_hook import HDFSHook
  File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", 
line 20, in 
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
  File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
^


> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis  
> wrote:
> 
> Hi,
> 
> Yes, it looks like all of the other sensors will work under Python3, but not 
> hdfs.  I am planning to use the s3 sensor, which is wrapped up in the 
> sensors.py module.  My issue is that when I test my dag or airflow 
> installation under python3, I get errors about the hdfs_hook, which I’m not 
> using.   Other then creating my own version of the sensors.py file and 
> removing the hdfs related functions/libraries, is there any thing else I can 
> do to work around this?
> 
> 
> 
>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko  wrote:
>> 
>> Hi Cindy,
>> 
>> The other sensors should work under Python3. We try to support Python3 as
>> much as possible, but sometimes libraries are used that are not compatible.
>> Could you describe what you are running into?
>> 
>> Cheers, Fokko
>> 
>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis :
>> 
>>> Hi,
>>> 
>>> Are there any plans to update the HDFS_hook.py script to remove the
>>> reference to the snakebite python library? I’d like to run airflow on
>>> python3, and this is causing some issues.   The hdfs_hook script is
>>> referenced in the sensors module.
>>> 
>>> Any suggestions?
>>> 
>>> Thanks,
>>> Cindy
> 

Re: Python3 and sensors module

[Driesprong, Fokko]

Hi Cindy,

The other sensors should work under Python3. We try to support Python3 as
much as possible, but sometimes libraries are used that are not compatible.
Could you describe what you are running into?

Cheers, Fokko

2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis :

> Hi,
>
> Are there any plans to update the HDFS_hook.py script to remove the
> reference to the snakebite python library? I’d like to run airflow on
> python3, and this is causing some issues.   The hdfs_hook script is
> referenced in the sensors module.
>
> Any suggestions?
>
> Thanks,
> Cindy