Re: Python3 and sensors module
Not a full patch to replace snakebite with libhdfs, but should temporarily unblock ci/cd: https://github.com/apache/incubator-airflow/pull/3365 On Wed, May 16, 2018 at 10:00 AM, Bolke de Bruin wrote: > Snakebite is not py3 compatible. We should move to libhdfs. Happy to take > patches! > > B. > > Verstuurd vanaf mijn iPad > > > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis < > cindyrottingh...@gmail.com> het volgende geschreven: > > > > Here is a sample of what I am seeing. > > I am running on ubuntu, python 3.5. This is a subset of the full > message. Once it went into the snakebite package, checked and the library > wasn’t supported in python3. > > …. > > File "/usr/local/airflow/dags/dim_date.py", line 5, in > >from airflow.operators.sensors import S3KeySensor > > File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", > line 34, in > >from airflow.hooks.hdfs_hook import HDFSHook > > File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", > line 20, in > >from snakebite.client import Client, HAClient, Namenode, > AutoConfigClient > > File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", > line 1473 > >baseTime = min(time * (1L << retries), cap); > >^ > > > > > >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis < > cindyrottingh...@gmail.com> wrote: > >> > >> Hi, > >> > >> Yes, it looks like all of the other sensors will work under Python3, > but not hdfs. I am planning to use the s3 sensor, which is wrapped up in > the sensors.py module. My issue is that when I test my dag or airflow > installation under python3, I get errors about the hdfs_hook, which I’m not > using. Other then creating my own version of the sensors.py file and > removing the hdfs related functions/libraries, is there any thing else I > can do to work around this? > >> > >> > >> > >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko > wrote: > >>> > >>> Hi Cindy, > >>> > >>> The other sensors should work under Python3. We try to support Python3 > as > >>> much as possible, but sometimes libraries are used that are not > compatible. > >>> Could you describe what you are running into? > >>> > >>> Cheers, Fokko > >>> > >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis < > cindyrottingh...@gmail.com>: > >>> > Hi, > > Are there any plans to update the HDFS_hook.py script to remove the > reference to the snakebite python library? I’d like to run airflow on > python3, and this is causing some issues. The hdfs_hook script is > referenced in the sensors module. > > Any suggestions? > > Thanks, > Cindy > >> > > > >
Re: Python3 and sensors module
Snakebite is not py3 compatible. We should move to libhdfs. Happy to take patches! B. Verstuurd vanaf mijn iPad > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis > het volgende geschreven: > > Here is a sample of what I am seeing. > I am running on ubuntu, python 3.5. This is a subset of the full message. > Once it went into the snakebite package, checked and the library wasn’t > supported in python3. > …. > File "/usr/local/airflow/dags/dim_date.py", line 5, in >from airflow.operators.sensors import S3KeySensor > File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", > line 34, in >from airflow.hooks.hdfs_hook import HDFSHook > File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", > line 20, in >from snakebite.client import Client, HAClient, Namenode, AutoConfigClient > File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473 >baseTime = min(time * (1L << retries), cap); >^ > > >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis >> wrote: >> >> Hi, >> >> Yes, it looks like all of the other sensors will work under Python3, but not >> hdfs. I am planning to use the s3 sensor, which is wrapped up in the >> sensors.py module. My issue is that when I test my dag or airflow >> installation under python3, I get errors about the hdfs_hook, which I’m not >> using. Other then creating my own version of the sensors.py file and >> removing the hdfs related functions/libraries, is there any thing else I can >> do to work around this? >> >> >> >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko wrote: >>> >>> Hi Cindy, >>> >>> The other sensors should work under Python3. We try to support Python3 as >>> much as possible, but sometimes libraries are used that are not compatible. >>> Could you describe what you are running into? >>> >>> Cheers, Fokko >>> >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis : >>> Hi, Are there any plans to update the HDFS_hook.py script to remove the reference to the snakebite python library? I’d like to run airflow on python3, and this is causing some issues. The hdfs_hook script is referenced in the sensors module. Any suggestions? Thanks, Cindy >> >
Re: Python3 and sensors module
Here is a sample of what I am seeing. I am running on ubuntu, python 3.5. This is a subset of the full message. Once it went into the snakebite package, checked and the library wasn’t supported in python3. …. File "/usr/local/airflow/dags/dim_date.py", line 5, in from airflow.operators.sensors import S3KeySensor File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", line 34, in from airflow.hooks.hdfs_hook import HDFSHook File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", line 20, in from snakebite.client import Client, HAClient, Namenode, AutoConfigClient File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473 baseTime = min(time * (1L << retries), cap); ^ > On May 16, 2018, at 9:24 AM, Cindy Rottinghuis > wrote: > > Hi, > > Yes, it looks like all of the other sensors will work under Python3, but not > hdfs. I am planning to use the s3 sensor, which is wrapped up in the > sensors.py module. My issue is that when I test my dag or airflow > installation under python3, I get errors about the hdfs_hook, which I’m not > using. Other then creating my own version of the sensors.py file and > removing the hdfs related functions/libraries, is there any thing else I can > do to work around this? > > > >> On May 16, 2018, at 1:33 AM, Driesprong, Fokko wrote: >> >> Hi Cindy, >> >> The other sensors should work under Python3. We try to support Python3 as >> much as possible, but sometimes libraries are used that are not compatible. >> Could you describe what you are running into? >> >> Cheers, Fokko >> >> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis : >> >>> Hi, >>> >>> Are there any plans to update the HDFS_hook.py script to remove the >>> reference to the snakebite python library? I’d like to run airflow on >>> python3, and this is causing some issues. The hdfs_hook script is >>> referenced in the sensors module. >>> >>> Any suggestions? >>> >>> Thanks, >>> Cindy >
Re: Python3 and sensors module
Hi Cindy, The other sensors should work under Python3. We try to support Python3 as much as possible, but sometimes libraries are used that are not compatible. Could you describe what you are running into? Cheers, Fokko 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis : > Hi, > > Are there any plans to update the HDFS_hook.py script to remove the > reference to the snakebite python library? I’d like to run airflow on > python3, and this is causing some issues. The hdfs_hook script is > referenced in the sensors module. > > Any suggestions? > > Thanks, > Cindy