[
https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927765#comment-16927765
]
ASF GitHub Bot commented on AIRFLOW-2697:
-----------------------------------------
stale[bot] commented on pull request #3560: [AIRFLOW-2697] Drop snakebite in
favour of hdfs3
URL: https://github.com/apache/airflow/pull/3560
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drop snakebite in favour of pyarrow
> -----------------------------------
>
> Key: AIRFLOW-2697
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2697
> Project: Apache Airflow
> Issue Type: Improvement
> Affects Versions: 1.9.0
> Reporter: Julian de Ruiter
> Assignee: Julian de Ruiter
> Priority: Blocker
> Fix For: 2.0.0
>
>
> The current HdfsHook relies on the snakebite library, which is unfortunately
> not compatible with Python 3. To add Python 3 support for the HdfsHook
> requires switching to a different library for interacting with HDFS. The
> hdfs3 library is an attractive alternative, as it supports Python 3 and seems
> to be stable and relatively well supported.
> Update: hdfs3 doesn't get any updates anymore. The best library right now
> seems to be pyarrow: https://arrow.apache.org/docs/python/filesystems.html
> Therefore I would like to upgrade to pyarrow instead of hdfs3.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
