[
https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk updated AIRFLOW-2697:
----------------------------------
Labels: gsoc gsoc2020 mentor (was: )
> Drop snakebite in favour of pyarrow
> -----------------------------------
>
> Key: AIRFLOW-2697
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2697
> Project: Apache Airflow
> Issue Type: Improvement
> Affects Versions: 1.9.0
> Reporter: Julian de Ruiter
> Assignee: Julian de Ruiter
> Priority: Blocker
> Labels: gsoc, gsoc2020, mentor
> Fix For: 2.0.0
>
>
> The current HdfsHook relies on the snakebite library, which is unfortunately
> not compatible with Python 3. To add Python 3 support for the HdfsHook
> requires switching to a different library for interacting with HDFS. The
> hdfs3 library is an attractive alternative, as it supports Python 3 and seems
> to be stable and relatively well supported.
> Update: hdfs3 doesn't get any updates anymore. The best library right now
> seems to be pyarrow: https://arrow.apache.org/docs/python/filesystems.html
> Therefore I would like to upgrade to pyarrow instead of hdfs3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
