[
https://issues.apache.org/jira/browse/ARROW-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612694#comment-15612694
]
Wes McKinney commented on ARROW-350:
------------------------------------
There is no reliable support for libhdfs3 -- it was proposed as a patch to
Hadoop and rejected. Meanwhile, a new official C++ HDFS client is in the works,
see HDFS-8707. If there is a bug we cannot really count on the Apache Hadoop
community to fix it.
Because libhdfs3 is mostly API compatible with libhdfs if someone will do the
work we could add an option to switch between libhdfs and libhdfs3 in the
HdfsClient factory function.
> Add Kerberos support to HDFS shim
> ---------------------------------
>
> Key: ARROW-350
> URL: https://issues.apache.org/jira/browse/ARROW-350
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Christopher Aycock
> Assignee: Christopher Aycock
>
> Speaking with [~wesmckinn], it would be really helpful to have Kerberos
> support in our HDFS logic. This should be straightforward; I would just need
> to switch to {{hdfsBuilderConnect()}} in the shim.
> On a side note, is there a reason we aren't using Pivotal's libhdfs3? It uses
> RPCs natively rather than JNI.
> https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3
> Dask has Python wrappers for this.
> https://github.com/dask/hdfs3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
