[
https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987853#comment-16987853
]
Fabian Höring edited comment on ARROW-7309 at 12/4/19 1:17 PM:
---------------------------------------------------------------
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = fs.get_cache_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doing this in my code.
was (Author: fhoering):
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = fs.get_cache_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doign this in my code.
> [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
> -------------------------------------------------------------------------
>
> Key: ARROW-7309
> URL: https://issues.apache.org/jira/browse/ARROW-7309
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 0.15.1
> Reporter: Fabian Höring
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
> - libhdfs already supports injecting the scheme and will automatically
> resolve federation in
> fs = FileSystem#get(URI, conf, ugi)
> - works with Hadoop 2/3
> see:
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
