[
https://issues.apache.org/jira/browse/ARROW-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215645#comment-16215645
]
ASF GitHub Bot commented on ARROW-1555:
---------------------------------------
benjigoldberg commented on a change in pull request #1240: ARROW-1555 [Python]
Implement Dask exists function
URL: https://github.com/apache/arrow/pull/1240#discussion_r146360160
##########
File path: python/pyarrow/filesystem.py
##########
@@ -251,6 +251,10 @@ def isfile(self, path):
def delete(self, path, recursive=False):
return self.fs.rm(path, recursive=recursive)
+ @implements(FileSystem.exists)
+ def exists(self, path):
+ return os.path.exists(path)
Review comment:
@wesm @johnjiang I added a new attribute to `FileSystem` which simply
indicates whether the FileSystem is an object-store or a file-store. Then added
an additional param to the mkdir conditional in `write_to_dataset` to check if
the `FileSystem` is a File store before attempting to call `mkdir`. I imagine
that tiny method could come in handy to distinguish other features between
object/file stores later. Let me know if that addresses your concerns,
otherwise happy to try something else out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] write_to_dataset on s3
> -------------------------------
>
> Key: ARROW-1555
> URL: https://issues.apache.org/jira/browse/ARROW-1555
> Project: Apache Arrow
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Young-Jun Ko
> Assignee: Florian Jetter
> Priority: Trivial
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> When writing a arrow table to s3, I get an NotImplemented Exception.
> The root cause is in _ensure_filesystem and can be reproduced as follows:
> import pyarrow
> import pyarrow.parquet as pqa
> import s3fs
> s3 = s3fs.S3FileSystem()
> pqa._ensure_filesystem(s3).exists("anything")
> It appears that the S3FSWrapper that is instantiated in _ensure_filesystem
> does not expose the exist method of s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
