[
https://issues.apache.org/jira/browse/ARROW-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510063#comment-17510063
]
Joris Van den Bossche commented on ARROW-15910:
-----------------------------------------------
OK, so it's the same bug in gcsfs from the mailing list thread that you are
running into: because gcsfs returns something different the first vs subsequent
times, pyarrow is confused about whether it is querying a file or directory. I
know that gcsfs uses size 0 empty files to mimic directories (since the
filesystem itself doesn't have that concept), but I think it can still be
expected from gcsfs to return a consistent info for the same file path.
I would open an issue about this on the gcsfs side (I don't think that happened
after the previous mailing list thread).
If you first do such an initial "info" call, and then do the {{read_table}}
call (which now should directly think the file path is a directory), does it
then work correctly?
> [Python] pyarrow.parquet.read_table either returns FileNotFound or
> ArrowInvalid
> -------------------------------------------------------------------------------
>
> Key: ARROW-15910
> URL: https://issues.apache.org/jira/browse/ARROW-15910
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet, Python
> Affects Versions: 6.0.1, 7.0.0
> Environment: GCP JupyterLab notebooks
> Reporter: Callista Rogers
> Priority: Major
>
> running below results in {{"GetFileIno() yielded path
> 'myBucket/features/MyParquet.parquet/year=2022/part-0019.snappy.parquet'
> which is outside base dir 'gs://myBucket/features/MyParquet.parquet/' "}}
> {code}
> import pyarrow.parquet as pq
> import gcsfs
> file_path="gs://myBucket/features/MyParquet.parquet/"
> fs=gcsfs.GCSFileSystem()
> table=pq.read_table(file_path,filesystem=fs)
> {code}
> Removing the gs:// from file_path results in a {{FileNotFoundError}}. Any
> variation of / or // at the beginning of the path gives me the 'outside base
> dir' error.
> I also ran the below and got valid results using both file_path patterns, so
> I know it finds the path just fine.
> {code}
> from pyarrow.fs import FileSelector, PyFileSystem, FSSpecHandler
> filesys = PyFileSystem(FSSpecHandler(fs))
> selector = FileSelector(file_path, recursive=True)
> filesys.get_file_info(selector)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
