> You might also try the GCS filesystem (released with 7.0.0) instead of going through fsspec.
I don't think the native GCS filesystem support is complete in 7.0.0, I think if you are willing to compile from the latest commit in the repo it might be useable. On Wed, Feb 23, 2022 at 11:41 AM Weston Pace <[email protected]> wrote: > I'm pretty sure GCS is similar to S3 in that there is no such thing as > a "directory". Instead a directory is often emulated by an empty > file. Note that the single file being detected is hires-sonde/ (with > a trailing slash). I'm pretty sure this is the convention for > creating mock directories. I'm guessing, if there were multiple > files, we would work ok because we just skip the empty files. > > So perhaps this is a problem unique to gcsfs/fsspec and trying to read > an "empty directory". > > You might also try the GCS filesystem (released with 7.0.0) instead of > going through fsspec. > > On Wed, Feb 23, 2022 at 2:23 AM Joris Van den Bossche > <[email protected]> wrote: > > > > > > On Mon, 21 Feb 2022 at 00:04, Kelton Halbert <[email protected]> > wrote: > >> > >> Hello, > >> > >> I’ve been learning and working with PyArrow recently for a project to > store some atmospheric science data as part of a partitioned dataset, and > recently the dataset class with the fsspec/gcsfs filesystem has started > producing a new error. > > > > > > Hi Kelton, > > > > One more question: you say that this started producing a new error, so I > suppose this worked a while ago? Do you know if you updated some packages > (eg gcsfs or fsspec) since then? Or something else that might have changed? > > > > Joris > > >
