I'm pretty sure GCS is similar to S3 in that there is no such thing as a "directory". Instead a directory is often emulated by an empty file. Note that the single file being detected is hires-sonde/ (with a trailing slash). I'm pretty sure this is the convention for creating mock directories. I'm guessing, if there were multiple files, we would work ok because we just skip the empty files.
So perhaps this is a problem unique to gcsfs/fsspec and trying to read an "empty directory". You might also try the GCS filesystem (released with 7.0.0) instead of going through fsspec. On Wed, Feb 23, 2022 at 2:23 AM Joris Van den Bossche <[email protected]> wrote: > > > On Mon, 21 Feb 2022 at 00:04, Kelton Halbert <[email protected]> wrote: >> >> Hello, >> >> I’ve been learning and working with PyArrow recently for a project to store >> some atmospheric science data as part of a partitioned dataset, and recently >> the dataset class with the fsspec/gcsfs filesystem has started producing a >> new error. > > > Hi Kelton, > > One more question: you say that this started producing a new error, so I > suppose this worked a while ago? Do you know if you updated some packages (eg > gcsfs or fsspec) since then? Or something else that might have changed? > > Joris >
