Re: Help with writing/reading from s3

2022-08-03 Thread Li Jin
Thanks! Removing the "gs://" prefix indeed fixes it. On Tue, Aug 2, 2022 at 4:01 PM Will Jones wrote: > Hi Li Jin, > > I'm not sure yet what changed, but I believe you can fix that error simply > by omitting the scheme prefix from the URI and just use the page when > loading the dataset. Here's

Re: Help with writing/reading from s3

2022-08-02 Thread Will Jones
Hi Li Jin, I'm not sure yet what changed, but I believe you can fix that error simply by omitting the scheme prefix from the URI and just use the page when loading the dataset. Here's my repro: import pyarrow as pa import pyarrow.dataset as ds from pyarrow.fs import S3FileSystem s3fs = S3FileSys

Help with writing/reading from s3

2022-08-01 Thread Li Jin
Hello! We recently updated Arrow to 7.0.0 and hit some error with our old code (Details below). I wonder if there is a new way to do this with the current version? import pyarrow import pyarrow.parquet as pq df = pd.DataFrame({"aa": [1, 2, 3], "bb": [1, 2, 3]}) uri = "gs://amp_bucket_liao/tr