Mokubyow commented on issue #10634: URL: https://github.com/apache/arrow/issues/10634#issuecomment-872305153
@westonpace I'm 100% sure I'm using the exact same `path` variable for each of these writing methods. I've annotated what works and what doesn't below. I can also confirm that the `path` variable returned from `fs.FileSystem.from_uri(target_uri)` does indeed start with the bucket name like so: `my-existing-bucket/new_prefix/` ``` filesystem, path = fs.FileSystem.from_uri(target_uri) # Throws InvalidLocationConstraint error ds.write_dataset(dataset, path, filesystem=filesystem, format="parquet") # Throws InvalidLocationConstraint error pq.write_to_dataset(dataset.to_table(), path, filesystem=filesystem) # Writes single file pq.write_table(dataset.to_table(), path, filesystem=filesystem) ``` Are there any solutions you can think of for writing a large dataset as many parquet files from pyarrow to unblock me while we debug? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
