prantogg opened a new pull request, #71: URL: https://github.com/apache/sedona-spatialbench/pull/71
This PR enables direct S3 streaming for generated data, eliminating local storage requirements for large-scale dataset generation. S3 URIs in `--output-dir` are automatically detected and routed to a streaming multipart upload implementation. ```bash spatialbench --scale-factor 1000 --output-dir s3://my-bucket/spatialbench/sf1000 ``` - Implements streaming S3 writer with `Write` trait compatibility. - Uses S3 multipart upload API with 32MB parts for efficient streaming. Small files (<5MB) use direct PUT requests. - Multipart completion handled asynchronously via Tokio runtime. Built on `object_store` crate's `AmazonS3Builder` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
