Re: S3 Writes using SIG4 Authentication

I was looking at SIG4's documentation for S3 here earlier. The section on Chunked Upload confused me because it said I need to pass Content-Length header in the request. I now realize that I was using the terms `chunked

Re: S3 Writes using SIG4 Authentication

Multi-part upload with finalization seems like a good approach for this problem. Bhavin Thaker. On Wed, Mar 7, 2018 at 7:45 AM Naveen Swamy wrote: > Rahul, > IMO It is not Ok to write to a local file before streaming, you have to > consider security implications such as: >

Re: S3 Writes using SIG4 Authentication

Rahul, IMO It is not Ok to write to a local file before streaming, you have to consider security implications such as: 1) will your local file be encrypted(encryption at rest) 2) what happens if the process crashes, you will have to make sure the local file is deleted in failure and process exit

Re: S3 Writes using SIG4 Authentication

Hi Rahul, Rahul> Is it okay to buffer it to a local file, and then upload this file to S3 at the end? Short answer: Yes. S3 is object storage. See: https://en.m.wikipedia.org/wiki/Object_storage The granularity of access for object storage is an object. This is different from a filesystem or

Re: S3 Writes using SIG4 Authentication

Hi Chris, S3 doesn't support append calls. They promote the use of multipart uploads to upload large files in parallel, or when network reliability is an issue. Writing like a stream does not seem to be the purpose of multipart uploads. I looked into what the AWS SDK does (in Java). It buffers

Re: S3 Writes using SIG4 Authentication

it seems strange that s3 would make such a major restriction. there’s literally no way to incrementally write a file without knowing the size beforehand? some sort of separate append calls, maybe? On Tue, Mar 6, 2018 at 8:53 PM Rahul Huilgol wrote: > Hi everyone, > > I