Re: [go-nuts] Go Optimizations for IO Intensive programs

On Sat, Feb 11, 2017 at 1:58 AM wrote: > Are there any other methodologies where I can optimize compresssion part and upload part Many ISPs have upload speed several times slower than download. The symptoms you described would be consistent with that. But maybe you

Re: [go-nuts] Go Optimizations for IO Intensive programs

On Sat, Feb 11, 2017 at 1:58 AM wrote: > Are there any other methodologies where I can optimize compresssion part > and upload part > > Start by establishing a baseline. If you just grab the files from S3 and do nothing to to them, how fast is that? If you are not

[go-nuts] Go Optimizations for IO Intensive programs

Just confirming, you are uploading each file as you're downloading it. Not downloading completely, then decompressing completely, followed by uploading. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop

[go-nuts] Go Optimizations for IO Intensive programs

Hello, I have written a GO program which downloads a 5G compressed CSV from Amazon S3, decompresses it and uploads the decompressed CSV (20G) to Amazon S3. Amazon S3 provides a default concurrent uploader/downloader and I am using a multithreaded approach to download files in parallel,