Re: downloading large files in chunks

The way I know it’s streaming is running the route. You’ll see the log entries (“Download Triggered” and “Writing File”) fairly close together. Then if you watch the filesystem, you’ll see the file size on disk growing. Also, I’m using default JVM parameters, so the heap isn’t big enough

Re: downloading large files in chunks

https://netty.io/4.0/api/io/netty/handler/codec/http/HttpChunkedInput.html That's why I thought the Camel Netty with chunked would only read the entire stream of the specified chunk size in. On Fri, Sep 2, 2016 at 10:11 AM, S Ahmed wrote: > I'm just the consumer

Re: downloading large files in chunks

Also, is there a way for me to test if the endpoint supports streaming? I'm on OSX so any open source tools to test this? On Fri, Sep 2, 2016 at 11:11 AM, S Ahmed wrote: > I'm just the consumer (downloading), the file can be anywhere like s3 or > centos.org! > > > > On

Re: downloading large files in chunks

I'm just the consumer (downloading), the file can be anywhere like s3 or centos.org! On Fri, Sep 2, 2016 at 11:09 AM, Brad Johnson wrote: > By the way S. Ahmed, do you have control of both ends of this I mean > client/server or are you just on the client/consumer

Re: downloading large files in chunks

By the way S. Ahmed, do you have control of both ends of this I mean client/server or are you just on the client/consumer side? On Fri, Sep 2, 2016 at 10:01 AM, Brad Johnson wrote: > Absolutely. Love to set up a VM for my server. I just had a "duh" moment > when

Re: downloading large files in chunks

Absolutely. Love to set up a VM for my server. I just had a "duh" moment when I did it. No harm, no foul. On Fri, Sep 2, 2016 at 10:00 AM, Quinn Stevenson < qu...@pronoia-solutions.com> wrote: > Sorry - I wanted to put in and example that worked, and download something > big to make sure it

Re: downloading large files in chunks

Sorry - I wanted to put in and example that worked, and download something big to make sure it was streaming. Hopefully you needed a new CentOS image :-) > On Sep 2, 2016, at 8:58 AM, Brad Johnson wrote: > > Neat. I accidentally clicked on the link and Chrome

Re: downloading large files in chunks

Neat. I accidentally clicked on the link and Chrome downloaded the ISO for me. Are you propagating Trojan horses here? Heh. On Fri, Sep 2, 2016 at 9:56 AM, Quinn Stevenson wrote: > I think something like this might work for you > > > > > > >

Re: downloading large files in chunks

Hmmm. That could be a problem if it doesn't actually chunk. I thought it read the entire chunk into memory before letting you read it. So if the chunk size is 10mb it would download that whole 10mb and then let you read, then fetch the next 10mb and let you read. But that may not be the case. I

Re: downloading large files in chunks

By the way, I don't know if you said or not but do you control both sides of this or just the consumer side? On Fri, Sep 2, 2016 at 9:51 AM, Brad Johnson wrote: > Hmmm. That could be a problem if it doesn't actually chunk. I thought it > read the entire chunk into

Re: downloading large files in chunks

I think something like this might work for you > On Sep 2, 2016, at 8:51 AM, Brad Johnson wrote: > > Hmmm. That could be a problem if it doesn't actually chunk. I thought it > read the entire chunk into memory before letting you read

Re: downloading large files in chunks

Brad, that page says this: "Notice Netty4 HTTP reads the entire stream into memory using io.netty.handler.codec.http.HttpObjectAggregator to build the entire full http message. But the resulting message is still a stream based message which is readable once." On Fri, Sep 2, 2016 at 10:26 AM, S

Re: downloading large files in chunks

Thanks. Just to be clear, I don't run the server where I am downloading the file. I want to download files that are very large, but stream them so they are not held in memory and then written to disk. I want to stream the download straight to a file and not hold the entire file in memory. Is

Re: downloading large files in chunks

http://camel.apache.org/netty4-http.html Look at netty and see if that works. It can control chunk size but it is also streaming in any case so you may not even need to be concerned about it. Brad On Thu, Sep 1, 2016 at 8:53 PM, S Ahmed wrote: > Does it have to be ftp,

Re: downloading large files in chunks

Does it have to be ftp, I just need http? On Thu, Sep 1, 2016 at 5:31 PM, Quinn Stevenson wrote: > Check out the section on the ftp component page about “Using a Local Work > Directory” (http://people.apache.org/~dkulp/camel/ftp2.html < >

Re: downloading large files in chunks

Check out the section on the ftp component page about “Using a Local Work Directory” (http://people.apache.org/~dkulp/camel/ftp2.html ) - I think that may be what you’re after. > On Sep 1, 2016, at 9:30 AM, S Ahmed wrote:

RE: downloading large files in chunks

Hello, Is there an example of how to download a large file in chunks and save the file as the file downloads. The goal is not to hold the entire file in memory and then save it to disk. Thanks.