I had the same problem and I wrote a patch to reuse the file with I already have in tmp directory for the second stream (and for all streams before the file is completely cached). Unfortunately I cannot share it but can give you an idea how to do it.
On Tue, Aug 27, 2013 at 8:43 PM, Alex Garzão <[email protected]> wrote: > Hello Wandenberg, > > Thanks for your reply. > > Using proxy_cache_lock, when the second request arrive, it will wait > until the object is complete in the cache (or until > proxy_cache_lock_timeout expires). But, in many cases, my upstream has > a really slow link and NGINX needs more than 30 minutes to download > the object. In practice, probably I will see a lote of parallel > downloads from the same object. > > Someone has other idea? Or I'm wrong about proxy_cache_lock? > > Regards. > -- > Alex Garzão > Projetista de Software > Azion Technologies > alex.garzao (at) azion.com > > > On Mon, Aug 26, 2013 at 6:58 PM, Wandenberg Peixoto > <[email protected]> wrote: > > Try to use the proxy_cache_lock configuration, I think this is what you > are > > looking for. > > Don't forget to configure the proxy_cache_lock_timeout to your use case. > > > > On Aug 26, 2013 6:54 PM, "Alex Garzão" <[email protected]> wrote: > >> > >> Hello guys, > >> > >> This is my first post to nginx-devel. > >> > >> First of all, I would like to congratulate NGINX developers. NGINX is > >> an amazing project :-) > >> > >> Well, I'm using NGINX as a proxy server, with cache enabled. I noted > >> that, when two (or more) users trying to download the same object, in > >> parallel, and the object isn't in the cache, NGINX download them from > >> the upstream. In this case, NGINX creates one connection to upstream > >> (per request) and download them to temp files. Ok, this works, but, in > >> some situations, in one server, we saw more than 70 parallel downloads > >> to the same object (in this case, an object with more than 200 MB). > >> > >> If possible, I would like some insights about how can I avoid this > >> situation. I looked to see if it's just a configuration, but I didn't > >> find nothing. > >> > >> IMHO, I think the best approach is share the temp file. If possible, I > >> would like to known your opinions about this approach. > >> > >> I looked at the code in ngx_http_upstream.c and ngx_http_proxy.c, and > >> I'm trying to fix the code to share the temp. I think that I need to > >> do the following tasks: > >> > >> 1) Register the current downloads from upstreams. Probably I can > >> address this with a rbtree, where each node has the unique object id > >> and a list with downstreams (requests?) waiting for data from the > >> temp. > >> > >> 2) Disassociate the read from upstream from the write to downstream. > >> Today, in the ngx_event_pipe function, NGINX reads from upstream, > >> writes to temp, and writes to downstream. But, as I can have N > >> downstreams waiting data from the same upstream, probably I need to > >> move the write to downstream to another place. The only way I think is > >> implementing a polling event, but I know that this is incorrect > >> because NGINX is event based, and polling waste a lote of CPU. > >> > >> 3) When I know that there more data in temp to be sent, which function > >> I must use? ngx_http_output_filter? > >> > >> Suggestions will welcome :-) > >> > >> Thanks people! > >> > >> -- > >> Alex Garzão > >> Projetista de Software > >> Azion Technologies > >> alex.garzao (at) azion.com > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> [email protected] > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > _______________________________________________ > > nginx-devel mailing list > > [email protected] > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > [email protected] > http://mailman.nginx.org/mailman/listinfo/nginx-devel >
_______________________________________________ nginx-devel mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx-devel
