Hi George, On Mon, Jun 12, 2017 at 10:02:16AM +0300, George . wrote: > Hi, > I've discovered following strange issue with http_slice_module > If I have a named location for internal 302 redirect and caching one slice > makes further request for whole object to brake upstream redirected request > (missing Rage header, see frame 254 in the attached capture > slice_redirect_problem.pcapng > <https://drive.google.com/a/ucdn.com/file/d/0ByZ2nt00gtJ2NmtqVUU3OVozNXM/view?usp=drive_web> > ).
What happens is: - client requests 0-4m - nginx creates the request for the 1st slice and proxies it to 8081 - after receiving 302, the request is redirected to @fetch_from_redirected_origin and the first slice is saved in the cache Note that in @fetch_from_redirected_origin there's a completely separate slice context. By this time nginx only knows what client sent. Previous slice context is completely lost as well as all other modules' contexts. Coincidentally, it does what you expect because only the first slice was requested. Then you request the entire file: - client request the entire file - first slice is sent from the cache - nginx creates a subrequest for the 2nd slice: 4m-8m and proxies it to 8081 - after receiving 302, the subrequest is redirected to @fetch_from_redirected_origin After the redirect nginx does not have any idea that it should fetch the second slice. Moreover, the $slice_range variable is not filled with actual range when first accessed in a subrequest (after error_page redirect it looks like the first access), so it remains empty. That's why the entire file is requested. But even if the variable was valid, that would still be bad since the slice context is lost after error_page redirect. You would get the whole file here instead of 4m-8m range. The takeaway is you should avoid using the slice module with redirects (error_page, X-Accel-Redirect) for fetching slices. Instead you should proxy directly to the origin server. > If there is no cached slice everything is okey (2nd capture > slice_redirect_no_problem.pcapng > <https://drive.google.com/a/ucdn.com/file/d/0ByZ2nt00gtJ2SUpnc2VVbzBKdWc/view?usp=drive_web> > ) No, it's not ok. The first redirect to @fetch_from_redirected_origin leads to caching all file slices instead of the first one. > Problem appears in main branch and also nginx/1.12 ... and may be in all > versions > > nginx version: nginx/1.13.2 > built by gcc 4.9.2 (Debian 4.9.2-10) > configure arguments: --prefix=/home/george/run/nginx_hg > --with-http_slice_module > > > > > nginx.conf > user cdnuser cdnuser; > worker_processes 1; > > error_log logs/error.log debug; > > events { > worker_connections 1024; > } > > > http { > include mime.types; > default_type application/octet-stream; > > > sendfile on; > tcp_nopush on; > > proxy_cache_path /home/george/run/nginx_hg/cache/ > keys_zone=zone_uid_default:4m levels=2:1 inactive=360d max_size=18329m; > > # our redirecting origin > server { > listen 8081; > > return 302 $scheme://127.0.0.1:8082$request_uri; > } > > # our final origin > server { > listen 8082; > add_header Cache-Control "max-age=3600"; > root /home/george/run/nginx_hg/root; > } > > server { > listen 8080; > server_name localhost; > > recursive_error_pages on; > proxy_intercept_errors on; > > > location / { > slice 4m; > proxy_cache zone_uid_default; > proxy_cache_key $uri$is_args$args$slice_range; > proxy_set_header Range $slice_range; > > proxy_pass http://localhost:8081; > > error_page 301 302 307 = @fetch_from_redirected_origin; > } > > location @fetch_from_redirected_origin { > slice 4m; > > internal; > > set $my_upstream_http_location $upstream_http_location; > > proxy_cache zone_uid_default; > proxy_cache_key $uri$is_args$args$slice_range; > proxy_set_header Range $slice_range; > > proxy_pass $my_upstream_http_location; > } > } > } > > > How to reproduce: > > 1. Create some empty object in our emulated origin > mkdir /home/george/run/nginx_hg/root > dd if=/dev/zero of=/home/george/run/nginx_hg/root/some_object bs=64M > count=1 > > 2. Ask our caching proxy for one 4m slice, so it will be cached > curl -v -r 0-4194303 "http://127.0.0.1:8080/some_object"; --header "Host: > localhost" -o /dev/null > > 3. See it really there > george@george ~/run/nginx_hg $ head > /home/george/run/nginx_hg/cache/81/c/00214df7041ea53dd335ed5b055bfc81 > Ļ:Y˩:Y��:YVʜ�r � "593aa9cb-4000000" > KEY: /some_objectbytes=0-4194303 > HTTP/1.1 206 Partial Content > Server: nginx/1.13.2 > Date: Fri, 09 Jun 2017 14:16:20 GMT > Content-Type: application/octet-stream > Content-Length: 4194304 > Last-Modified: Fri, 09 Jun 2017 13:59:39 GMT > Connection: close > ETag: "593aa9cb-4000000" > > 4. This time request the whole object > curl -v "http://127.0.0.1:8080/some_object"; --header "Host: localhost" -o > /dev/null > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel@nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Roman Arutyunyan _______________________________________________ nginx-devel mailing list nginx-devel@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel
