Order of directive call during request handling
I am not able to get the order in which the directive handlers are called when a request comes. I have a directive to connect to the database and the other to do put operation. In the conf file, i write the connect directive first, but when request comes, the connect directive handler is not called, and the put handler is called first. Can you please help me with the order in which they are called? ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Accessing the location configuration of module 2 during post configuration processing of module 1 for a particular server
Hello Maxim
Thanks for the response.
As previously suggested, you should consider using merge callbacksto
validate configuration instead.
The requirement is *not* to validate the configuration. The requirement is to
find the final value set by one of the directives once the configuration has
been parsed. e..g lets say we have a directive my_set_flag that sets a value to
0 or 1.
So if in configuration we have
location \ {
my_set_flag 0;
---
my_set_flag 1;
---
my_set_flag 0;
}
then the merge callback will be called thrice for each invocation of the
directive. Let's assume the logic is to set a variable with whatever the value
of the directive was. So once parsing completes, the value of the variable
should be 0.
I can get this value during request processing. However, I cannot get this
value *after* the parsing of the configuration has completed. What is the
nginx recommended way to get this value. As noted in earlier post, I am not
seeing the correct values in post configuration - and so perhaps that is not
the right way.
Thanks for your continued inputs
Hello!
On Thu, Jun 12, 2014 at 01:46:58PM +0800, Rv Rv wrote:
[...]
In the post configuration, I see that flag is not properly set
but somestring is. Flag is properly set during request
processing though.
Are the values set during processing of a directive in location
struct guaranteed to be set by the time post configuration is
executed?
When is the time that one can check for the values set during
configuration. I need to test these values to ensure that they
are sane when nginx is executed with -t option
Again: there are lots of location configurations, and by trying to
access them at postconfiguration callback you are likely checking
a wrong one. Note that even a simple config with a single
location in a single server{} block, like this:
http { server { location { ... } } }
has 3 location configurations for each http module.
As previously suggested, you should consider using merge callbacks
to validate configuration instead.
--
Maxim Dounin
http://nginx.org/
On Thursday, 12 June 2014 11:16 AM, Rv Rv rvrv7...@yahoo.com wrote:
Hello Maxim
Thanks for your response. Here is a related query.
Say in module 1 I have a
typedef struct {
int flag;
ngx_str somestring;
} module1;
flag gets initialized with the following code
{ ngx_string(module1_directive),
NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
ngx_conf_set_flag_slot,
NGX_HTTP_LOC_CONF_OFFSET,
offsetof(module1,configured),
NULL },
somestring gets iniitialized with a handler written in the module (i.e not
ngx_conf_set_flag_slot or any inbuilt handler).
In the post configuration, I see that flag is not properly set but somestring
is. Flag is properly set during request processing though.
Are the values set during processing of a directive in location struct
guaranteed to be set by the time post configuration is executed?
When is the time that one can check for the values set during configuration. I
need to test these values to ensure that they are sane when nginx is executed
with -t option
Thanks
Hello!
On Tue, Jun 10, 2014 at 02:09:13AM +0800, Rv Rv wrote:
How do we access the configuration of a an unrelated module in a
given module. This may be required for example to check if the
directives pertaining to module 2 were specified in location for
a particular server that has directives for module 1 in its
configuration.
I don't think it's something you should do at postconfiguration -
location structure is complex and not easily accessible. There
are location configuration merge callbacks where you are expected
to work with location configs and, in particular, can use
ngx_http_conf_get_module_loc_conf() macro to access a
configuration of other modules (note though, that order of modules
may be important in this case).
[...]
I did not find any documentation on how the configuration is stored within
nginx using these structs?
It's under src/, in C language.
I would rather say it's not a part of the API, and you'd better
avoid using it directly.
--
Maxim Dounin
http://nginx.org/
--
On Monday, 9 June 2014 11:39 PM, Rv Rv rvrv7...@yahoo.com wrote:
How do we access the configuration of a an unrelated module in a given module.
This may be required for example to check if the directives pertaining to
module 2 were specified in location for a particular server that has directives
for module 1 in its configuration.
From what I understand, code similar to this can be used
/* Get http main configuration */
cmcf = ctx-main_conf[ngx_http_core_module.ctx_index];
/* Get the list of servers */
cscfp = cmcf-servers.elts;
/* Iterate through the list */
for (s = 0; s cmcf-servers.nelts; s++) {
/* Problem : how to get the configuration of module 2*/
cscfp[s]-ctx-loc_conf[module2.ctx_index];-
does not yield the correct location
nginx +
Sorry to ask this, but does using this *require* a support contract? I'm interested in the streaming stuff, but it's not worth $1300/yr to me... Cheers, Steve -- Steve Holdoway BSc(Hons) MIITP http://www.greengecko.co.nz Linkedin: http://www.linkedin.com/in/steveholdoway Skype: sholdowa ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
RE: GeoIP FirstNonPrivateXForwardedForIP
Hi, Thanks for your reply. I have already tried http://nginx.org/en/docs/http/ngx_http_geoip_module.html#geoip_proxy But this needs a list of subnets / networks to be whitelisted first as a trusted source. I do not (Can not) have a list of such networks as they can be intermediate proxy of any company. Eg : Google chrome on smartphone uses Google compression proxy in between before reaching the actual server where website is hosted. Opera mini also does the same and similarly don't know who all does it. So I can not have a list of all trusted networks. You cannot trust X-F-F headers of untrusted third party networks and proxies, otherwise everyone can spoof whatever remote IP they want. Don't do this. Lukas ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: RE: GeoIP FirstNonPrivateXForwardedForIP
Hi Lucas, Noted! Agreed! How do I tell nginx to do GeoIP on FirstNonPrivateXForwardedForIP ? Regards, Keyur Posted at Nginx Forum: http://forum.nginx.org/read.php?2,250823,250876#msg-250876 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
nginx -s reload problem
hi all I use a endless rtmp stream /usr/local/nginx/wsgi/ffmpeg -i haha.mp4 -c:v libx264 -b:v 500k -c:a copy -f flv rtmp://172.16.205.50:1936/publish/you as you known, if I use nginx -s reload , then I got a lot of nginx: worker process is shutting down pplive 15355 13642 0 14:56 ?00:00:00 nginx: worker process is shutting down pplive 15356 13642 0 14:56 ?00:00:00 nginx: worker process is shutting down pplive 15357 13642 0 14:56 ?00:00:00 nginx: worker process is shutting down pplive 15358 13642 0 14:56 ?00:00:00 nginx: worker process is shutting down pplive 15359 13642 0 14:56 ?00:00:00 nginx: worker process is shutting down pplive 15360 13642 0 14:56 ?00:00:00 nginx: worker process is shutting down because the connection will not quit, so nginx: worker process is shutting down more and more so How to aviod this status using nginx -s reload -- Best Regards, Harold Miao ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
How to redirect error page (html) self built html error page
Hi everyone, Sorry for my english. My scenario like below ; I setup my webserver in cloud and I have my ERP server running at different DC. The connection from cloud to DC is very fast, no interuption at all. When my customer access www.example.com it will show the website that i build in cloud (Webserver). Each customer have vpn access when they subscribe with us. So, when they wat to use ERP system, they need to connect to vpn first and they can go through https://erp.example.com. I configured NGINX to allow only server subnet IP and tunnel network subnet for VPN. The problem is when they click to login page, it will show error 403 not my html error page. I want it to trigger error page that I build in HTML. My problem is it not show the error page (html) that I build. It wil show error 403. Please anyone help me Posted at Nginx Forum: http://forum.nginx.org/read.php?2,250878,250878#msg-250878 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: How to redirect error page (html) self built html error page
Hi,
Am Montag, 16. Juni 2014, 05:14:59 schrieb mknazri:
Hi everyone,
Sorry for my english. My scenario like below ;
I setup my webserver in cloud and I have my ERP server running at different
DC. The connection from cloud to DC is very fast, no interuption at all.
When my customer access www.example.com it will show the website that i
build in cloud (Webserver). Each customer have vpn access when they
subscribe with us. So, when they wat to use ERP system, they need to connect
to vpn first and they can go through https://erp.example.com. I configured
NGINX to allow only server subnet IP and tunnel network subnet for VPN. The
problem is when they click to login page, it will show error 403 not my
html error page. I want it to trigger error page that I build in HTML.
My problem is it not show the error page (html) that I build. It wil show
error 403.
How did you configure your error_page?
Here's a snippet I use
error_page 500 502 503 504 =200 /maintenance.html;
location /maintenance.html {
root /etc/nginx;
}
You can add 403 and 404 status.
Please anyone help me
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,250878,250878#msg-250878
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Regards, Axel
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: How to redirect error page (html) self built html error page
Hi Axel,
This is my nginx conf in /etc/nginx/sites-available/erp;
upstream webserver {
server 127.0.0.1:8078 weight=1 fail_timeout=300s;
}
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://127.0.0.1:5000;
proxy_set_headerHost$host;
proxy_set_headerX-Real-IP $remote_addr;
proxy_set_headerX-Forwarded-for $remote_addr;
port_in_redirect off;
proxy_connect_timeout 300;
}
}
server {
listen 80;
server_name_;
# server_name erp.example.com;
# Strict Transport Security
add_header Strict-Transport-Security max-age=2592000;
rewrite ^/.*$ https://$host$request_uri? permanent;
}
server {
# server port and name
listen443 default;
server_name erp.example.com;
# Specifies the maximum accepted body size of a client request,
# as indicated by the request header Content-Length.
client_max_body_size 200m;
# ssl log files
access_log/var/log/nginx/openerp-access.log;
error_log/var/log/nginx/openerp-error.log;
# ssl certificate files
ssl on;
ssl_certificate/etc/ssl/nginx/server.crt;
ssl_certificate_key/etc/ssl/nginx/server.key;
# add ssl specific settings
keepalive_timeout60;
# limit ciphers
ssl_ciphersHIGH:!ADH:!MD5;
ssl_protocolsSSLv3 TLSv1;
ssl_prefer_server_cipherson;
# increase proxy buffer to handle some ERP web requests
proxy_buffers 16 64k;
proxy_buffer_size 128k;
allow 10.8.8.0/24;
allow 10.9.8.0/24;
deny all;
error_page 403 /error403.html;
location = /index.html {
index index.html;
root /usr/share/nginx/www;
allow all;
}
location / {
proxy_passhttp://webserver;
# force timeouts if the backend dies
proxy_next_upstream error timeout invalid_header http_500 http_502
http_503;
# set headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
# Let the OpenERP web service know that we're using HTTPS,
otherwise
# it will generate URL using http:// and not https://
proxy_set_header X-Forwarded-Proto https;
# by default, do not forward anything
proxy_redirect off;
allow all;
}
# cache some static data in memory for 60mins.
# under heavy load this should relieve stress on the ERP web interface a
bit.
location ~* /web/static/ {
proxy_cache_valid 200 60m;
proxy_bufferingon;
expires 864000;
proxy_pass http://webserver;
}
}
For your information, 10.8.8.0/24 is tunnel vpn that I allowed and
10.9.8.0/24 is server subnet also allowed.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,250878,250880#msg-250880
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: tmp directory filling up
On 11 Jun 2014, at 23:23, Maxim Dounin mdou...@mdounin.ru wrote: Hello! On Wed, Jun 11, 2014 at 10:58:47AM -0400, Tatonka wrote: Hi, I have a rails application that is hosted through nginx and passenger. In this application I want provide very large files for the users to download (2GB) using send_file .. which is working just fine on the development and staging system. On the production system however the system tmp directory is limited to 1GB (separately mounted disk). When triggering a download, the tmp folder quickly fills up and the download breaks once it is completely full. I already moved passengers /tmp directory to a new location but could find how to do the same for nginx (I did set $tmp and $tmpdir with no effect). When looking into the /tmp directory however, I cannot find any large files that would explain what is happening, nevertheless, df reports it is filling up at the same time .. Lastly .. I also specified the proxy_temp_path directive in the nginx config. Again with no effect. The proxy_temp_path is related to the problem, but it's for proxy, not for passenger, and it's expected that it has no effect in your case. Is there any way to specify which directory nginx uses for its tmp data? Is nginx even the culprit here? That's not about nginx, but rather about passenger module for nginx. Last time I checked, passenger module for nginx implemented its own protocol for the upstream module (like proxy/fastcgi/etc), and should have its own ..._temp_path directive, as well as ..._max_temp_file_size and so on. It’s still true: https://github.com/phusion/passenger/blob/master/ext/nginx/Configuration.c#L55-L57 It uses NGX_HTTP_PROXY_TEMP_PATH which is set at configure stage. ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: ngx_http_process_header_line function in source code
Hello!
On Sun, Jun 15, 2014 at 06:03:27PM -0400, alayim wrote:
Hi,
I'm browsing through the source code of the project, and looked at
ngx_http_request.c where the function ngx_http_process_header_line() creates
a pointer to a pointer to a large struct(ngx_http_request_t) containing a
smaller one(ngx_http_headers_in_t), containing yet another one.
ngx_http_process_header_line(ngx_http_request_t *r, ngx_table_elt_t *h,
ngx_uint_t offset) {
ph = (ngx_table_elt_t **) ((char *) r-headers_in + offset);
// ... then check if ph is NULL, and if so point it to h
}
Why is it done in this way? It seems quite complex and error prone, doesen't
it?
Is there any reason something like this wasn't done instead?
(where range are ONE of those structures in headers_in that are a
ngx_table_elt_t)
if(r-headers_in-range == NULL) {
r-headers_in-range = h;
}
The ngx_http_process_header_line() function is used to handle lots
of header lines. You can't hardcode just one name into it -
you'll have to write 15 functions instead (and add another one on
each header line added). Using one universal function instead
greatly reduces code size and therefore less error prone.
--
Maxim Dounin
http://nginx.org/
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Order of directive call during request handling
Hello! On Mon, Jun 16, 2014 at 12:23:44PM +0530, Adarsh Pugalia wrote: I am not able to get the order in which the directive handlers are called when a request comes. I have a directive to connect to the database and the other to do put operation. In the conf file, i write the connect directive first, but when request comes, the connect directive handler is not called, and the put handler is called first. Can you please help me with the order in which they are called? You may want to read Evan Miller's guides, see links here: http://nginx.org/en/links.html -- Maxim Dounin http://nginx.org/ ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Really high disk i/o !!
Our server HDD i/o is constant on 8MB/s and i/o utilization + await is very high due to which nginx video streaming is really slow and we're receiving complains from our users regarding slow streaming of the videos. We're using 12X3TB SATA HDD Hardware-Raid10 with 16G RAM OS Centos 6.4 8MB/s w/r should not be issue for 12X3TB SATA HDD. Maybe i need to tweak some nginx buffers or kernels in order to reduce the high io wait ? Could someone point me to the right direction ? We can't afford SAS Drives right now and have to go with the SATA. Linux 2.6.32-431.17.1.el6.x86_64 (storage17)06/16/2014 _x86_64_(8 CPU) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 123.66 336.85 118.877.67 19496.19 2759.42 175.88 2.40 18.93 6.27 79.35 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 75.00 8.67 108.002.00 18117.3385.33 165.48 1.88 17.05 6.86 75.47 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 76.00 0.00 94.000.33 17192.00 2.67 182.28 1.50 16.04 7.47 70.47 Any help would be highly appreciated. Regards. Shahzaib ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Really high disk i/o !!
On 16/06/14 12:49, shahzaib shahzaib wrote: 8MB/s w/r should not be issue for 12X3TB SATA HDD. Maybe i need to tweak some nginx buffers or kernels in order to reduce the high io wait ? if you have a high number of concurrent connections and/or use limit_rate, then expect hdd (sata or sas) to quickly run out of iops tip from me: use nmon (linux) to see the real time status of your disks. If they are wrapped up in iowait then you need a faster seeking disk (e.g. ssd because sas won't give you a significant amount extra) if only one of them is in iowait then it's probably a dead/dying disk ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Accessing the location configuration of module 2 during post configuration processing of module 1 for a particular server
Hello!
On Mon, Jun 16, 2014 at 02:58:50PM +0800, Rv Rv wrote:
Hello Maxim
Thanks for the response.
As previously suggested, you should consider using merge callbacksto
validate configuration instead.
The requirement is *not* to validate the configuration. The
requirement is to find the final value set by one of the
directives once the configuration has been parsed. e..g lets say
we have a directive my_set_flag that sets a value to 0 or 1.
So if in configuration we have
location \ {
my_set_flag 0;
---
my_set_flag 1;
---
my_set_flag 0;
}
Such a configuration is invalid due to duplicate my_set_flag
directive, and will be rejected during configuration parsing.
then the merge callback will be called thrice for each
invocation of the directive. Let's assume the logic is to set a
variable with whatever the value of the directive was. So once
parsing completes, the value of the variable should be 0.
You are misunderstanding what merge callback is, what it does and
how it's called.
I can get this value during request processing. However, I
cannot get this value *after* the parsing of the configuration
has completed. What is the nginx recommended way to get this
value. As noted in earlier post, I am not seeing the correct
values in post configuration - and so perhaps that is not the
right way.
Again: accessing a location configuration from postconfiguration
callback isn't trivial and not supported. You should use merge
callback instead, or use main or server configuration instead,
not location.
Before proceeding any further, I would recommend you to spend some
time digging into nginx configuration basics. In particular, Evan
Miller's guide may be helpful, see links here:
http://nginx.org/en/links.html
--
Maxim Dounin
http://nginx.org/
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: tmp directory filling up
Hello! On Mon, Jun 16, 2014 at 03:22:11PM +0400, Andrei Belov wrote: On 11 Jun 2014, at 23:23, Maxim Dounin mdou...@mdounin.ru wrote: Hello! On Wed, Jun 11, 2014 at 10:58:47AM -0400, Tatonka wrote: Hi, I have a rails application that is hosted through nginx and passenger. In this application I want provide very large files for the users to download (2GB) using send_file .. which is working just fine on the development and staging system. On the production system however the system tmp directory is limited to 1GB (separately mounted disk). When triggering a download, the tmp folder quickly fills up and the download breaks once it is completely full. I already moved passengers /tmp directory to a new location but could find how to do the same for nginx (I did set $tmp and $tmpdir with no effect). When looking into the /tmp directory however, I cannot find any large files that would explain what is happening, nevertheless, df reports it is filling up at the same time .. Lastly .. I also specified the proxy_temp_path directive in the nginx config. Again with no effect. The proxy_temp_path is related to the problem, but it's for proxy, not for passenger, and it's expected that it has no effect in your case. Is there any way to specify which directory nginx uses for its tmp data? Is nginx even the culprit here? That's not about nginx, but rather about passenger module for nginx. Last time I checked, passenger module for nginx implemented its own protocol for the upstream module (like proxy/fastcgi/etc), and should have its own ..._temp_path directive, as well as ..._max_temp_file_size and so on. It’s still true: https://github.com/phusion/passenger/blob/master/ext/nginx/Configuration.c#L55-L57 It uses NGX_HTTP_PROXY_TEMP_PATH which is set at configure stage. And it looks like it doesn't provide relevant directives, so the only option left is to switch off buffering completely, using the passenger_buffer_response directive (which is again incorrectly named, it should be ..._buffering). Well, I never had a reason to say anything good about passenger, so it's at least consistent... ;) -- Maxim Dounin http://nginx.org/ ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: nginx +
Hello Steve! From: Steve Holdoway st...@greengecko.co.nz Sorry to ask this, but does using this *require* a support contract? I'm interested in the streaming stuff, but it's not worth $1300/yr to me... At this point we only offer a combined license and support model at the $1350/yr price point. We do evaluate our product offerings and user feedback regularly; so, thanks for reaching out. Since this is the FOSS mailing list, I’ll leave the topic here. If you’d like more information please reach out to me directly. Sarah -- sarah.novotny ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
rewrite url in upstream block
Hi, I am using a query parameter in upstream module to serve request based on consistent hashing. This query parameter is introduced in the request handler module and not originally coming from the downstream. I would like to remove this parameter once the job is done before sending it to an upstream server but I couldn't see a place where to do this. Rewrite rules are not allowed in upstream block. Is there a better way of doing this? Thanks, Gona Posted at Nginx Forum: http://forum.nginx.org/read.php?2,250892,250892#msg-250892 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Hello itpp,i have been abled to use ngx_http_geo_module. Now the request
coming from local ISP will first go to the main server (US) and then main
server will check if the ip is 1.2.3.4 so it'll direct the request to the
local caching server and than caching server will check if the file is
cached or it should again get the file from main server and cache it
locally.
When i tested it locally, it worked fine but the file URL in firebug is
coming from MAIN server when it should have come from the local caching
server. I can also see the caching directory size increases when the
matching client via geo module is directed to the local caching server but
the URL remains the same in firebug.
US config :-
geo $TW {
default 0;
192.168.1.0/24 1;
}
server {
listen 80;
server_name 002.files.com;
# limit_rate 600k;
location / {
root /var/www/html/files;
index index.html index.htm index.php;
# autoindex on;
}
location ~ \.(mp4|jpeg|jpg)$ {
mp4;
root /var/www/html/files;
if ($TW) {
proxy_pass http://192.168.22.32:80;
}
expires 7d;
valid_referers none blocked domain.com *.domain.com blog.domain.com
*.facebook.com *.twitter.com *.files.com *.pump.net domain.tv *.domain.tv
domainmedia.tv www.domainmedia.tv embed.domainmedia.tv;
if ($invalid_referer) {
return 403;
}
}
}
Edge config :-
proxy_ignore_headers Set-Cookie;
proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=static:100m
loader_threshold=300 loader_files=10 inactive=1d
max_size=30m;
proxy_temp_path /data/nginx/tmp 1 2;
add_header X-Cache-Status $upstream_cache_status;
add_header Accept-Ranges bytes;
max_ranges 512;
server {
listen 80;
server_name 192.168.22.32;
root /var/www/html/files;
location ~ \.(mp4|jpeg|jpg)$ {
root /var/www/html/files;
mp4;
try_files $uri @getfrom_origin;
}
location @getfrom_origin {
proxy_pass http://002.files.com:80;
proxy_cache_valid 200 302 60m;
proxy_cache_valid any 1m;
proxy_cache static;
proxy_cache_min_uses 1;
}
Maybe i need to add some variable to get original server ip ?
http://nginx.org/en/docs/http/ngx_http_geo_module.html
http://nginx.org/en/docs/http/ngx_http_geo_module.html
On Fri, Jun 6, 2014 at 8:56 PM, shahzaib shahzaib shahzaib...@gmail.com
wrote:
Thanks a lot itpp. :) I'll look into it and get back to you.
Thanks again for quick solution :)
On Fri, Jun 6, 2014 at 8:26 PM, itpp2012 nginx-fo...@nginx.us wrote:
shahzaib1232 Wrote:
---
@itpp I am currenlty proceeding with proxy_cache method just because i
had
to done this in emergency mode due to boss pressure :-|. I have a
quick
question, can i make nginx to cache files for specific clients ?
Like, if our caching servers are deployed by only single ISP named
ptcl.
So if ip from ptcl client is browsing video, only his requested file
should
be cached not for any other client, does nginx support that ??
You could do this based on some IP ranges or via
https://github.com/flant/nginx-http-rdns
See
http://serverfault.com/questions/380642/nginx-how-to-redirect-users-with-certain-ip-to-special-page
and
http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250707#msg-250707
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Really high disk i/o !!
If they are wrapped up in iowait then you need a faster seeking disk (e.g. ssd because sas won't give you a significant amount extra) or need to tune something. You don't necessarily need to change always the hdw if you have high iowaits. There are many variables around. -- Stefan Parvu spa...@systemdatarecorder.org ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote: --- Maybe i need to add some variable to get original server ip ? https://www.google.nl/#q=nginx+geo+remote+ip+address http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/ Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250895#msg-250895 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Problem with big files
Hi Recently I hit quite big problem with huge files. Nginx is a cache fronting an origin which serves huge files (several GB). Clients use mostly range requests (often to get parts towards the end of the file) and I use a patch Maxim provided some time ago allowing range requests to receive HTTP 206 if a resource is not in cache but it's determined to be cacheable... When a file is not in cache and I see a flurry of requests for the same file I see that after proxy_cache_lock_timeout - at that time the download didn't reach the first requested byte of a lot of requests - nginx establishes a new connection to upstream for each client and initiates another download of the same file. I understand why this happens and that it's by design but... That kills the server. Multiple writes to temp directory basically kill the disk performance (which in turn blocks nginx worker processes). Is there anything that can be done to help that? Keeping in mind that I can't afford serving HTTP 200 to a range request and also I'd like to avoid clients waiting for the first requested byte forever... Thanks in advance! Regards, Kuba Posted at Nginx Forum: http://forum.nginx.org/read.php?2,250899,250899#msg-250899 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Problem with big files
I use a patch Maxim provided some time ago allowing range requests to receive HTTP 206 if a resource is not in cache but it's determined to be cacheable... Can you please link to this patch? Regards, Justin Dorfman http://www.twitter.com/jdorfman Director of Developer Relations MaxCDN http://twitter.com/MaxCDNDeveloper On Mon, Jun 16, 2014 at 2:05 PM, jakubp nginx-fo...@nginx.us wrote: Hi Recently I hit quite big problem with huge files. Nginx is a cache fronting an origin which serves huge files (several GB). Clients use mostly range requests (often to get parts towards the end of the file) and I use a patch Maxim provided some time ago allowing range requests to receive HTTP 206 if a resource is not in cache but it's determined to be cacheable... When a file is not in cache and I see a flurry of requests for the same file I see that after proxy_cache_lock_timeout - at that time the download didn't reach the first requested byte of a lot of requests - nginx establishes a new connection to upstream for each client and initiates another download of the same file. I understand why this happens and that it's by design but... That kills the server. Multiple writes to temp directory basically kill the disk performance (which in turn blocks nginx worker processes). Is there anything that can be done to help that? Keeping in mind that I can't afford serving HTTP 200 to a range request and also I'd like to avoid clients waiting for the first requested byte forever... Thanks in advance! Regards, Kuba Posted at Nginx Forum: http://forum.nginx.org/read.php?2,250899,250899#msg-250899 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: how to set timer?
how works? i lost. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4265,250903#msg-250903 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
upstream on OpenBSD not executing requests
Hello everybody!
I am a new and excited nginx user and I just had to hit a problem complex
enough for me to post a message here hoping to get some help.
OS: OpenBSD 5.5 amd64
nginx -v: nginx version: nginx/1.4.7
nginx.conf:
user www;
worker_processes 10;
error_log /var/log/nginx/error.log error;
worker_rlimit_nofile 1024;
events {
worker_connections 800;
}
http {
include mime.types;
default_type application/octet-stream;
index index.jsp;
keepalive_timeout 4;
upstream shipmaticacluster {
server 10.0.0.10:8080;
server 10.0.0.11:8080;
}
server {
server_tokens off;
access_log /var/log/nginx/proxy.log;
location / {
proxy_pass http://shipmaticacluster;
}
}
}
Here is the problem: if the following is used:
proxy_pass http://10.1.1.10:8080;
or this is used:
proxy_pass http://10.1.1.11:8080;
then the requests are executed and the proxy log has this in it:
192.168.0.13 - - [16/Jun/2014:21:22:56 -0400] GET /Shipmatica/ HTTP/1.1
200 8 - Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0)
Gecko/20100101 Firefox/16.0
and that is great!
However when I use upstream cluster settings:
proxy_pass http://shipmaticacluster;
then the request executes a long time until it expires or until I hit escape
in the browser and then these lines are printed into the proxy log:
192.168.0.13 - - [16/Jun/2014:23:03:26 -0400] GET /Shipmatica HTTP/1.1 499
0 - Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101
Firefox/16.0
and info log:
2014/06/16 23:03:26 [info] 29349#0: *1 kevent() reported that client
prematurely closed connection, so upstream connection is closed too while
connecting to upstream, client: 192.168.0.13, server: , request: GET
/Shipmatica HTTP/1.1, upstream: http://10.0.0.10:8080/Shipmatica;, host:
192.168.0.28
I have gone through about 5 or 6 hours of internet searches and experiments
by now, looked at the system log files and payed attention to pflog, no
results anywhere, nothing is found in the OS log files, pf doesn't block any
traffic. I switch back to the specific IP address in the proxy_pass and the
requests flow through just fine.
This is as far as I can go without some help, I hope somebody has insights
on this issue.
Thank you!
Roman
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,250904,250904#msg-250904
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Об одной малоизвестной уязвимости в веб сайтах
Hello! On Sun, Jun 15, 2014 at 11:07:58PM +0300, Gena Makhomed wrote: On 14.06.2014 21:14, Maxim Dounin wrote: http://habrahabr.ru/post/166855/ ... Прокси-сервер, помимо всего прочего, является сервером, и процитированное требование вернуть 400 в ответ на Host header field with an invalid field-value - к нему тоже относится. Соответственно, если трактовать термин invalid field-value как включающий проверку не только синтаксиса, но и требований, предъявляемых к формированию заголовка Host клиентом, то требование ignore the received Host header теряет смысл. Максим, теперь я понял, в чем была моя ошибка, спасибо! Теримн field-value расшифровывается в 3.2. Header Fields, http://tools.ietf.org/html/rfc7230#section-3.2: field-value= *( field-content / obs-fold ) Соответственно invalid - это то, что не соответствует указанному синтаксису. Понял, спасибо! Но есть один не совсем понятный нюанс - озвученные в RFC 7230 требования к клиенту: http://tools.ietf.org/html/rfc7230#section-5.4 A client MUST send a Host header field in all HTTP/1.1 request messages. If the target URI includes an authority component, then a client MUST send a field-value for Host that is identical to that authority component, excluding any userinfo subcomponent and its @ delimiter (Section 2.7.1). - это ведь требования к синтаксису, которые обязательны для выполнения. Нет, это не требования к синтаксису, это требования к семантике. И они обязательны для клиента, но не для сервера. Соответственно, что делать в случае, если клиент их не выполнил - на совести сервера. А вот что есть в RFC 7231: http://tools.ietf.org/html/rfc7231#section-6.5.1 6.5.1. 400 Bad Request The 400 (Bad Request) status code indicates that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing). Следовательно, сервер имеет право вернуть 400 статус, если получит запрос с разными authority component в заголовке Host и в absolute Request URI ? Сервер имеет право вернуть 400 более или менее в любой момент. Кроме того, в разделе 5.5. Effective Request URI http://tools.ietf.org/html/rfc7230#section-5.5 даже прямо говорят, что запрос может быть misdirected, deliberately or accidentally, и что origin server должен сам решить, обрабатывать такой запрос или нет, потому что it might indicate an attempt to bypass security filters, trick the server into delivering non-public content, or poison a cache. именно это ведь и происходит в случае запроса GET http://apple.com/ HTTP/1.1 Host: samsung.com Разве ответить на такой запрос 400-м статусом не будет лучше? Это зависит от многих факторов. Вот тут Валентин давеча напрограммировал возврат 400, если имя, указанное в SNI, не совпадало с именем, используемым в запросе, ибо RFC 6066 говорит: ... If the server_name is established in the TLS session handshake, the client SHOULD NOT attempt to request a different server name at the application layer. Так пришлось распрограмировать обратно - потому что Chrome использует a different server name at the application layer, когда считает нужным/возможным. То, как ведёт себя nginx по умолчанию - вполне себе безопасно, и проблемы нет. Какая-либо проблема появляется тогда и только тогда, когда администратор начинает писать в конфиге $http_host, не думая о последствиях. Есть мнение, что совсем простое решение этой проблемы - не делать так (c) анекдот. -- Maxim Dounin http://nginx.org/ ___ nginx-ru mailing list nginx-ru@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-ru
Re: Об одной малоизвестной уязвимости в веб сайтах
Maxim Dounin Wrote: --- Какая-либо проблема появляется тогда и только тогда, когда администратор начинает писать в конфиге $http_host, не думая о последствиях. Есть мнение, что совсем простое решение этой проблемы - не делать так (c) анекдот. Вся проблема в том что сами программисты Nginx в модуле FastCGI используют переменную $http_host для HTTP_HOST, вместо нормализованной переменой $host, администраторам на оборот приходится исправлять это своими руками и писать в конфиге fastcgi_param HTTP_HOST $host; Если следовать вашей логике, надо изменить код модуля FastCGI, чтобы по умолчанию Nginx был вполне себе безопасный. Posted at Nginx Forum: http://forum.nginx.org/read.php?21,246086,250890#msg-250890 ___ nginx-ru mailing list nginx-ru@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-ru
Re: Об одной малоизвестной уязвимости в веб сайтах
Hello! On Mon, Jun 16, 2014 at 11:19:57AM -0400, S.A.N wrote: Maxim Dounin Wrote: --- Какая-либо проблема появляется тогда и только тогда, когда администратор начинает писать в конфиге $http_host, не думая о последствиях. Есть мнение, что совсем простое решение этой проблемы - не делать так (c) анекдот. Вся проблема в том что сами программисты Nginx в модуле FastCGI используют переменную $http_host для HTTP_HOST, вместо нормализованной переменой $host, администраторам на оборот приходится исправлять это своими руками и писать в конфиге fastcgi_param HTTP_HOST $host; Если следовать вашей логике, надо изменить код модуля FastCGI, чтобы по умолчанию Nginx был вполне себе безопасный. По умолчанию в fastcgi http-заголовки передаются как есть, в виде параметров HTTP_*. Каноническое же имя сервера доступно в параметре SERVER_NAME. Что из этого и как использовать - это вопрос к приложению, а не к nginx'у. -- Maxim Dounin http://nginx.org/ ___ nginx-ru mailing list nginx-ru@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-ru
Re: Об одной малоизвестной уязвимости в веб сайтах
Maxim Dounin Wrote:
---
Какая-либо проблема появляется тогда и только
тогда, когда администратор начинает писать в конфиге $http_host,
не думая о последствиях. Есть мнение, что совсем простое решение
этой проблемы - не делать так (c) анекдот.
Вся проблема в том что сами программисты Nginx в модуле FastCGI
используют
переменную $http_host для HTTP_HOST, вместо нормализованной
переменой $host,
администраторам на оборот приходится исправлять это своими руками и
писать в
конфиге
fastcgi_param HTTP_HOST $host;
Если следовать вашей логике, надо изменить код модуля FastCGI, чтобы
по
умолчанию Nginx был вполне себе безопасный.
По умолчанию в fastcgi http-заголовки передаются как есть, в виде
параметров HTTP_*. Каноническое же имя сервера доступно в
параметре SERVER_NAME. Что из этого и как использовать - это
вопрос к приложению, а не к nginx'у.
Я знаю, и уже выше объяснял почему бекенд-приложениям нужно использовать
HTTP_HOST вместо SERVER_NAME.
Значения SERVER_NAME не может использоваться, если в директиве server_name
используется маска или бекенд обрабатывает запросы для default_server.
Ещё раз приведу пример, в котором мы легко можем получить доступ, к закрытым
на уровне Nginx ресурсам.
server
{
server_name *.example.com;
...
fastcgi_pass php;
}
server
{
server_name private.example.com;
location / {
allow 192.168.1.0/24;
deny all;
fastcgi_pass php;
auth_request /auth;
}
...
}
Как видно два хоста используют один upstream, на котором бекенд приложения
даёт расширенные привилегии юзерам private.example.com, в расчете на то что
Nginx предварительно сделал проверку по IP и успешно провел аунтификацию,
всё вроде логично и удобно сделано, но такая схема легко ломается таким
запросом.
GET http://example.com/SecureData/ HTTP/1.1
Host: private.example.com
Бекенд получает HTTP_HOST=private.example.com, даёт юзеру привилегии, но
Nginx не проводил проверки по IP и не делал запрос аунтификацию, потому что
отработал хост конфигурации для example.com вместо private.example.com
В данном случаи можно было бы использовать переменную SERVER_NAME, но я выше
писал почему это не всегда возможно.
Если мы оба понимаем, что ситуация когда authority component и значения Host
разные, это исключительная ситуация и её нужно обрабатывать как Exception, в
этом случаи есть три варианта поведения, выбросить Exception (отдать 400
статус), исправить ошибку (использовать переменую $host), и самый плохой
вариант (антишаблон) ничего не делать и не обрабатывать эту исключительную
ситуацию и оставить все как есть.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?21,246086,250896#msg-250896
___
nginx-ru mailing list
nginx-ru@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-ru
Re: Об одной малоизвестной уязвимости в веб сайтах
On 16.06.2014 15:00, Maxim Dounin wrote: озвученные в RFC 7230 требования к клиенту: http://tools.ietf.org/html/rfc7230#section-5.4 A client MUST send a Host header field in all HTTP/1.1 request messages. If the target URI includes an authority component, then a client MUST send a field-value for Host that is identical to that authority component, excluding any userinfo subcomponent and its @ delimiter (Section 2.7.1). - это ведь требования к синтаксису, которые обязательны для выполнения. Нет, это не требования к синтаксису, это требования к семантике. Запись target URI в виде valid request message - это разве не syntax? http://tools.ietf.org/html/rfc7231#section-2 When a client constructs an HTTP/1.1 request message, it sends the target URI in one of various forms, as defined in (Section 5.3 of [RFC7230]). When a request is received, the server reconstructs an effective request URI for the target resource (Section 5.5 of [RFC7230]). One design goal of HTTP is to separate resource identification from request semantics, which is made possible by vesting the request semantics in the request method (Section 4) and a few request-modifying header fields (Section 5). effective request URI - это именно что resource identification. Они ведь разнесли syntax и semantics по двум отдельным документам: RFC 7230: Message Syntax and Routing RFC 7231: Semantics and Content http://tools.ietf.org/html/rfc7231#section-6.5.1 6.5.1. 400 Bad Request The 400 (Bad Request) status code indicates that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing). Следовательно, сервер имеет право вернуть 400 статус, если получит запрос с разными authority component в заголовке Host и в absolute Request URI ? Сервер имеет право вернуть 400 более или менее в любой момент. Да, но здесь явно указывается, когда сервер возвращает 400: в случае client error, например, malformed request syntax или deceptive request routing - эти две причины подходят. Кроме того, в разделе 5.5. Effective Request URI http://tools.ietf.org/html/rfc7230#section-5.5 даже прямо говорят, что запрос может быть misdirected, deliberately or accidentally, и что origin server должен сам решить, обрабатывать такой запрос или нет, потому что it might indicate an attempt to bypass security filters, trick the server into delivering non-public content, or poison a cache. именно это ведь и происходит в случае запроса GET http://apple.com/ HTTP/1.1 Host: samsung.com Разве ответить на такой запрос 400-м статусом не будет лучше? Это зависит от многих факторов. Вот тут Валентин давеча напрограммировал возврат 400, если имя, указанное в SNI, не совпадало с именем, используемым в запросе, ибо RFC 6066 говорит: ... If the server_name is established in the TLS session handshake, the client SHOULD NOT attempt to request a different server name at the application layer. Так пришлось распрограмировать обратно - потому что Chrome использует a different server name at the application layer, когда считает нужным/возможным. SHOULD NOT - это не запрет, а только лишь рекомендация: http://tools.ietf.org/html/rfc2119#section-4 так что формально и фактически Chrome ничего не нарушает. В обсуждаемой выше ситуации написано совсем иначе: MUST. http://tools.ietf.org/html/rfc2119#section-1 и если клиент этого не сделал - он сам прислал невалидный запрос, так что ответить 400-м статусом на такой запрос - вполне разумно. да и в RFC 7230 об этом тоже есть (про поведение хрома): http://tools.ietf.org/html/rfc7230#section-2.3 server MUST NOT assume that two requests on the same connection are from the same user agent unless the connection is secured and specific to that agent. Some non-standard HTTP extensions (e.g., [RFC4559]) have been known to violate this requirement, resulting in security and interoperability problems. То, как ведёт себя nginx по умолчанию - вполне себе безопасно, и проблемы нет. Какая-либо проблема появляется тогда и только тогда, когда администратор начинает писать в конфиге $http_host, не думая о последствиях. Есть мнение, что совсем простое решение этой проблемы - не делать так (c) анекдот. Когда nginx выполняет роль origin server или HTTP reverse proxy - тогда проблем действительно нет, но когда он выполняет роль gateway и communicates с backend`ом по протоколу FastCGI - тогда получаются очень неприятные проблемы. Причина проблем: malformed request syntax, который не получается корректным образом передать на backend. Что мешает в этой ситуации вернуть 400 статус? Вместо того, чтобы отсылать на backend невалидный запрос, интерпретируя клиентский запрос не по спецификации HTTP/1.1 а по спецификации CGI/1.1, которая не согласуется со спецификацией протокола HTTP/1.1. Ничего ведь не сломается и на
