Optimizing nginx mail proxy

[Azusa Taroura]

Hi everyone,  

I’m trying to optimize mail-proxy. 
My performance test is 1 client sends many request to 1 nginx server.


This is my current settings:

worker_processes auto;
worker_rlimit_nofile  10;

#error_log  /var/log/nginx/error.log debug;
#error_log  /var/log/nginx/error.log warn;
error_log /var/log/nginx/error.log crit;
events {
  worker_connections 1024;
  #worker_connections 4000;
  #multi_accept on;
  #use epoll;
}

mail {
  auth_http localhost:80/auth/smtp;
  proxy_pass_error_message on;
  proxy on;
  smtp_auth login plain;
  xclient   on;
  server {
  listen25;
  protocol  smtp;
  }
  server {
  listen465;
  protocol  smtp;
  ssl   on;
  ssl_certificate  /etc/nginx/ssl/server.crt;
  ssl_certificate_key  /etc/nginx/ssl/server.key;

  ssl_session_cache shared:SSL:20m;
  ssl_session_timeout 180m;

  #ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  #ssl_prefer_server_ciphers on;
  #ssl_ciphers
ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DHE+AES128:!ADH:!AECDH:!MD5;
  #ssl_dhparam /etc/nginx/cert/dhparam.pem;
  #ssl_stapling on;
  #ssl_stapling_verify on;
  #ssl_trusted_certificate /etc/nginx/cert/trustchain.crt;
  #resolver 8.8.8.8 8.8.4.4;
  }
}


Question>>
Low cpu usage, but the performance result is not good.
Do yoy know how to take full advantage of nginx?

Thank you for your time. 
Azusa Taroura

Posted at Nginx Forum: 
https://forum.nginx.org/read.php?2,278656,278656#msg-278656

___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: Mail proxy the destination server by ssl (Postfix)

[Azusa Taroura]

Thank you for your reply!

Posted at Nginx Forum: 
https://forum.nginx.org/read.php?2,278532,278655#msg-278655

___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re[2]: О заголовке content-type

[CoDDoC]

А... голова дырявая. Забыл про types.
Спасибо.

>Понедельник, 19 февраля 2018, 19:26 +03:00 от Maxim Dounin 
>:
>
>Hello!
>
>On Mon, Feb 19, 2018 at 06:18:52PM +0300, CoDDoC wrote:
>
>> Доброе время суток!
>> 
>> Есть такой локейшен:
>> location ~ "^/img/" { internal; }
>> 
>> Естественно, прямой запрос 'GET /img/file.jpg' получает 404
>> Все хорошо, но нужно вместо стандартной nginx страницы отдать кастомную.
>> Можно решать разными способами, я решил попробовать через 'return 404 
>> ' (минимум внутренних реврайтов/редиректов).
>> 
>> Получилось так (упрощенно):
>> 
>> error_page 404 = @err404;
>> location @err404 {
>>   return 404 ' WTF ? 
>>';
>>   add_header "Content-Type" "text/html; charset=UTF-8" always;
>> }
>> 
>> Оно работает, одно смущает: дублирование заголовка Content-Type: сперва 
>> 'image/jpeg', затем уже 'text/html; charset=UTF-8'
>> Браузер-то, ясное дело, возьмет по итогу второй заголовок. Но, может, есть 
>> какой-либо цивилизованный способ оставить один Content-Type без 
>> прикручивания костыля типа headers-more ?
>
>Правильно - не пытаться прибить левый Content-Type гвоздями с 
>помощью add_header, а задать его штатными средствами.  Например 
>так:
>
>error_page 404 = /error404.html;
>location = /error404.html {
>charset utf-8;
>return 404 ' ...';
>}
>
>Или, если по каким-то причинам очень хочется именно именованный 
>location, то так:
>
>error_page 404 = @err404;
>location @err404 {
>types {}
>default_type text/html;
>charset utf-8;
>return 404 ' ...';
>}
>
>-- 
>Maxim Dounin
>http://mdounin.ru/
>___
>nginx-ru mailing list
>nginx-ru@nginx.org
>http://mailman.nginx.org/mailman/listinfo/nginx-ru


--
___
nginx-ru mailing list
nginx-ru@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-ru

Fwd: Question about wildcard nginx entry.

[bukow bukowiec]

-- Forwarded message --
From: bukow bukowiec 
Date: Mon, Feb 19, 2018 at 9:44 PM
Subject: Question about wildcard nginx entry.
To: nginx@nginx.org



Hello, first time i am writing here, i don't know if my message will go
further :)

Anyway, can anyone tell me why nginx does not support wildcard type like:

"*.domain.tld" ?

Is that possible for nginx to this as a valid wildcard entry?

The point i ask is, there is a lot of tools and software, PaaS and others
that work on nginx.
We cant simply add "*.domain.tld" in the software because there must be
custom rule build.
( ~^(.*)\.domain\.tld$  )

Hope that makes sense.
Can someone tell me what is the truth about it, as i have no idea?

Regards to you
Thank you
Mateusz Kurowski
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: О заголовке content-type

[Maxim Dounin]

Hello!

On Mon, Feb 19, 2018 at 06:18:52PM +0300, CoDDoC wrote:

> Доброе время суток!
> 
> Есть такой локейшен:
> location ~ "^/img/" { internal; }
> 
> Естественно, прямой запрос 'GET /img/file.jpg' получает 404
> Все хорошо, но нужно вместо стандартной nginx страницы отдать кастомную.
> Можно решать разными способами, я решил попробовать через 'return 404 ' 
> (минимум внутренних реврайтов/редиректов).
> 
> Получилось так (упрощенно):
> 
> error_page 404 = @err404;
> location @err404 {
>   return 404 ' WTF ?  
>   ';
>   add_header "Content-Type" "text/html; charset=UTF-8" always;
> }
> 
> Оно работает, одно смущает: дублирование заголовка Content-Type: сперва 
> 'image/jpeg', затем уже 'text/html; charset=UTF-8'
> Браузер-то, ясное дело, возьмет по итогу второй заголовок. Но, может, есть 
> какой-либо цивилизованный способ оставить один Content-Type без прикручивания 
> костыля типа headers-more ?

Правильно - не пытаться прибить левый Content-Type гвоздями с 
помощью add_header, а задать его штатными средствами.  Например 
так:

error_page 404 = /error404.html;
location = /error404.html {
charset utf-8;
return 404 ' ...';
}

Или, если по каким-то причинам очень хочется именно именованный 
location, то так:

error_page 404 = @err404;
location @err404 {
types {}
default_type text/html;
charset utf-8;
return 404 ' ...';
}

-- 
Maxim Dounin
http://mdounin.ru/
___
nginx-ru mailing list
nginx-ru@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-ru

RE: Clientcertificate authentication only for a single URL

[Gbg]

I think this will set the headers only for the login URL but still ask for the 
certificate on all URLs. And this is not what I need, I only want to have to 
present a certificate for a single URL


Am 19. Februar 2018 16:35:59 MEZ schrieb Jason Whittington 
:
>I would think "location=" would solve this.  What about something like
>the following?
>
>server {
>listen 443 ssl http2;
>server_name localhost;
>
>ssl_certificate ...
>ssl_certificate_key ...
>ssl_session_cache   shared:SSL:1m;
>include templates/ssl_setup.conf;
>
>location = /login {
>proxy_set_header X-SSL-Client-Serial $ssl_client_serial;
>proxy_set_header X-SSL-Client-...
>
>proxy_pass http://localhost:8080;
>}
>
>location  / {
>root /var/www/...;
>}
>}
>
>Jason
>
>
>-Original Message-
>From: nginx [mailto:nginx-boun...@nginx.org] On Behalf Of Gbg
>Sent: Monday, February 19, 2018 9:12 AM
>To: nginx@nginx.org
>Subject: [IE] Clientcertificate authentication only for a single URL
>
>
>
>I need to secure only a single URL on my server by demanding or
>enforcing client certificate based authentication. My application is
>called by opening "myapp.local" and if necessary it logs in a user by
>issuing a call to "myapp.local/login". I can not create a second
>hostname to do the login, so specifying a second `server` with
>`server_name myapplogin.local` does not work.
>Because the login is not necessary all the time I do not want to
>encorce ssl_verify for `/` because then the user would be prompted with
>a certificate selection dialog even before he can see the start page of
>my application.
>
>This is my current setup which does not work because the first `server`
>definition block has higher priority. I tried to keep the example
>short, because of this you see some `...`, the ssl/tls stuff is in my
>config file but is not repeated here because I think it is not part of
>the problem.
>Replacing `server_name localhost` with `server_name myapp.local` didn't
>make any difference. I am on mainline 1.13.8
>
>http {
>server {
>listen 443 ssl http2;
>server_name localhost;
>
>ssl_certificate ...
>ssl_certificate_key ...
>ssl_session_cache   shared:SSL:1m;
>include templates/ssl_setup.conf;
>
>location / {
>root /var/www/...;
>}
>
>}
>
>server {
>listen 443 ssl http2;
>server_name localhost;
>
>ssl_certificate ...
>ssl_certificate_key ...
>ssl_session_cache   shared:SSL:1m;
>
>ssl_client_certificate /.../acceptedcas.pem;
>ssl_verify_depth 2;
>ssl_verify_client on;
>
>location /login {
>proxy_set_header X-SSL-Client-Serial $ssl_client_serial;
>proxy_set_header X-SSL-Client-...
>
>proxy_pass http://localhost:8080;
>}
>}
>}
>___
>nginx mailing list
>nginx@nginx.org
>http://mailman.nginx.org/mailman/listinfo/nginx
>
>This message contains proprietary information from Equifax which may be
>confidential. If you are not an intended recipient, please refrain from
>any disclosure, copying, distribution or use of this information and
>note that such actions are prohibited. If you have received this
>transmission in error, please notify by e-mail postmas...@equifax.com.
>Equifax® is a registered trademark of Equifax Inc. All rights reserved.
>___
>nginx mailing list
>nginx@nginx.org
>http://mailman.nginx.org/mailman/listinfo/nginx

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

RE: Clientcertificate authentication only for a single URL

[Jason Whittington]

I would think "location=" would solve this.  What about something like the 
following?

server {
listen 443 ssl http2;
server_name localhost;

ssl_certificate ...
ssl_certificate_key ...
ssl_session_cache   shared:SSL:1m;
include templates/ssl_setup.conf;

location = /login {
proxy_set_header X-SSL-Client-Serial $ssl_client_serial;
proxy_set_header X-SSL-Client-...

proxy_pass http://localhost:8080;
}

location  / {
root /var/www/...;
}
}

Jason


-Original Message-
From: nginx [mailto:nginx-boun...@nginx.org] On Behalf Of Gbg
Sent: Monday, February 19, 2018 9:12 AM
To: nginx@nginx.org
Subject: [IE] Clientcertificate authentication only for a single URL



I need to secure only a single URL on my server by demanding or enforcing 
client certificate based authentication. My application is called by opening 
"myapp.local" and if necessary it logs in a user by issuing a call to 
"myapp.local/login". I can not create a second hostname to do the login, so 
specifying a second `server` with `server_name myapplogin.local` does not work.
Because the login is not necessary all the time I do not want to encorce 
ssl_verify for `/` because then the user would be prompted with a certificate 
selection dialog even before he can see the start page of my application.

This is my current setup which does not work because the first `server` 
definition block has higher priority. I tried to keep the example short, 
because of this you see some `...`, the ssl/tls stuff is in my config file but 
is not repeated here because I think it is not part of the problem.
Replacing `server_name localhost` with `server_name myapp.local` didn't make 
any difference. I am on mainline 1.13.8

http {
server {
listen 443 ssl http2;
server_name localhost;

ssl_certificate ...
ssl_certificate_key ...
ssl_session_cache   shared:SSL:1m;
include templates/ssl_setup.conf;

location / {
root /var/www/...;
}

}

server {
listen 443 ssl http2;
server_name localhost;

ssl_certificate ...
ssl_certificate_key ...
ssl_session_cache   shared:SSL:1m;

ssl_client_certificate /.../acceptedcas.pem;
ssl_verify_depth 2;
ssl_verify_client on;

location /login {
proxy_set_header X-SSL-Client-Serial $ssl_client_serial;
proxy_set_header X-SSL-Client-...

proxy_pass http://localhost:8080;
}
}
}
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

This message contains proprietary information from Equifax which may be 
confidential. If you are not an intended recipient, please refrain from any 
disclosure, copying, distribution or use of this information and note that such 
actions are prohibited. If you have received this transmission in error, please 
notify by e-mail postmas...@equifax.com. Equifax® is a registered trademark of 
Equifax Inc. All rights reserved.
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

О заголовке content-type

[CoDDoC]

Доброе время суток!

Есть такой локейшен:
location ~ "^/img/" { internal; }

Естественно, прямой запрос 'GET /img/file.jpg' получает 404
Все хорошо, но нужно вместо стандартной nginx страницы отдать кастомную.
Можно решать разными способами, я решил попробовать через 'return 404 ' 
(минимум внутренних реврайтов/редиректов).

Получилось так (упрощенно):

error_page 404 = @err404;
location @err404 {
  return 404 ' WTF ?  
  ';
  add_header "Content-Type" "text/html; charset=UTF-8" always;
}

Оно работает, одно смущает: дублирование заголовка Content-Type: сперва 
'image/jpeg', затем уже 'text/html; charset=UTF-8'
Браузер-то, ясное дело, возьмет по итогу второй заголовок. Но, может, есть 
какой-либо цивилизованный способ оставить один Content-Type без прикручивания 
костыля типа headers-more ?

proxy_hide_header не годится - нет проксирования. Отправлять все 
"не-пойми-какие" запросы на бэкенд - не вижу в этом  особого смысла.

Спасибо.
--___
nginx-ru mailing list
nginx-ru@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-ru

Clientcertificate authentication only for a single URL

[Gbg]

I need to secure only a single URL on my server by demanding or enforcing 
client certificate based authentication. My application is called by opening 
"myapp.local" and if necessary it logs in a user by issuing a call to 
"myapp.local/login". I can not create a second hostname to do the login, so 
specifying a second `server` with `server_name myapplogin.local` does not work.
Because the login is not necessary all the time I do not want to encorce 
ssl_verify for `/` because then the user would be prompted with a certificate 
selection dialog even before he can see the start page of my application.

This is my current setup which does not work because the first `server` 
definition block has higher priority. I tried to keep the example short, 
because of this you see some `...`, the ssl/tls stuff is in my config file but 
is not repeated here because I think it is not part of the problem.
Replacing `server_name localhost` with `server_name myapp.local` didn't make 
any difference. I am on mainline 1.13.8

http {
server {
listen 443 ssl http2;
server_name localhost;

ssl_certificate ...
ssl_certificate_key ...
ssl_session_cache   shared:SSL:1m;
include templates/ssl_setup.conf;

location / {
root /var/www/...;
}
}

server {
listen 443 ssl http2;
server_name localhost;

ssl_certificate ...
ssl_certificate_key ...
ssl_session_cache   shared:SSL:1m;

ssl_client_certificate /.../acceptedcas.pem;
ssl_verify_depth 2;
ssl_verify_client on;

location /login {
proxy_set_header X-SSL-Client-Serial $ssl_client_serial;
proxy_set_header X-SSL-Client-...

proxy_pass http://localhost:8080;
}
}
}
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: [PATCH] Limit req: r/h and r/d support

[Maxim Dounin]

Hello!

On Sat, Feb 17, 2018 at 10:07:27PM +0100, Bernhard Reutner-Fischer wrote:

> On 16 February 2018 at 17:52, Maxim Dounin  wrote:
> 
> >> Limit req: r/h and r/d support
> 
> > Do you use it yourself, or your patch is based on the feature
> > requests in question?  If yes, what is your use case?
> 
> I have to limit certain use-cases to a handful of requests per day (don't 
> ask).
> Mentioned feature-requests just show that this oddish corner case is a
> real knock-out criterion in the real world so i mentioned them to
> emphasis that it wasn't just /me but that you're losing users due to
> this. Often this is a hard requirement. You don't seem to offer any
> sensible, off the shelve (or documented) way to handle "long" request
> limits anyway -- think shm -- at least none that i was able to find?

The limit_req module was created mostly to limit high rates - in 
particular, to limit requests to costly services to fight DoS, and 
to limit requests to password checking to fight bruteforce 
attempts.  It was never meant to limit normal user activity, and 
hence the data format choosen does not support rates less than 
1r/m.  Yet it works effectively with high rates the module was 
created for.

While it is understood that some people might want to use longer 
limits, it's not something the module tries to provide now.  Also, 
it might not actually be a good idea to provide such limits in the 
module, as any nginx restart / upgrade will mean that limits won't 
work as expected for significant time.  If you want to limits 
users to several requests per day, it might be a better idea to 
keep these limits outside of nginx.

> > Style: please add a trailing dot.
> 
> Sure, please excuse my sloppy reading of previous logs, i honestly
> failed to notice that, added.
> 
> > On 32-bit platforms this means that the maximum supported rate
> > would be about ~40kr/s.  And I suspect this can hurt real setups.
> 
> agreed. Not sure about current typical HTTP server rates, admittedly.
> Getting esoteric, but current servers usually satisfy millions of
> requests but i'm not sure if you'd usually limit those to let's say
> 100k/s, but of course i see your point.

I doubt rates like 100kr/s are needed in practice (especially on 
32-bit platforms), but I'm pretty sure that configurations with 
something like this are real.  In particular, googling "limit_req 
rate site:nginx.org/pipermail/" shows examples like 
"rate=99r/s" pretty quickly.

[...]

> > Note that I have no good solution for 32-bit platforms and the
> > 40kr/s (20kr/s) limit.
> 
> my notes read:
> disentangle rate. Should have rate(per unit) and unit(in seconds?
> minutes? calc!) where excess would have same base as rate; common
> timer conversion helper for rate/unit?
> 
> This has much more impact and hence is a much larger patch but would
> certainly be way easier to grok compared to the current handling.
> Would you be willing to fix the existing rate mess^whandling?

Current handling is pretty clear and simple: all calculations are 
done in requests per millisecond, and this requires just one value 
in various structures.  We can tolerate switching to something 
different, though it would be a good idea to keep code logic close 
to the current implementation.

-- 
Maxim Dounin
http://mdounin.ru/
___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Routing based on ALPN

[Konstantin Pavlov]

On 19/02/2018 14:02, Wiktor Kwapisiewicz via nginx wrote:
> Hello,
> 
> I'm looking for a way to route traffic on port 443 based on ALPN value
> without SSL termination.
> 
> ssl_preread_module [1] does something similar but the only exposed
> variable ($ssl_preread_server_name) is for SNI, not ALPN.
> 
> A bit of context. I'd like to use nginx to host regular HTTPS server on port
> 443 but if the ALPN value is 'xmpp-client' transparently proxy the traffic
> to my local Jabber server. This feature [2] is already supported by several
> XMPP clients.
> 
> Is there a way to access and save ALPN value to a variable?

It should possible to parse the incoming buffer with 
https://nginx.org/r/js_filter and create a variable to make a routing decision 
on.

-- 
Konstantin Pavlov
www.nginx.com
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: Files still on disc after inactive time

[Maxim Dounin]

Hello!

On Mon, Feb 19, 2018 at 04:38:00AM -0500, Andrzej Walas wrote:

> Thanks for your replay I will update nginx I search again.
> 
> Can you anwser on my couple questions:
> 1. Can you tell me what happend with downloaded part of files when I have:
> 1.1 [error] 16082#0: *1264804 upstream prematurely closed connection while
> reading upstream
> 1.2 [crit] 16082#0: *1264770 pwritev()  has written only 49152 of
> 151552 while reading upstream
>   This part of file is still on disc and don't deleted after error? 

In both cases temporary files will be removed.

> 2. Why when I set 40g for cache I have 90-100GB in cache?

As previously explained, your problems with cache are clearly due to 
nginx processes being killed.  You have resolve this first.

> 3. Can you tell me when in offcial repo for redhat 7 will be nginx 1.13.+?
> Currently the newest is 1.12.2.

Most likely redhat package maintainers will either wait till 
1.14.x branch, or won't update packages for redhat 7 at all.  You 
can install 1.13.x either from source, or via a linux packages 
provided at nginx.org:

http://nginx.org/en/download.html
http://nginx.org/en/linux_packages.html#mainline

-- 
Maxim Dounin
http://mdounin.ru/
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx