Re: Workaround of race condition between systemd and nginx.

[Daniel K.]

On 12/30/2015 03:50 PM, Gena Makhomed wrote:
> On 30.12.2015 16:51, Jim Popovitch wrote:
>> On Dec 30, 2015 8:46 AM, "Gena Makhomed"  wrote:
>>> Workaround of race condition between systemd and nginx.
>>>
>>> Just replace network.target with network-online.target in systemd unit
>>> files.
>>> More details:
>>> http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
>>
>> From that page, wrt network-online.target:
>>
>> "It is strongly recommended not to pull in this target too liberally: for
>> example network server software should generally not pull this in (since
>> server software generally is happy to accept local connections even before
>> any routable network interface is up), it's primary purpose is network
>> client software that cannot operate without network"
> 
> nginx now requires configured and up network, before starting daemon.
> Replace network.target with network-online.target is easy workaround.

Actually it does not require that at all.

It would be more helpful if you posted your config files, but from your
log file i gather they look something like:

server {
listen  172.22.22.202:80;
[...]
}

And that, due to using systemd, the nginx service gets started before
the network-interface have been configured with the IP address shown.


Two ways you can work around this issue comes to mind.

1) Allow non-local binds

# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

Put 'net.ipv4.ip_nonlocal_bind = 1' in /etc/sysctl.conf to make it stick.


2) Configure nginx to listen to *:80

Add this to your config files somewhere.

server {
listen  80;
}

to your config files somewhere, and nginx will listen to 0.0.0.0:80
instead of every IP address you mention.


Hope that helps,


Daniel K.

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Jim Popovitch]

On Dec 30, 2015 12:44 PM, "Gena Makhomed"  wrote:
>
> On 30.12.2015 19:08, Jim Popovitch wrote:
>
>>> Do you know how nginx and systemd work right now?
>>> You understand race condition between nginx and systemd?
>
>
>> I understand nginx, systemd, and race conditions.  I understand why
>> *you* have a race condition, and I understand why I do not have a race
>> condition.
>
>
> And you want to tell this "mantra" to all OpenVZ / CentOS 7.2 users?
>
>
>>> You say, what nginx should work fine if no network available,
>
>
>> Yes, or even if only localhost (lo) exists.
>
>
> lo exists.
>
> nginx startup failed.
>
> logs - see in previous messages.
>
>
>> BTW, you can read about how openvpn handled this very issue
>> https://community.openvpn.net/openvpn/ticket/462
>
>
> You can provide patch with solution?

I see no need for a patch, and I already explained why, including a link to
how another service daemon handled the very same issue.   I've also
explainec how your patch breaks things.  Good day.

-Jim P.
___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Gena Makhomed]

On 30.12.2015 16:51, Jim Popovitch wrote:


# HG changeset patch
# User Gena Makhomed 
# Date 1451482795 18000
#  Wed Dec 30 08:39:55 2015 -0500
# Node ID a340d271b3ffa51c0396a5afc5270cb02b701204
# Parent  1073d7e4e430ddb53b603d151e1a403d10aa420b
Workaround of race condition between systemd and nginx.

Just replace network.target with network-online.target in systemd unit

files.

More details:

http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/


From that page, wrt network-online.target:


"It is strongly recommended not to pull in this target too liberally: for
example network server software should generally not pull this in (since
server software generally is happy to accept local connections even before
any routable network interface is up), it's primary purpose is network
client software that cannot operate without network"


nginx is FreeBSD daemon, ant it is not systemd-aware,
and nginx not follow many strange systemd guidelines.

Current race condition between systemd and nginx follow
to non-working nginx daemon at least under CentOS 7.2
templates under OpenVZ after system reboot.

nginx now requires configured and up network, before starting daemon.
Replace network.target with network-online.target is easy workaround.

P.S.

# cat /var/log/messages

Dec 27 19:08:38 stage-ideil-com systemd: Starting nginx - high 
performance web server...

Dec 27 19:08:39 stage-ideil-com systemd: Starting System Logging Service...
Dec 27 19:08:39 stage-ideil-com systemd: Starting LSB: Bring up/down 
networking...
Dec 27 19:08:39 stage-ideil-com systemd: Starting Postfix Mail Transport 
Agent...

[...]
Dec 27 19:09:24 stage-ideil-com systemd: nginx.service: control process 
exited, code=exited status=1
Dec 27 19:09:24 stage-ideil-com systemd: Failed to start nginx - high 
performance web server.
Dec 27 19:09:24 stage-ideil-com systemd: Unit nginx.service entered 
failed state.

Dec 27 19:09:24 stage-ideil-com systemd: nginx.service failed.
Dec 27 19:09:24 stage-ideil-com systemd-sysctl: Failed to write '16' to 
'/proc/sys/kernel/sysrq': Permission denied
Dec 27 19:09:24 stage-ideil-com systemd-sysctl: Failed to write '1' to 
'/proc/sys/kernel/core_uses_pid': Permission denied
>Dec 27 19:09:24 stage-ideil-com systemd: Started LSB: Bring up/down 
networking.

>Dec 27 19:09:24 stage-ideil-com systemd: Reached target Network is Online.
>Dec 27 19:09:24 stage-ideil-com systemd: Starting Network is Online.
Dec 27 19:09:24 stage-ideil-com systemd: Started The PHP FastCGI Process 
Manager.
>Dec 27 19:09:24 stage-ideil-com nginx: nginx: the configuration file 
/etc/nginx/nginx.conf syntax is ok
>Dec 27 19:09:24 stage-ideil-com nginx: nginx: [emerg] bind() to 
172.22.22.202:80 failed (99: Cannot assign requested address)
>Dec 27 19:09:24 stage-ideil-com nginx: nginx: configuration file 
/etc/nginx/nginx.conf test failed
>Dec 27 19:09:24 stage-ideil-com network: Bringing up loopback 
interface:  [  OK  ]
>Dec 27 19:09:25 stage-ideil-com network: Bringing up interface venet0: 
 arping: Device venet0 not available.

>Dec 27 19:09:25 stage-ideil-com network: [  OK  ]

=

# cat /var/log/messages

Dec 24 18:55:14 hroniky-com systemd: Starting Sockets.
Dec 24 18:55:14 hroniky-com systemd: Reached target Basic System.
Dec 24 18:55:14 hroniky-com systemd: Starting Basic System.
Dec 24 18:55:14 hroniky-com systemd: Started D-Bus System Message Bus.
Dec 24 18:55:14 hroniky-com systemd: Starting D-Bus System Message Bus...
Dec 24 18:55:14 hroniky-com systemd: Starting Permit User Sessions...
Dec 24 18:55:14 hroniky-com systemd: Starting Postfix Mail Transport 
Agent...

Dec 24 18:55:14 hroniky-com systemd: Started OpenSSH Server Key Generation.
Dec 24 18:55:14 hroniky-com systemd: Starting /etc/rc.d/rc.local 
Compatibility...
Dec 24 18:55:15 hroniky-com systemd: Starting nginx - high performance 
web server...

Dec 24 18:55:18 hroniky-com systemd: Starting System Logging Service...
Dec 24 18:55:18 hroniky-com systemd: Starting LSB: Bring up/down 
networking...
Dec 24 18:55:18 hroniky-com nginx: nginx: the configuration file 
/etc/nginx/nginx.conf syntax is ok
Dec 24 18:55:18 hroniky-com nginx: nginx: [emerg] bind() to 
172.23.23.161:80 failed (99: Cannot assign requested address)
Dec 24 18:55:18 hroniky-com nginx: nginx: configuration file 
/etc/nginx/nginx.conf test failed
Dec 24 18:55:18 hroniky-com systemd: Starting The PHP FastCGI Process 
Manager...

Dec 24 18:55:18 hroniky-com systemd: Started OpenSSH server daemon.
Dec 24 18:55:18 hroniky-com systemd: Starting OpenSSH server daemon...
Dec 24 18:55:18 hroniky-com systemd: Starting Login Service...
Dec 24 18:55:18 hroniky-com systemd: Starting Dump dmesg to 
/var/log/dmesg...

Dec 24 18:55:18 hroniky-com systemd: Started Permit User Sessions.
Dec 24 18:55:18 hroniky-com systemd: Started /etc/rc.d/rc.local 
Compatibility.
Dec 24 18:55:18 hroniky-com systemd: nginx.service: control 

Re: Workaround of race condition between systemd and nginx.

[Jim Popovitch]

On Wed, Dec 30, 2015 at 11:50 AM, Gena Makhomed  wrote:
> On 30.12.2015 18:09, Jim Popovitch wrote:
>
>>> nginx now requires configured and up network, before starting daemon.
>
>
>> Specifically it's your configuration.
>> You are hardcoding an IP address to bind to
>> thereby telling nginx to not start until that IP is active.
>
>
> Do you know how nginx and systemd work right now?
> You understand race condition between nginx and systemd?

I understand nginx, systemd, and race conditions.  I understand why
*you* have a race condition, and I understand why I do not have a race
condition.

>> That will prevent nginx from staring in situations where systemd
>> determines that the external network is not yet active (dhcp, etc.,
>> etc), yet nginx may still run perfectly fine with split interfaces,
>> localhost, etc.
>
>
> You say, what nginx should work fine if no network available,

Yes, or even if only localhost (lo) exists.

BTW, you can read about how openvpn handled this very issue
https://community.openvpn.net/openvpn/ticket/462

> I say what nginx *must* work fine if network *IS* available.

It does, it currently works if the network IS or ISNT available, and
all possibilities in-between.

>
> So, I need create my own fork, for example, nginx-fixed,
> which I can use with OpenVZ and CentOS 7.2 templates?

No, you can simply modify your local /etc/systemd/system/nginx.service
file to specify a local startup policy.


-Jim P.

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Gena Makhomed]

On 30.12.2015 19:08, Jim Popovitch wrote:


Do you know how nginx and systemd work right now?
You understand race condition between nginx and systemd?



I understand nginx, systemd, and race conditions.  I understand why
*you* have a race condition, and I understand why I do not have a race
condition.


And you want to tell this "mantra" to all OpenVZ / CentOS 7.2 users?


You say, what nginx should work fine if no network available,



Yes, or even if only localhost (lo) exists.


lo exists.

nginx startup failed.

logs - see in previous messages.


BTW, you can read about how openvpn handled this very issue
https://community.openvpn.net/openvpn/ticket/462


You can provide patch with solution?

If you can't - can you please stop flame war against my patch?


I say what nginx *must* work fine if network *IS* available.



It does  it currently works if the network IS or ISNT available, and
all possibilities in-between.


No.

nginx config is valid.

logs - see in previous messages.


So, I need create my own fork, for example, nginx-fixed,
which I can use with OpenVZ and CentOS 7.2 templates?


No, you can simply modify your local /etc/systemd/system/nginx.service
file to specify a local startup policy.


Inside all containers on all hardware nodes? Manually?

And same way this bug should be fixed by all other OpenVZ users?

P.S.

This is nginx-devel mail list for developers, not for users.

$ curl -s http://nginx.org/en/CHANGES | grep "Jim Popovitch"

$ curl -s http://nginx.org/en/CHANGES | grep "Gena Makhomed"

--
Best regards,
 Gena

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Das, Jagannath]

How to reproduce this issue?

From: "Daniel K." >
Reply-To: "nginx-devel@nginx.org" 
>
Date: Wednesday, December 30, 2015 at 11:58 PM
To: "nginx-devel@nginx.org" 
>
Subject: Re: Workaround of race condition between systemd and nginx.

On 12/30/2015 03:50 PM, Gena Makhomed wrote:
On 30.12.2015 16:51, Jim Popovitch wrote:
On Dec 30, 2015 8:46 AM, "Gena Makhomed" 
> wrote:
Workaround of race condition between systemd and nginx.

Just replace network.target with network-online.target in systemd unit
files.
More details:
http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

From that page, wrt network-online.target:

"It is strongly recommended not to pull in this target too liberally: for
example network server software should generally not pull this in (since
server software generally is happy to accept local connections even before
any routable network interface is up), it's primary purpose is network
client software that cannot operate without network"
nginx now requires configured and up network, before starting daemon.
Replace network.target with network-online.target is easy workaround.

Actually it does not require that at all.

It would be more helpful if you posted your config files, but from your
log file i gather they look something like:

server {
listen 172.22.22.202:80;
[...]
}

And that, due to using systemd, the nginx service gets started before
the network-interface have been configured with the IP address shown.


Two ways you can work around this issue comes to mind.

1) Allow non-local binds

# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

Put 'net.ipv4.ip_nonlocal_bind = 1' in /etc/sysctl.conf to make it stick.


2) Configure nginx to listen to *:80

Add this to your config files somewhere.

server {
listen 80;
}

to your config files somewhere, and nginx will listen to 0.0.0.0:80
instead of every IP address you mention.


Hope that helps,


Daniel K.

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Gena Makhomed]

On 30.12.2015 20:28, Daniel K. wrote:


nginx now requires configured and up network, before starting daemon.
Replace network.target with network-online.target is easy workaround.


Actually it does not require that at all.


nginx failed to start if network is down via systemd race condition.


It would be more helpful if you posted your config files, but from your
log file i gather they look something like:

server {
listen  172.22.22.202:80;
[...]
}


this is allowed syntax:

http://nginx.org/en/docs/http/ngx_http_core_module.html#listen


And that, due to using systemd, the nginx service gets started before
the network-interface have been configured with the IP address shown.


Yes. And nginx failed to start with *correct* config.


Two ways you can work around this issue comes to mind.

1) Allow non-local binds

# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

Put 'net.ipv4.ip_nonlocal_bind = 1' in /etc/sysctl.conf to make it stick.


2) Configure nginx to listen to *:80

Add this to your config files somewhere.

server {
listen  80;
}

to your config files somewhere, and nginx will listen to 0.0.0.0:80
instead of every IP address you mention.


Hope that helps,


And I should send this text fragment to all nginx users?

or this text fragment should be included in manual

http://nginx.org/en/docs/http/ngx_http_core_module.html#listen

?

or (better way) workaround should just be included in nginx unit file?

--
Best regards,
 Gena

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Gena Makhomed]

On 30.12.2015 19:59, Jim Popovitch wrote:


BTW, you can read about how openvpn handled this very issue
https://community.openvpn.net/openvpn/ticket/462



You can provide patch with solution?



I see no need for a patch, and I already explained why


nginx failed to start with correct config.

And you don't see any problems with nginx.


including a link to how another service daemon handled the very same issue.


In theory, there is no difference between theory and practice.

But, in practice, there is.

--
Best regards,
 Gena

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Jim Popovitch]

On Wed, Dec 30, 2015 at 10:50 AM, Gena Makhomed  wrote:
> nginx now requires configured and up network, before starting daemon.

Specifically it's your configuration.  You are hardcoding an IP
address to bind to, thereby telling nginx to not start until that IP
is active.

> Replace network.target with network-online.target is easy workaround.

That will prevent nginx from staring in situations where systemd
determines that the external network is not yet active (dhcp, etc.,
etc), yet nginx may still run perfectly fine with split interfaces,
localhost, etc.

-Jim P.

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Artem S. Povaluhin]

Hi!

On 12/30/2015 11:40 PM, Daniel K. wrote:

I never said it wasn't. I just wanted to express what I had pulled out
of my hat based on reading the log you provided. That way you can see if
I'm completely off track, and tell me, and other readers can get the
context of the conversation more easily.


the context is simple.



And that, due to using systemd, the nginx service gets started before
the network-interface have been configured with the IP address shown.


Yes. And nginx failed to start with *correct* config.


Well, syntactically correct, and logically correct is not the same thing.


why this config is correct everywhere except systemd,


Your config makes nginx try to bind to a non-assigned IP address, which
fails. A logical error in your config files.

You have two options to fix it.


1) Allow non-local binds

and we have to hack the OS

2) Configure nginx to listen to *:80

or change it,

in order to not to misconfigure systemd
because of sombody's recommendations?

wbr,
Artem

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Re: Workaround of race condition between systemd and nginx.

[Gena Makhomed]

On 30.12.2015 22:40, Daniel K. wrote:


nginx failed to start if network is down via systemd race condition.



Again, no, nginx failed to start due to a local misconfiguration.


Configuration is correct.

"nginx -t" syntax check say:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok


listen 172.22.22.202:80;



this is allowed syntax:
http://nginx.org/en/docs/http/ngx_http_core_module.html#listen



I never said it wasn't.


You say about "misconfiguration".


And that, due to using systemd, the nginx service gets started before
the network-interface have been configured with the IP address shown.


Yes. And nginx failed to start with *correct* config.


Well, syntactically correct, and logically correct is not the same thing.


My config is syntactically correct *and* it is logically correct too.


Your config makes nginx try to bind to a non-assigned IP address,
which fails. A logical error in your config files.


No. A logical error in nginx unit file
or in systemd source code or in nginx source code.

Result of error is race condition between systemd and nginx.
The simplest workaround of race condition is to fix nginx unit file.


And I should send this text fragment to all nginx users?


I don't know what you should do, I feel like I am still missing a part
of the puzzle.


Yes.

OpenVZ used by hosting providers on multiple hardware nodes.

Not always possible use only "listen 80;" and "listen 443;" directives.


Arguably not better. The link you provided (repeated for context) tells
you this on using network-online.target.

http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

   It is strongly recommended not to pull in this target too liberally:
   [...] network server software should generally not pull this in


"should generally not pull this in".

Workaround is not "generally".


There you have it; the systemd folks tell us that your suggested
workaround is not a good idea to use for server software.


Systemd folks tell me and other nginx developers how *exactly* nginx
should work. You have time and money to rewrite core parts of nginx?

--
Best regards,
 Gena

___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

Workaround of race condition between systemd and nginx.

[Gena Makhomed]

# HG changeset patch
# User Gena Makhomed 
# Date 1451482795 18000
#  Wed Dec 30 08:39:55 2015 -0500
# Node ID a340d271b3ffa51c0396a5afc5270cb02b701204
# Parent  1073d7e4e430ddb53b603d151e1a403d10aa420b
Workaround of race condition between systemd and nginx.

Just replace network.target with network-online.target in systemd unit 
files.
More details: 
http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/


diff -r 1073d7e4e430 -r a340d271b3ff rpm/SOURCES/nginx-debug.service
--- a/rpm/SOURCES/nginx-debug.service   Wed Dec 09 18:31:08 2015 +0300
+++ b/rpm/SOURCES/nginx-debug.service   Wed Dec 30 08:39:55 2015 -0500
@@ -1,7 +1,7 @@
 [Unit]
 Description=nginx - high performance web server
 Documentation=http://nginx.org/en/docs/
-After=network.target remote-fs.target nss-lookup.target
+After=network-online.target remote-fs.target nss-lookup.target

 [Service]
 Type=forking
diff -r 1073d7e4e430 -r a340d271b3ff rpm/SOURCES/nginx.service
--- a/rpm/SOURCES/nginx.service Wed Dec 09 18:31:08 2015 +0300
+++ b/rpm/SOURCES/nginx.service Wed Dec 30 08:39:55 2015 -0500
@@ -1,7 +1,7 @@
 [Unit]
 Description=nginx - high performance web server
 Documentation=http://nginx.org/en/docs/
-After=network.target remote-fs.target nss-lookup.target
+After=network-online.target remote-fs.target nss-lookup.target

 [Service]
 Type=forking
# HG changeset patch
# User Gena Makhomed 
# Date 1451482795 18000
#  Wed Dec 30 08:39:55 2015 -0500
# Node ID a340d271b3ffa51c0396a5afc5270cb02b701204
# Parent  1073d7e4e430ddb53b603d151e1a403d10aa420b
Workaround of race condition between systemd and nginx.

Just replace network.target with network-online.target in systemd unit files.
More details: http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

diff -r 1073d7e4e430 -r a340d271b3ff rpm/SOURCES/nginx-debug.service
--- a/rpm/SOURCES/nginx-debug.service   Wed Dec 09 18:31:08 2015 +0300
+++ b/rpm/SOURCES/nginx-debug.service   Wed Dec 30 08:39:55 2015 -0500
@@ -1,7 +1,7 @@
 [Unit]
 Description=nginx - high performance web server
 Documentation=http://nginx.org/en/docs/
-After=network.target remote-fs.target nss-lookup.target
+After=network-online.target remote-fs.target nss-lookup.target
 
 [Service]
 Type=forking
diff -r 1073d7e4e430 -r a340d271b3ff rpm/SOURCES/nginx.service
--- a/rpm/SOURCES/nginx.service Wed Dec 09 18:31:08 2015 +0300
+++ b/rpm/SOURCES/nginx.service Wed Dec 30 08:39:55 2015 -0500
@@ -1,7 +1,7 @@
 [Unit]
 Description=nginx - high performance web server
 Documentation=http://nginx.org/en/docs/
-After=network.target remote-fs.target nss-lookup.target
+After=network-online.target remote-fs.target nss-lookup.target
  
 [Service]
 Type=forking
___
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel