Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Jonathan T. Looney]

On Wed, Mar 31, 2021 at 3:55 AM Christos Chatzaras 
wrote:

>
> With the patch I can't reproduce the issue. Can you please MFC it to
> 13.0-STABLE and 13.0 ?
>

Thanks for doing the testing! I've committed the MFC to stable/13. I'll
request re@ approval to pull it up to 13.0.

Jonathan
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Christos Chatzaras]

> On 31 Mar 2021, at 02:31, Jonathan T. Looney  wrote:
> 
> On Tue, Mar 30, 2021 at 12:01 PM Christos Chatzaras  > wrote:
> 
> Did you notice it with nginx too or something else? I also run monit, bind, 
> dovecot, postfix, mysql, php-fpm and pure-ftpd and didn't notice any restart 
> of these services to "stuck".
> 
> Yes, we noticed it in Nginx; however, we only saw it on a very small 
> percentage of servers. I think we might have seen one other instance with a 
> different process, but I can't remember for sure.
> 
> Jonathan


Hello,

With the patch I can't reproduce the issue. Can you please MFC it to 
13.0-STABLE and 13.0 ?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Christos Chatzaras]

> On 31 Mar 2021, at 02:31, Jonathan T. Looney  wrote:
> 
> On Tue, Mar 30, 2021 at 12:01 PM Christos Chatzaras  > wrote:
> 
> Did you notice it with nginx too or something else? I also run monit, bind, 
> dovecot, postfix, mysql, php-fpm and pure-ftpd and didn't notice any restart 
> of these services to "stuck".
> 
> Yes, we noticed it in Nginx; however, we only saw it on a very small 
> percentage of servers. I think we might have seen one other instance with a 
> different process, but I can't remember for sure.
> 
> Jonathan

I rebuild the kernel with the patch. I will reply again in few hours with the 
results.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Jonathan T. Looney]

On Tue, Mar 30, 2021 at 12:01 PM Christos Chatzaras 
wrote:

>
> Did you notice it with nginx too or something else? I also run monit,
> bind, dovecot, postfix, mysql, php-fpm and pure-ftpd and didn't notice any
> restart of these services to "stuck".
>

Yes, we noticed it in Nginx; however, we only saw it on a very small
percentage of servers. I think we might have seen one other instance with a
different process, but I can't remember for sure.

Jonathan
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Christos Chatzaras]

> On 30 Mar 2021, at 18:40, Jonathan T. Looney  wrote:
> 
> Yes, it could be related. Because of the timing of when we first saw the 
> behavior, I assumed the trigger for the behavior (which remains unknown) only 
> existed in main. However, it is possible that this behavior is now being 
> triggered in 13.0.
> 
> Are you able to reliably reproduce this? I was never able to do that. A 
> reliable reproduction may help narrow down the change which triggered this 
> behavior.
> 
> I can certainly MFC the patch to stable/13. re@ will need to decide whether 
> to admit it to the release branch.
> 
> Jonathan

Hello Jonathan,

I have 100+ nginx servers and I can reproduce it in many of them. It doesn't 
happen always in the same servers. Also after a "killlall -9 nginx && service 
nginx restart"  a second "service nginx restart" few seconds after the first 
restart completes successfully. I believe it needs few minutes/hours running 
nginx before the restart "stuck" in "Waiting for PIDS: 85499."

Did you notice it with nginx too or something else? I also run monit, bind, 
dovecot, postfix, mysql, php-fpm and pure-ftpd and didn't notice any restart of 
these services to "stuck".
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Jonathan T. Looney]

On Mon, Mar 29, 2021 at 3:57 PM Christos Chatzaras 
wrote:

> Hello,
>
>
> I upgrade from 12.2 to 13.0-RC4 and I notice a strange issue with Nginx.
>
> When I run "service nginx restart" in some (random) servers it
> doesn't complete the restart and it "stucks" at "Waiting for PIDS: 20536." .
>
> I can kill the 20536 process and then restart completes.
>
>
> procstat -kk 20536:
>
>   PIDTID COMMTDNAME  KSTACK
> 63094 100505 nginx   -   mi_switch+0xc1
> sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1be
> kern_sigsuspend+0x164 sys_sigsuspend+0x31 amd64_syscall+0x10c
> fast_syscall_common+0xf8
>
>
> I found this commit:
>
>
> https://cgit.freebsd.org/src/commit/?id=dbec10e08808e375365fb2a2462f306e0cdfda32
>
> Could this be related?
>

Yes, it could be related. Because of the timing of when we first saw the
behavior, I assumed the trigger for the behavior (which remains unknown)
only existed in main. However, it is possible that this behavior is now
being triggered in 13.0.

Are you able to reliably reproduce this? I was never able to do that. A
reliable reproduction may help narrow down the change which triggered this
behavior.

I can certainly MFC the patch to stable/13. re@ will need to decide whether
to admit it to the release branch.

Jonathan
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Glen Barber]

On Tue, Mar 30, 2021 at 04:48:25PM +0300, Christos Chatzaras wrote:
> 
> 
> > On 30 Mar 2021, at 16:45, Mark Johnston  wrote:
> > 
> > On Mon, Mar 29, 2021 at 10:57:09PM +0300, Christos Chatzaras wrote:
> >> Hello,
> >> 
> >> 
> >> I upgrade from 12.2 to 13.0-RC4 and I notice a strange issue with Nginx.
> >> 
> >> When I run "service nginx restart" in some (random) servers it doesn't 
> >> complete the restart and it "stucks" at "Waiting for PIDS: 20536." .
> >> 
> >> I can kill the 20536 process and then restart completes.
> >> 
> >> 
> >> procstat -kk 20536:
> >> 
> >>  PIDTID COMMTDNAME  KSTACK
> >> 63094 100505 nginx   -   mi_switch+0xc1 
> >> sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1be 
> >> kern_sigsuspend+0x164 sys_sigsuspend+0x31 amd64_syscall+0x10c 
> >> fast_syscall_common+0xf8 
> >> 
> >> 
> >> I found this commit:
> >> 
> >> https://cgit.freebsd.org/src/commit/?id=dbec10e08808e375365fb2a2462f306e0cdfda32
> >>  
> >> 
> >> 
> >> Could this be related? If yes can we have the patch in releng/13.0 ?
> > 
> > I think it is hard to say without some testing.  Are you able to verify
> > that backporting the patch fixes the hangs?
> 
> 
> I will use the patch today night, so I can verify it tomorrow morning.
> 
> If it works do we still have time to land the patch in 13.0?

If it resolves the issue, yes.

Glen



signature.asc
Description: PGP signature

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Christos Chatzaras]

> On 30 Mar 2021, at 16:45, Mark Johnston  wrote:
> 
> On Mon, Mar 29, 2021 at 10:57:09PM +0300, Christos Chatzaras wrote:
>> Hello,
>> 
>> 
>> I upgrade from 12.2 to 13.0-RC4 and I notice a strange issue with Nginx.
>> 
>> When I run "service nginx restart" in some (random) servers it doesn't 
>> complete the restart and it "stucks" at "Waiting for PIDS: 20536." .
>> 
>> I can kill the 20536 process and then restart completes.
>> 
>> 
>> procstat -kk 20536:
>> 
>>  PIDTID COMMTDNAME  KSTACK
>> 63094 100505 nginx   -   mi_switch+0xc1 
>> sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1be 
>> kern_sigsuspend+0x164 sys_sigsuspend+0x31 amd64_syscall+0x10c 
>> fast_syscall_common+0xf8 
>> 
>> 
>> I found this commit:
>> 
>> https://cgit.freebsd.org/src/commit/?id=dbec10e08808e375365fb2a2462f306e0cdfda32
>>  
>> 
>> 
>> Could this be related? If yes can we have the patch in releng/13.0 ?
> 
> I think it is hard to say without some testing.  Are you able to verify
> that backporting the patch fixes the hangs?


I will use the patch today night, so I can verify it tomorrow morning.

If it works do we still have time to land the patch in 13.0?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC4 and Nginx process "stuck" during restart

[Mark Johnston]

On Mon, Mar 29, 2021 at 10:57:09PM +0300, Christos Chatzaras wrote:
> Hello,
> 
> 
> I upgrade from 12.2 to 13.0-RC4 and I notice a strange issue with Nginx.
> 
> When I run "service nginx restart" in some (random) servers it doesn't 
> complete the restart and it "stucks" at "Waiting for PIDS: 20536." .
> 
> I can kill the 20536 process and then restart completes.
> 
> 
> procstat -kk 20536:
> 
>   PIDTID COMMTDNAME  KSTACK
> 63094 100505 nginx   -   mi_switch+0xc1 
> sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1be 
> kern_sigsuspend+0x164 sys_sigsuspend+0x31 amd64_syscall+0x10c 
> fast_syscall_common+0xf8 
> 
> 
> I found this commit:
> 
> https://cgit.freebsd.org/src/commit/?id=dbec10e08808e375365fb2a2462f306e0cdfda32
>  
> 
> 
> Could this be related? If yes can we have the patch in releng/13.0 ?

I think it is hard to say without some testing.  Are you able to verify
that backporting the patch fixes the hangs?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"