Re: [Touch-packages] [Bug 1629226] [NEW] systemd's service killed by cgroup controller pids
Steve,
Can someone from Foundations evaluate this bug? Core issue is that when
child process dies, the parent process is also killed.
Michael
On 09/30/2016 02:02 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> Problem Description
> ===
> I write a simple systemd service which will fork child processes fiercely.
> But quickly the service failed:
>
> % sudo systemctl status reproducer.service
> ? reproducer.service - Reproducer of systemd services killed by ips
>Loaded: loaded (/etc/systemd/system/reproducer.service; disabled; vendor
> preset: enabled)
>Active: failed (Result: exit-code) since Fri 2016-03-18 06:58:37 CDT; 2min
> 43s ago
> Process: 5103 ExecStart=/home/hpt/reproducer/reproducer.sh (code=exited,
> status=0/SUCCESS)
> Main PID: 5105 (code=exited, status=254)
>
> Mar 18 06:58:36 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:36 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Main process exited,
> code=exited, status=254/n/a
> Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
> /home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
> Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Unit entered failed
> state.
> Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Failed with result
> 'exit-code'.
>
> The default task limit of systemd services is 512. Looks like the
> service is terminated by the kernel's ips cgroup controller. I think
> this isn't correct. Child processes cannot be forked shouldn't cause
> parent to die.
>
>
> % cat /etc/systemd/system/reproducer.service
> [Unit]
> Description=Reproducer of systemd services killed by ips
> After=multi-user.target
>
> [Service]
> ExecStart=/home/hpt/reproducer/reproducer.sh
> Type=forking
>
> [Install]
> WantedBy=multi-user.target
>
> % cat /home/hpt/reproducer/reproducer.sh
> #!/bin/bash
>
> foo()
> {
> #exec sh -c "echo $1: \$\$;sleep 60"
> echo $1:
> sleep 60
> }
>
> bar()
> {
> c=1
> while true
> do
> for ((i=1;i<=2048;i++))
> do
> foo $c &
> ((c++))
> done
>
> wait
> c=1
> done
> }
>
> # main
> bar &
>
> disown -a
>
> exit 0
>
>
> ---uname output---
> Linux pinelp3 4.4.0-12-generic #28-Ubuntu SMP Wed Mar 9 00:40:38 UTC 2016
> ppc64le ppc64le ppc64le GNU/Linux
>
> Machine Type = IBM,8408-E8E,lpar
>
> Steps to Reproduce
>
> 1. install the simple service in "Problem description"
> 2. sudo systemctl start reproducer.service
> 3. wait 2~3 minutes
>
> == Comment: #3 - Vaishnavi Bhat - 2016-03-22 11:21:55 ==
> >From the machine,
> root@pinelp3:~# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 48192
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 48192
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> root@pinelp3:~# ps aux | wc -l ->While the service is
> running
> 1084
> root@pinelp3:~# ps aux | wc -l "
> 1084
> root@pinelp3:~# ps aux | wc -l "
> 1084
> root@pinelp3:~# ps aux | wc -l "
> 1084
> root@pinelp3:~# ps aux | wc -l-->While the service is not
> running.
> 572
>
> root@pinelp3:~# free -m --> While service is running
> totalusedfree shared buff/cache
> available
> Mem: 12117 628 459 22 11029
> 9541
> Swap: 2052 82044
>
> root@pinelp3:~# free -m
[Touch-packages] [Bug 1629226] [NEW] systemd's service killed by cgroup controller pids
Public bug reported:
Problem Description
===
I write a simple systemd service which will fork child processes fiercely. But
quickly the service failed:
% sudo systemctl status reproducer.service
? reproducer.service - Reproducer of systemd services killed by ips
Loaded: loaded (/etc/systemd/system/reproducer.service; disabled; vendor
preset: enabled)
Active: failed (Result: exit-code) since Fri 2016-03-18 06:58:37 CDT; 2min
43s ago
Process: 5103 ExecStart=/home/hpt/reproducer/reproducer.sh (code=exited,
status=0/SUCCESS)
Main PID: 5105 (code=exited, status=254)
Mar 18 06:58:36 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:36 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Main process exited,
code=exited, status=254/n/a
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Unit entered failed
state.
Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Failed with result
'exit-code'.
The default task limit of systemd services is 512. Looks like the
service is terminated by the kernel's ips cgroup controller. I think
this isn't correct. Child processes cannot be forked shouldn't cause
parent to die.
% cat /etc/systemd/system/reproducer.service
[Unit]
Description=Reproducer of systemd services killed by ips
After=multi-user.target
[Service]
ExecStart=/home/hpt/reproducer/reproducer.sh
Type=forking
[Install]
WantedBy=multi-user.target
% cat /home/hpt/reproducer/reproducer.sh
#!/bin/bash
foo()
{
#exec sh -c "echo $1: \$\$;sleep 60"
echo $1:
sleep 60
}
bar()
{
c=1
while true
do
for ((i=1;i<=2048;i++))
do
foo $c &
((c++))
done
wait
c=1
done
}
# main
bar &
disown -a
exit 0
---uname output---
Linux pinelp3 4.4.0-12-generic #28-Ubuntu SMP Wed Mar 9 00:40:38 UTC 2016
ppc64le ppc64le ppc64le GNU/Linux
Machine Type = IBM,8408-E8E,lpar
Steps to Reproduce
1. install the simple service in "Problem description"
2. sudo systemctl start reproducer.service
3. wait 2~3 minutes
== Comment: #3 - Vaishnavi Bhat - 2016-03-22 11:21:55 ==
>From the machine,
root@pinelp3:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 48192
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 48192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@pinelp3:~# ps aux | wc -l ->While the service is running
1084
root@pinelp3:~# ps aux | wc -l "
1084
root@pinelp3:~# ps aux | wc -l "
1084
root@pinelp3:~# ps aux | wc -l "
1084
root@pinelp3:~# ps aux | wc -l-->While the service is not running.
572
root@pinelp3:~# free -m --> While service is running
totalusedfree shared buff/cache available
Mem: 12117 628 459 22 110299541
Swap: 2052 82044
root@pinelp3:~# free -m > while the service is not running.
totalusedfree shared buff/cache available
Mem: 12117 308 809 22 109999890
Swap: 2052 82044
== Comment: #4 - Breno Henrique Leitao - 2016-03-22
11:48:57 ==
This is a new feature in Ubuntu and Systemd that limits the amount of
processes/child created.
You can disable it as doucmented in
[Touch-packages] [Bug 1629226] [NEW] systemd's service killed by cgroup controller pids
You have been subscribed to a public bug:
Problem Description
===
I write a simple systemd service which will fork child processes fiercely. But
quickly the service failed:
% sudo systemctl status reproducer.service
? reproducer.service - Reproducer of systemd services killed by ips
Loaded: loaded (/etc/systemd/system/reproducer.service; disabled; vendor
preset: enabled)
Active: failed (Result: exit-code) since Fri 2016-03-18 06:58:37 CDT; 2min
43s ago
Process: 5103 ExecStart=/home/hpt/reproducer/reproducer.sh (code=exited,
status=0/SUCCESS)
Main PID: 5105 (code=exited, status=254)
Mar 18 06:58:36 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:36 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Main process exited,
code=exited, status=254/n/a
Mar 18 06:58:37 pinelp3 reproducer.sh[5103]:
/home/hpt/reproducer/reproducer.sh: fork: Resource temporarily unavailable
Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Unit entered failed
state.
Mar 18 06:58:37 pinelp3 systemd[1]: reproducer.service: Failed with result
'exit-code'.
The default task limit of systemd services is 512. Looks like the
service is terminated by the kernel's ips cgroup controller. I think
this isn't correct. Child processes cannot be forked shouldn't cause
parent to die.
% cat /etc/systemd/system/reproducer.service
[Unit]
Description=Reproducer of systemd services killed by ips
After=multi-user.target
[Service]
ExecStart=/home/hpt/reproducer/reproducer.sh
Type=forking
[Install]
WantedBy=multi-user.target
% cat /home/hpt/reproducer/reproducer.sh
#!/bin/bash
foo()
{
#exec sh -c "echo $1: \$\$;sleep 60"
echo $1:
sleep 60
}
bar()
{
c=1
while true
do
for ((i=1;i<=2048;i++))
do
foo $c &
((c++))
done
wait
c=1
done
}
# main
bar &
disown -a
exit 0
---uname output---
Linux pinelp3 4.4.0-12-generic #28-Ubuntu SMP Wed Mar 9 00:40:38 UTC 2016
ppc64le ppc64le ppc64le GNU/Linux
Machine Type = IBM,8408-E8E,lpar
Steps to Reproduce
1. install the simple service in "Problem description"
2. sudo systemctl start reproducer.service
3. wait 2~3 minutes
== Comment: #3 - Vaishnavi Bhat - 2016-03-22 11:21:55 ==
>From the machine,
root@pinelp3:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 48192
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 48192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@pinelp3:~# ps aux | wc -l ->While the service is running
1084
root@pinelp3:~# ps aux | wc -l "
1084
root@pinelp3:~# ps aux | wc -l "
1084
root@pinelp3:~# ps aux | wc -l "
1084
root@pinelp3:~# ps aux | wc -l-->While the service is not running.
572
root@pinelp3:~# free -m --> While service is running
totalusedfree shared buff/cache available
Mem: 12117 628 459 22 110299541
Swap: 2052 82044
root@pinelp3:~# free -m > while the service is not running.
totalusedfree shared buff/cache available
Mem: 12117 308 809 22 109999890
Swap: 2052 82044
== Comment: #4 - Breno Henrique Leitao - 2016-03-22
11:48:57 ==
This is a new feature in Ubuntu and Systemd that limits the amount of
processes/child created.
You can disable it as doucmented in
