Ok, I've narrowed it down to the brk and stack stressors in stress-ng.
I've recreated this on amd64, ppc64el and s390x across 4.4, 4.10 and
4.13 kernels.
Stress-NG reports the tests as successful so I guess now the only thing
remaining is to confirm that hitting OOM killer is expected behaviour
when running these stressors.
I've tried setting up swap space that is three times the amount of
physical RAM and this is still invoking the OOMkiller.
ubuntu@xwing:~$ free -lm
total used free shared buff/cache available
Mem: 3838 105 3598 0 133 3517
Low: 3838 239 3598
High: 0 0 0
Swap: 11514 68 11446
And from looking at vmstat while it's running:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 37 3618 5 138 0 2 26 2880 154 138 0 1 60 39 0
0 9 257 82 1 114 0 22 183 23495 1284 907 0 3 61 36 0
0 9 891 84 0 110 0 63 9 65792 3266 1865 0 2 6 92 0
0 10 1601 87 0 108 0 71 1 73444 3466 1682 0 3 6 92 0
1 8 2300 83 0 108 0 70 6 72316 3494 1673 0 3 6 91 0
0 10 2926 82 1 106 0 63 165 64884 2940 1593 0 2 8 90 0
0 9 3592 95 0 106 0 67 69 68825 2987 1672 0 3 8 90 0
0 9 4326 82 0 104 0 73 16 76003 3376 1645 0 3 5 92 0
0 9 4910 81 0 103 0 58 177 60429 2713 1596 0 2 8 90 0
0 10 5581 81 0 104 0 67 122 69519 2680 1581 0 2 13 85 0
0 10 6317 89 0 105 0 74 13 76143 3389 1613 0 3 5 92 0
0 9 6972 83 1 104 0 66 103 67807 2964 1711 0 3 7 90 0
0 9 7591 103 1 105 0 62 39 64041 2918 1530 0 3 8 89 0
0 9 8248 84 1 105 0 66 17 68101 3208 1573 0 3 6 91 0
0 10 8946 82 1 105 0 70 0 72289 2934 1615 0 3 7 90 0
0 9 9586 82 1 104 0 64 60 66192 2953 1725 0 3 6 90 0
0 9 10276 100 1 104 0 69 3 71434 2986 1694 0 3 9 88 0
0 10 10958 82 1 103 0 68 11 70660 3365 1704 0 4 8 88 0
0 9 11455 82 1 102 0 50 53 51572 2269 1533 0 2 15 82 0
0 11 10249 87 0 106 0 26 783 27601 1323 1721 0 2 16 82 0
0 10 10407 82 0 105 0 16 6 16883 1487 1438 0 1 39 60 0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 11 10543 82 0 109 0 14 558 14774 1523 2202 0 1 31 68 0
0 12 10672 82 1 113 0 13 598 13945 1552 2592 0 1 17 82 0
1 8 10844 84 1 115 0 17 499 18516 1434 1921 0 1 22 77 0
0 9 11080 81 1 113 0 24 0 25004 1782 1547 0 2 32 67 0
0 9 11338 82 1 113 0 26 2 27162 1874 1606 0 2 25 73 0
0 13 9950 82 0 105 0 25 636 26475 1672 1657 0 2 21 77 0
0 10 10167 81 0 105 0 22 111 23060 1956 1766 0 1 26 73 0
0 9 10368 81 0 106 0 20 62 21257 1687 1470 0 1 30 68 0
1 8 10559 81 1 106 0 19 92 20380 1660 1542 0 1 30 68 0
0 9 10815 81 1 106 0 26 0 26964 1960 1523 0 2 30 68 0
0 9 237 95 1 116 0 34 714 35614 2065 2124 0 5 32 63 0
0 9 851 84 1 115 0 61 31 63467 3612 2607 0 2 4 94 0
0 10 1468 82 1 115 0 62 130 63985 3167 2316 0 2 5 92 0
0 9 2156 83 1 115 0 69 12 71113 3476 2294 0 3 4 93 0
0 9 2837 88 1 115 0 68 33 70577 3007 2155 0 3 5 92 0
0 10 3501 82 1 110 0 66 10 68809 3015 2202 0 2 8 90 0
0 9 4194 82 1 109 0 69 37 71556 3648 2597 0 3 5 92 0
7 1 4901 82 1 109 0 71 3 73264 3368 2087 0 3 8 90 0
0 10 5551 82 0 105 0 65 183 67353 3121 2189 0 3 9 88 0
0 10 6241 82 0 105 0 69 79 71325 3448 2414 0 3 7 90 0
0 9 6901 86 1 105 0 66 58 68414 3055 2254 0 3 6 91 0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 10 7619 93 1 105 0 72 12 74423 3439 2329 0 3 7 89 0
0 9 8338 96 1 105 0 72 10 74165 3498 2213 0 4 7 90 0
0 10 8851 82 1 104 0 51 201 53453 3791 2960 0 3 5 92 0
0 9 9406 88 1 105 0 55 113 57404 4274 3311 0 3 5 91 0
1 8 10015 84 1 106 0 61 4 63194 4405 3599 0 4 5 91 0
0 9 10607 82 1 106 0 59 6 61462 4426 3353 0 4 6 90 0
0 9 11143 81 1 105 0 54 0 55738 4029 3214 0 3 5 92 0
1 9 11512 82 1 105 0 37 2 38579 3300 2528 0 3 5 92 0
0 11 10280 82 0 106 0 25 1129 26677 2329 2912 0 2 12 86 0
1 8 10373 82 1 110 0 8 454 9262 1409 1976 0 1 34 65 0
1 10 10476 82 1 112 0 10 327 11458 1415 1763 0 1 37 63 0
1 8 10592 82 2 112 0 12 56 12682 1437 1510 0 1 40 59 0
0 9 10726 81 2 112 0 13 12 14585 1489 1449 0 1 46 53 0
0 9 10927 82 2 112 0 20 0 21362 1916 1577 0 2 41 57 0
0 9 11125 82 2 113 0 20 113 21121 1948 1669 0 2 32 67 0
0 10 11328 82 2 113 0 20 16 21774 2122 1532 0 2 39 59 0
0 14 9879 82 0 102 0 23 220 24158 2258 2051 0 2 37 61 0
0 10 10075 81 0 106 0 19 547 20253 1692 1635 0 1 22 77 0
0 10 10236 83 0 105 0 16 0 17262 1903 1699 0 1 30 69 0
0 9 10368 82 0 105 0 13 0 14341 1674 1488 0 1 34 65 0
0 9 10516 82 0 106 0 15 0 15947 1595 1433 0 1 36 63 0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 9 10696 82 0 107 0 18 60 19262 1875 1551 0 1 31 68 0
0 9 10893 82 0 106 0 20 7 21052 1972 1624 0 2 34 65 0
0 0 69 3624 0 110 0 5 519 6506 416 471 0 1 81 18 0
0 0 69 3624 0 110 0 0 0 1230 36 51 0 0 100 0 0
0 0 69 3624 0 110 0 0 0 1027 36 60 0 0 100 0 0
those two tests are consuming everything available and then swapping
until there's nothing left, thus hitting the OOMKiller.
So this started because I'd not noticed the stack traces before because
stress-ng says it passed without error.
Thus, as mentioned above, we just need to determine if this is a
legitimate issue with testing, or expected behaviour for the brk and
stack stressors to hit OOMkiller.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1729878
Title:
stress-ng triggering oomkiller running brk and stack stressors on all
arches.
Status in Stress-ng:
New
Status in linux package in Ubuntu:
Confirmed
Status in linux source package in Artful:
Confirmed
Bug description:
During regression testing on a z/VM instance, I noticed call traces
being dumped to dmesg that seem to be related to stress-ng.
The stress-ng invocation we're using is:
stress-ng --aggressive --verify --timeout $runtime \
--metrics-brief --tz --times \
--af-alg 0 --bsearch 0 --context 0 --cpu 0 \
--crypt 0 --hsearch 0 --longjmp 0 --lsearch
0 \
--matrix 0 --qsort 0 --str 0 --stream 0 \
--tsearch 0 --vecmath 0 --wcs 0
as executed by the cpu_stress script from the cpu/cpu_stress_ng test
in the certification suite.
ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: linux-image-4.13.0-16-generic 4.13.0-16.19
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic s390x
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code
2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.7-0ubuntu3.1
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord':
'arecord'
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211
not found.
Date: Fri Nov 3 11:12:04 2017
HibernationDevice: RESUME=UUID=1ce0f037-449a-43a1-af49-e730f6f99ac4
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1:
PciMultimedia:
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C
SHELL=/bin/bash
ProcFB: Error: [Errno 2] No such file or directory: '/proc/fb'
ProcKernelCmdLine: root=/dev/mapper/hwe0008_vg-lv crashkernel=196M
BOOT_IMAGE=0
RelatedPackageVersions:
linux-restricted-modules-4.13.0-16-generic N/A
linux-backports-modules-4.13.0-16-generic N/A
linux-firmware 1.169
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to artful on 2017-11-02 (0 days ago)
---
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code
2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.7-0ubuntu3.1
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211
not found.
DistroRelease: Ubuntu 17.10
HibernationDevice: RESUME=UUID=1ce0f037-449a-43a1-af49-e730f6f99ac4
IwConfig: Error: [Errno 2] No such file or directory
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C
SHELL=/bin/bash
ProcFB: Error: [Errno 2] No such file or directory: '/proc/fb'
ProcKernelCmdLine: root=/dev/mapper/hwe0008_vg-lv crashkernel=196M
BOOT_IMAGE=0
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
RelatedPackageVersions:
linux-restricted-modules-4.13.0-16-generic N/A
linux-backports-modules-4.13.0-16-generic N/A
linux-firmware 1.169
RfKill: Error: [Errno 2] No such file or directory
Tags: artful
Uname: Linux 4.13.0-16-generic s390x
UpgradeStatus: Upgraded to artful on 2017-11-02 (0 days ago)
UserGroups: adm cdrom cpacfstats dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
To manage notifications about this bug go to:
https://bugs.launchpad.net/stress-ng/+bug/1729878/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
