Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory
Hi Ian, Thanks for your reply. In the meantime I found out that I forgot to change the device name of the swap devices in fstab of the domUs when upgrading to XEN 4.0. So in fact all domUs had no swap space at all. I think this might be the problem that a domU ran out of memory... However, the dom0 always had a swap space, so the question is still why the dom0 also runs out of memory. May 2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 May 2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0 May 2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1 Am I right that nrpe is part of nagios? (it's probably just the unluck process so tells us nothing really) You are right - this log message was just an example of what happens. What sort of load are the domUs experiencing? i.e. CPU, network, disk etc. CPU and memory consumption What does your storage stack look like? (are you using LVM, iSCSI, DRDB, SW RAID, filesystems etc). SW Raid Are you running anything interesting in domain 0 other than the Xen toolstack, nagios, sshd etc? nagios nrpe, sshd, fail2ban-server, bacula-fd, rsyncd, postfix What does /proc/meminfo look like after a fresh boot? Lastly, please can you provide a dmesg log of the initial bootup. I will provide the requested information as soon as I can reboot the machine. We also will do some tests with high load again on the domUs with enabled swap space to see if the problem still exists. Cheers, Ian. Thanks Sebastian -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4dd531b5.2040...@sourcepark.de
Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory
Hi Sebastian, Thanks for your report. On Mon, 2011-05-02 at 16:50 +0200, Sebastian Hofmann wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-31 Severity: critical Justification: breaks the whole system Hi, I have 64 bit xen kernel from squeeze installed on a dual xeon maschine. Usually everything runs fine until it comes to heavy load on a domU with high I/O and memory consumption. This causes the dom0 to run out of memory and to kill several processes (see log below). As a consequence of this, the whole system becomes unusable. I tried several things like assign dedicated memory to dom0, disable balloning, increase scheduler domain weights and assigned dedicated CPUs to dom0 as described in http://wiki.xensource.com/xenwiki/XenBestPractices but had no success. I think a domU should never break the whole system, so this might be a bug. Please let me know if you need further information. Thanks Sebastian May 2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 May 2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0 May 2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1 Am I right that nrpe is part of nagios? (it's probably just the unluck process so tells us nothing really) What sort of load are the domUs experiencing? i.e. CPU, network, disk etc. What does your storage stack look like? (are you using LVM, iSCSI, DRDB, SW RAID, filesystems etc). Are you running anything interesting in domain 0 other than the Xen toolstack, nagios, sshd etc? What does /proc/meminfo look like after a fresh boot? Lastly, please can you provide a dmesg log of the initial bootup. Cheers, Ian. -- Ian Campbell Current Noise: Behemoth - Alas, Lord Is Upon Me To stay young requires unceasing cultivation of the ability to unlearn old falsehoods. -- Lazarus Long, Time Enough For Love -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1305188425.26692.471.ca...@zakaz.uk.xensource.com
Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory
Hi, The maschine has 12G of RAM. For dom0 I tried several configurations - 4G with balloning enabled, at least 1G statically assigned. DomUs have different memory settings. xm info | grep mem: total_memory : 12224 free_memory: 1589 node_to_memory : node0:1077 node_to_dma32_mem : node0:1074 xen_commandline: placeholder dom0_mem=1024M dom0_max_vcpus=2 dom0_vcpus_pin dom0 free -m: total used free sharedbuffers cached Mem: 1019 1008 10 0249528 -/+ buffers/cache:231788 Swap: 1913 29 1884 xm list: NameID Mem VCPUs State Time(s) Domain-0 0 1019 2 r- 12087.5 alfresco 1 3096 1 -b 57.0 destiny 3 384 1 -b 919.5 hudson32 4 384 1 -b 165.2 hudson64 5 1536 1 -b 2697.1 ms2cistage 7 2048 1 -b 10.5 ms2istage6 2048 1 -b 19.1 Thanks Sebastian Am 02.05.2011 19:35, schrieb Ben Hutchings: On Mon, 2011-05-02 at 16:50 +0200, Sebastian Hofmann wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-31 Severity: critical Justification: breaks the whole system Hi, I have 64 bit xen kernel from squeeze installed on a dual xeon maschine. Usually everything runs fine until it comes to heavy load on a domU with high I/O and memory consumption. This causes the dom0 to run out of memory and to kill several processes (see log below). As a consequence of this, the whole system becomes unusable. I tried several things like assign dedicated memory to dom0, disable balloning, increase scheduler domain weights and assigned dedicated CPUs to dom0 as described in http://wiki.xensource.com/xenwiki/XenBestPractices but had no success. I think a domU should never break the whole system, so this might be a bug. Please let me know if you need further information. [...] How much physical RAM is in the system and how much memory do you assign to each domain? Ben. -- -- M.Sc. Sebastian Hofmann SOURCEPARK GmbH Hohenzollerndamm 150 Haus 7a 14199 Berlin Tel:+49 (0)30/398 068 30 Fax:+49 (0)30/398 068 39 e-mail: sebastian.hofm...@sourcepark.de www:www.sourcepark.de -- SOURCEPARK GmbH Sitz der Gesellschaft: Berlin / Amtsgericht Charlottenburg HRB 80254 Geschäftsführung: Matthias Barmeier, Harald Dürr -- Wichtiger Hinweis: Die vorgenannten Angaben werden jeder E-Mail automatisch hinzugefügt und lassen keine Rückschlüsse auf den Rechtscharakter der E-Mail zu. Diese E-Mail kann vertrauliche und/oder rechtlich geschützte Informationen enthalten. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4dbfb1e2.6080...@sourcepark.de
Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory
Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-31 Severity: critical Justification: breaks the whole system Hi, I have 64 bit xen kernel from squeeze installed on a dual xeon maschine. Usually everything runs fine until it comes to heavy load on a domU with high I/O and memory consumption. This causes the dom0 to run out of memory and to kill several processes (see log below). As a consequence of this, the whole system becomes unusable. I tried several things like assign dedicated memory to dom0, disable balloning, increase scheduler domain weights and assigned dedicated CPUs to dom0 as described in http://wiki.xensource.com/xenwiki/XenBestPractices but had no success. I think a domU should never break the whole system, so this might be a bug. Please let me know if you need further information. Thanks Sebastian May 2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 May 2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0 May 2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1 May 2 16:05:26 hercules kernel: [ 1768.319889] Call Trace: May 2 16:05:26 hercules kernel: [ 1768.319899] [810b7104] ? oom_kill_process+0x7f/0x23f May 2 16:05:26 hercules kernel: [ 1768.319903] [810b7628] ? __out_of_memory+0x12a/0x141 May 2 16:05:26 hercules kernel: [ 1768.319906] [810b777f] ? out_of_memory+0x140/0x172 May 2 16:05:26 hercules kernel: [ 1768.319912] [81042ace] ? pick_next_task_fair+0xca/0xd6 May 2 16:05:26 hercules kernel: [ 1768.319916] [810bb520] ? __alloc_pages_nodemask+0x4e5/0x5f5 May 2 16:05:26 hercules kernel: [ 1768.319922] [810d9c04] ? read_swap_cache_async+0x5d/0xf3 May 2 16:05:27 hercules kernel: [ 1768.319926] [810d9d23] ? swapin_readahead+0x89/0x98 May 2 16:05:27 hercules kernel: [ 1768.319931] [8100c2f1] ? __raw_callee_save_xen_pte_val+0x11/0x1e May 2 16:05:27 hercules kernel: [ 1768.319935] [810cdc18] ? handle_mm_fault+0x47f/0x80f May 2 16:05:27 hercules kernel: [ 1768.319940] [8106d41f] ? ktime_get_ts+0x68/0xb2 May 2 16:05:27 hercules kernel: [ 1768.319945] [8130f616] ? do_page_fault+0x2e0/0x2fc May 2 16:05:27 hercules kernel: [ 1768.319951] [8130d4b5] ? page_fault+0x25/0x30 May 2 16:05:27 hercules kernel: [ 1768.319953] Mem-Info: May 2 16:05:27 hercules kernel: [ 1768.319955] Node 0 DMA per-cpu: May 2 16:05:27 hercules kernel: [ 1768.319958] CPU0: hi:0, btch: 1 usd: 0 May 2 16:05:27 hercules kernel: [ 1768.319960] CPU1: hi:0, btch: 1 usd: 0 May 2 16:05:27 hercules kernel: [ 1768.319961] Node 0 DMA32 per-cpu: May 2 16:05:27 hercules kernel: [ 1768.319964] CPU0: hi: 186, btch: 31 usd: 18 May 2 16:05:27 hercules kernel: [ 1768.319966] CPU1: hi: 186, btch: 31 usd: 0 May 2 16:05:28 hercules kernel: [ 1768.319971] active_anon:1 inactive_anon:25 isolated_anon:0 May 2 16:05:28 hercules kernel: [ 1768.319972] active_file:15047 inactive_file:196012 isolated_file:128 May 2 16:05:28 hercules kernel: [ 1768.319973] unevictable:5 dirty:196071 writeback:143 unstable:0 May 2 16:05:28 hercules kernel: [ 1768.319974] free:2030 slab_reclaimable:14633 slab_unreclaimable:4593 May 2 16:05:28 hercules kernel: [ 1768.319975] mapped:3592 shmem:0 pagetables:957 bounce:0 May 2 16:05:28 hercules kernel: [ 1768.319977] Node 0 DMA free:4016kB min:52kB low:64kB high:76kB active_anon:0kB inactive_anon:0kB active_file:196kB inactive_file:9112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13828kB mlocked:0kB dirty:9128kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:592kB slab_unreclaimable:60kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10656 all_unreclaimable? no May 2 16:05:28 hercules kernel: [ 1768.319986] lowmem_reserve[]: 0 994 994 994 May 2 16:05:28 hercules kernel: [ 1768.319991] Node 0 DMA32 free:4104kB min:4004kB low:5004kB high:6004kB active_anon:4kB inactive_anon:100kB active_file:59992kB inactive_file:774936kB unevictable:20kB isolated(anon):0kB isolated(file):512kB present:1018080kB mlocked:20kB dirty:775156kB writeback:572kB mapped:14368kB shmem:0kB slab_reclaimable:57940kB slab_unreclaimable:18312kB kernel_stack:1376kB pagetables:3828kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:441765 all_unreclaimable? no May 2 16:05:28 hercules kernel: [ 1768.320001] lowmem_reserve[]: 0 0 0 0 May 2 16:05:28 hercules kernel: [ 1768.320005] Node 0 DMA: 4*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4016kB May 2 16:05:28 hercules kernel: [ 1768.320019] Node 0 DMA32: 28*4kB 265*8kB 115*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4104kB May 2 16:05:28 hercules kernel: [ 1768.320030] 211215 total pagecache pages May 2 16:05:28
Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory
On Mon, 2011-05-02 at 16:50 +0200, Sebastian Hofmann wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-31 Severity: critical Justification: breaks the whole system Hi, I have 64 bit xen kernel from squeeze installed on a dual xeon maschine. Usually everything runs fine until it comes to heavy load on a domU with high I/O and memory consumption. This causes the dom0 to run out of memory and to kill several processes (see log below). As a consequence of this, the whole system becomes unusable. I tried several things like assign dedicated memory to dom0, disable balloning, increase scheduler domain weights and assigned dedicated CPUs to dom0 as described in http://wiki.xensource.com/xenwiki/XenBestPractices but had no success. I think a domU should never break the whole system, so this might be a bug. Please let me know if you need further information. [...] How much physical RAM is in the system and how much memory do you assign to each domain? Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
