Yeah, may be the kernel bug Problem have been solved with updated kernel 2.6.32-358.11.1.el6.x86_64 and latest network card driver igb 4.2.16
Thanks and Regards -yao -----邮件原件----- 发件人: Serge Hallyn [mailto:serge.hal...@ubuntu.com] 发送时间: 2013年6月13日 20:44 收件人: yaozhicheng 抄送: lxc-users@lists.sourceforge.net 主题: Re: [Lxc-users] 转发: lxc scalability problem Uh, yeah, wow. This looks like a bug in the debian kernel. I'd suggest trying to reproduce this with a small shellscript so you can report it. -serge Quoting yaozhicheng (yaozhich...@emails.bjut.edu.cn): > > > -----邮件原件----- > 发件人: yaozhicheng [mailto:yaozhich...@emails.bjut.edu.cn] > 发送时间: 2013年6月8日 23:32 > 收件人: 'Serge Hallyn' > 主题: 答复: [Lxc-users] lxc scalability problem > > Dear serge, > Sorry for my delayed reply :) > Problem Still persists with selinux disabled. Maybe it is not the > memory problem because that my machine has 2TB memory and 8 x E7-8830 cpus. > The OS in the containers is debian6.0. I have run filebench and > lmbench in 8 containers simultaneously, they performed very well. > 100 more containers can be started when physical eth2 removed out from br0. > May be the veth network problem? > > This is part of the vmcore-dmesg.txt(selinux disabled): > > <6>br0: port 5(vethOaLJ1U) entering forwarding state > <6>br0: port 4(vethZZ0xuP) entering forwarding state > <6>br0: port 8(vethjAE1Pb) entering forwarding state > <6>br0: port 9(veth0SqVmc) entering forwarding state > <6>br0: port 16(vethtVuHLb) entering forwarding state > <6>br0: port 7(vethtTiLzY) entering forwarding state > <6>br0: port 15(vethxWoVMd) entering forwarding state > <6>br0: port 12(vethvSfzsf) entering forwarding state > <6>br0: port 13(vethw78ho7) entering forwarding state > <6>br0: port 11(vethrf5HMe) entering forwarding state > <6>br0: port 6(veth4Vm0m0) entering forwarding state > <6>br0: port 10(vethho7Oae) entering forwarding state > <6>br0: port 20(vethIsYsGj) entering forwarding state > <6>br0: port 14(vethXaQOHa) entering forwarding state > <6>br0: port 17(veth8rD1je) entering forwarding state <4>------------[ > cut here ]------------ <2>kernel BUG at mm/slab.c:3069! > <4>invalid opcode: 0000 [#1] SMP > <4>last sysfs file: /sys/devices/virtual/net/vetha5mnvF/flags > <4>CPU 70 > <4>Modules linked in: veth autofs4 sunrpc cpufreq_ondemand > acpi_cpufreq freq_table mperf bridge stp llc ipt_REJECT > nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT > nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter > ip6_tables ipv6 microcode mlx4_core igb dca ptp pps_core i2c_i801 > i2c_core iTCO_wdt iTCO_vendor_support sg i7core_edac edac_core ext4 > mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic > ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last > unloaded: scsi_wait_scan] <4> > <4>Pid: 0, comm: swapper Not tainted 2.6.32-358.6.2.el6.x86_64 #1 > Supermicro X8OBN/X8OBN > <4>RIP: 0010:[<ffffffff81167354>] [<ffffffff81167354>] > cache_alloc_refill+0x1e4/0x240 > <4>RSP: 0018:ffff8800282c3ad0 EFLAGS: 00010096 > <4>RAX: 000000000000003c RBX: ffff8940274f0140 RCX: 00000000ffffffcc > <4>RDX: 000000000000003c RSI: 0000000000000000 RDI: ffff884026fa7800 > <4>RBP: ffff8800282c3b30 R08: 0000000000000000 R09: 0000000000000000 > <4>R10: 0000000000000000 R11: 0000000000000000 R12: ffff884026fa7800 > <4>R13: ffff884026fa0440 R14: 000000000000003c R15: ffff88402455c000 > <4>FS: 0000000000000000(0000) GS:ffff8800282c0000(0000) > knlGS:0000000000000000 > <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > <4>CR2: 00007f5599065000 CR3: 0000000001a85000 CR4: 00000000000007e0 > <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > <4>Process swapper (pid: 0, threadinfo ffff88c027c06000, task > ffff8880283d1500) > <4>Stack: > <4> ffff8800282c3b10 000000048144caec ffff884026fa0480 > 000412201096c080 <4><d> ffff884026fa0460 ffff884026fa0450 > ffff8940259fa6e8 0000000000000000 <4><d> 0000000000000020 > ffff8940274f0140 0000000000000020 0000000000000246 <4>Call Trace: > <4> <IRQ> > <4> [<ffffffff8116840f>] kmem_cache_alloc+0x15f/0x190 <4> > [<ffffffffa02726e0>] ? __br_forward+0x0/0xd0 [bridge] <4> > [<ffffffff8143f15f>] skb_clone+0x6f/0xb0 <4> [<ffffffffa02726e0>] ? > __br_forward+0x0/0xd0 [bridge] <4> [<ffffffffa0272320>] > deliver_clone+0x30/0x60 [bridge] <4> [<ffffffffa0272549>] > br_flood+0x79/0xd0 [bridge] <4> [<ffffffffa02725b5>] > br_flood_forward+0x15/0x20 [bridge] <4> [<ffffffffa02736ee>] > br_handle_frame_finish+0x27e/0x2a0 [bridge] <4> [<ffffffffa02738ba>] > br_handle_frame+0x1aa/0x250 [bridge] <4> [<ffffffff81448599>] > __netif_receive_skb+0x529/0x750 <4> [<ffffffff8143da41>] ? > __alloc_skb+0x81/0x190 <4> [<ffffffff8144a8f8>] > netif_receive_skb+0x58/0x60 <4> [<ffffffff8143da41>] ? > __alloc_skb+0x81/0x190 <4> [<ffffffff8144a8f8>] > netif_receive_skb+0x58/0x60 <4> [<ffffffff8144aa00>] > napi_skb_finish+0x50/0x70 <4> [<ffffffff8144cfa9>] > napi_gro_receive+0x39/0x50 <4> [<ffffffffa015045c>] > igb_poll+0x7ec/0xc70 [igb] <4> [<ffffffff81033ef7>] ? > native_apic_msr_write+0x37/0x40 <4> [<ffffffff8144851b>] ? > __netif_receive_skb+0x4ab/0x750 <4> [<ffffffff810a7b05>] ? > tick_dev_program_event+0x65/0xc0 <4> [<ffffffff81012bb9>] ? > read_tsc+0x9/0x20 <4> [<ffffffff8144d0c3>] > net_rx_action+0x103/0x2f0 <4> [<ffffffff81076fb1>] > __do_softirq+0xc1/0x1e0 <4> [<ffffffff810e1720>] ? > handle_IRQ_event+0x60/0x170 <4> [<ffffffff8100c1cc>] > call_softirq+0x1c/0x30 <4> [<ffffffff8100de05>] > do_softirq+0x65/0xa0 <4> [<ffffffff81076d95>] irq_exit+0x85/0x90 <4> > [<ffffffff81517145>] do_IRQ+0x75/0xf0 <4> [<ffffffff8100b9d3>] > ret_from_intr+0x0/0x11 <4> <EOI> <4> [<ffffffff812d39ae>] ? > intel_idle+0xde/0x170 <4> [<ffffffff812d3991>] ? intel_idle+0xc1/0x170 > <4> [<ffffffff81415277>] cpuidle_idle_call+0xa7/0x140 <4> > [<ffffffff81009fc6>] > cpu_idle+0xb6/0x110 <4> [<ffffffff81506fcc>] > start_secondary+0x2ac/0x2ef > <4>Code: 89 ff e8 70 1c 12 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 > 00 00 > 00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b > 66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31 <1>RIP > [<ffffffff81167354>] cache_alloc_refill+0x1e4/0x240 <4> RSP > <ffff8800282c3ad0> > > Sorry for my poor English :) > > Sincere > -yao > > -----邮件原件----- > 发件人: Serge Hallyn [mailto:serge.hal...@ubuntu.com] > 发送时间: 2013年6月7日 4:16 > 收件人: yaozhicheng > 抄送: lxc-users@lists.sourceforge.net > 主题: Re: [Lxc-users] lxc scalability problem > > Quoting yaozhicheng (yaozhich...@emails.bjut.edu.cn): > > Hi all > > > > My OS get crashed when I start more then 20 containers. > > Seems like an selinux labeling related problem. If you boot without > selinux enabled do you still have this problem? (I'm not suggesting > that as a workaround - only to verify that the problem is with selinux > using too much memory for the dev tmpfs labeling, so ppl know where to > look) > > -serge > > > ---------------------------------------------------------------------- > -------- This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Lxc-users mailing list > Lxc-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/lxc-users ------------------------------------------------------------------------------ This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev _______________________________________________ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
