I suspect that the new crossbow architecture might impose additional
resource constraints on vlans -- it makes sense that it would create
additional rings and worker threads for each vlan. Anyone from
crossbow-discuss want to comment?
- Garrett
Saurabh Misra wrote:
>
> I'm somewhat sure that there is no leak. Anyway I managed to get crash
> dump (had to reinstall OS with larger dump device).
>
> Findleak output :-
>
> findleaks: thread ffffff014c477740's stack swapped out; false
> positives possible
> findleaks: thread ffffff014c4768e0's stack swapped out; false
> positives possible
> findleaks: thread ffffff014c47e1a0's stack swapped out; false
> positives possible
> CACHE LEAKED BUFCTL CALLER
> ffffff014622b2e0 1 ffffff01506a8510 impl_acc_hdl_alloc+0x34
> ffffff0146226b20 1 ffffff0150fc6298 impl_acc_hdl_alloc+0x4a
> ffffff014622a020 1 ffffff014eff4e58 impl_acc_hdl_alloc+0x64
> ffffff014ed55020 1 ffffff01510db9c8 rootnex_coredma_allochdl+0x5c
> ffffff014622ab20 1 ffffff015098be70 uhci_polled_create_tw+0x2a
> ------------------------------------------------------------------------
> Total 5 buffers, 3232 bytes
> bash-3.2#
> which does not reveal anything...Another findleak output when freemem
> was 0x1
>
> CACHE LEAKED BUFCTL CALLER
> ffffff01462342e0 1 ffffff01677fb1e8 allocb+0x64
> ffffff014eaeeb20 1 ffffff016a595658 cralloc_flags+0x21
> ffffff0146232b20 1 ffffff01697da048 dblk_constructor+0x3b
> ffffff014622b2e0 1 ffffff01506fe660 impl_acc_hdl_alloc+0x34
> ffffff0146226b20 1 ffffff0150a73050 impl_acc_hdl_alloc+0x4a
> ffffff014622a020 1 ffffff014f02ba48 impl_acc_hdl_alloc+0x64
> ffffff014ed87020 1 ffffff01510b1aa8 rootnex_coredma_allochdl+0x5c
> ffffff014622ab20 1 ffffff0150a2d008 uhci_polled_create_tw+0x2a
> ------------------------------------------------------------------------
> Total 8 buffers, 3676 bytes
>
> bash-3.2# grep "THREAD:" t.1 | wc -l
> 24666 // Total number of threads
>
> bash-3.2# grep "THREAD: mac_srs_worker()" t.1 | wc -l
> 8193 // mac_src_worker threads
> bash-3.2#
>
> PC: _resume_from_idle+0xf1 THREAD: mac_srs_worker()
> stack pointer for thread ffffff0004bf0c60: ffffff0004bf0b80
> [ ffffff0004bf0b80 _resume_from_idle+0xf1() ]
> swtch+0x147()
> cv_wait+0x61()
> mac_srs_worker+0x1cb()
> thread_start+8()
>
>
> bash-3.2# grep "THREAD: mac_soft_ring_worker()" t.1 | wc -l
> 12291 // Total number of mac_soft_ring_worker() threads.
>
> PC: _resume_from_idle+0xf1 THREAD: mac_soft_ring_worker()
> stack pointer for thread ffffff0005e47c60: ffffff0005e47b80
> [ ffffff0005e47b80 _resume_from_idle+0xf1() ]
> swtch+0x147()
> cv_wait+0x61()
> mac_soft_ring_worker+0xb0()
> thread_start+8()
>
> I don't understand why MAC layer has created so many threads. The vlan
> test did create 4094 vlan's.
>
> From ::kmausers output, which reports current medium and large users
> of the kmem allocator
>
> bash-3.2# more kma.1
> 83111936 bytes for 20291 allocations with data size 4096:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> vmem_alloc+0x1bc
> segkmem_xalloc+0x94
> segkmem_alloc_vn+0xcd
> segkmem_alloc+0x24
> vmem_xalloc+0x547
> vmem_alloc+0x161
> kmem_slab_create+0x81
> kmem_slab_alloc+0x5b
> kmem_cache_alloc+0x130
> allocb+0x64
> udp_input+0xeee
> ip_fanout_udp_conn+0x2b2
> 33538048 bytes for 4094 allocations with data size 8192:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> kmem_zalloc+0x6a
> mac_flow_tab_create+0x65
> mac_flow_l2tab_create+0x31
> mac_register+0x56f
> vnic_dev_create+0x42b
> vnic_ioc_create+0x157
> drv_ioctl+0x137
> cdev_ioctl+0x45
> spec_ioctl+0x83
> fop_ioctl+0x7b
> ioctl+0x18e
> 28307456 bytes for 6911 allocations with data size 4096:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> vmem_alloc+0x1bc
> segkmem_xalloc+0x94
> segkmem_alloc_vn+0xcd
> segkmem_alloc+0x24
> vmem_xalloc+0x547
> vmem_alloc+0x161
> kmem_slab_create+0x81
> kmem_slab_alloc+0x5b
> kmem_cache_alloc+0x130
> dblk_constructor+0x3b
> kmem_cache_alloc_debug+0x249
> kmem_cache_alloc+0x164
> 27232000 bytes for 106375 allocations with data size 256:
> kmem_cache_alloc_debug+0x283
> kmem_cache_alloc+0x164
> allocb+0x64
> udp_input+0xeee
> ip_fanout_udp_conn+0x2b2
> ip_fanout_udp+0xc72
> ip_wput_local+0x6ce
> ip_multicast_loopback+0x2cb
> udp_xmit+0x4a9
> udp_send_data+0x3b3
> udp_output_v4+0x9c6
> udp_send_not_connected+0xeb
> udp_send+0x246
> so_sendmsg+0x1c7
> socket_sendmsg+0x61
> 27172864 bytes for 6634 allocations with data size 4096:
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> vmem_alloc+0x1bc
> segkmem_xalloc+0x94
> segkmem_alloc_vn+0xcd
> segkmem_alloc+0x24
> vmem_xalloc+0x547
> vmem_alloc+0x161
> kmem_slab_create+0x81
> kmem_slab_alloc+0x5b
> kmem_cache_alloc+0x130
> allocb+0x64
> allocb_tmpl+0x24
> copyb+0x77
> [.]
> 324608 bytes for 128 allocations with data size 2536: // bfe comes
> for 128 buffer allocation
> kmem_slab_alloc_impl+0x116
> kmem_slab_alloc+0xa1
> kmem_cache_alloc+0x130
> rootnex_coredma_allochdl+0x84
> rootnex_dma_allochdl+0x7d
> ddi_dma_allochdl+0x35
> ddi_dma_alloc_handle+0xb8
> bfe_ring_buf_alloc+0x4b
> bfe_ring_desc_alloc+0x149
> bfe_rings_alloc+0xa6
> bfe_attach+0x2d0
> devi_attach+0x80
> attach_node+0x95
> i_ndi_config_node+0xa5
> i_ddi_attachchild+0x40
> [.]
>
> Failures were seen on following caches :
>
> > ::kmastat
> cache buf buf buf memory alloc alloc
> name size in use total in use succeed fail
> ------------------------- ------ ------ ------ ---------- --------- -----
> streams_mblk 64 683051 683130 66621440B 39257487 993
> streams_dblk_16 128 1469 1533 299008B 1324225 0
> streams_dblk_80 192 61535 61568 15761408B 7558613 75
> streams_dblk_144 256 279841 279852 95522816B 4124673 1594
> streams_dblk_208 320 592 600 245760B 279677 0
> streams_dblk_272 384 1032 1044 475136B 2666792 71
>
> And all threads stuck in page throttle because of no memory.
>
> Is there a way to control creating number of vlan's in NICDRV?
>
> I'm thinking of running NICDRV on e1000g and that can tell whether
> it's driver's fault or something in MAC layer.
>
> Looks like findleaks does not do good job in finding leaks related to
> allocb or mblk.
>
> cheers,
>
