On Tue, Jul 27, 2010 at 10:47:59PM -0700, Joe Eykholt wrote: > Hi Tristan, > > One thing you could do is figure out how much is leaking > (if any) per I/O. Look at /proc/slabinfo (assuming you're > using the slab allocator ... if not you could reconfigure to > do that). Copy /proc/slabinfo somewhere, run 1000 or so > I/Os and then diff to see which slabs are changing by how much. > You could also look at vmstats before and after the I/O. > I used something like that to find a leak in another area > recently.
/proc/slabinfo is a really easy first check to rule out skb leaks, because the skb structures are allocated out of a dedicated slab (2 actually, depending on if they are allocated with room for fast cloning or not). On my FCoE test system if I grep for skb in /proc/slabinfo I see skbuff_fclone_cache 7 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 10255 10650 256 15 1 : tunables 120 60 8 : slabdata 710 710 0 The first # is the number of active objects, or outstanding allocations. The 7 skbuff_fclone_cache allocations are probably all TCP packets being held until they've been acked (I'm connected in via SSH). The FCoE stack allocates fast clone frames for transmit also, so it doesn't look like any of those are leaking. The 10k skbuff_head_cache allocations are not unreasonable, they're probably receive buffers allocated by network drivers waiting for data. I've got a 10Gb ixgbe interface with 16 active queues, each with 512 receive buffers, so that's 8k right there. This system also has two other gigabit interfaces active, so 10k sounds about right. Let us know what your slabinfo looks like, so we can hopefully rule out skb leaks. Then we just need to figure out what's actually going on :) - Chris Chris, Joe, Vasu, Thanks for the helpful pointers. I added a destructor function with a counter and the allocated/destructed numbers match. I also checked slabinfo before/during/after "dd" with ~2-3 second intervals. Here's the slabinfo output for "dd if=/dev/zero of=/dev/sde1 bs=4M count=100 oflag=direct": :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 7 8 448 8 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 79099 79230 256 15 1 : tunables 120 60 8 : slabdata 5282 5282 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 7 8 448 8 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 79147 79230 256 15 1 : tunables 120 60 8 : slabdata 5282 5282 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 200 200 448 8 1 : tunables 54 27 8 : slabdata 25 25 146 skbuff_head_cache 79290 79290 256 15 1 : tunables 120 60 8 : slabdata 5286 5286 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 200 200 448 8 1 : tunables 54 27 8 : slabdata 25 25 135 skbuff_head_cache 79305 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 197 208 448 8 1 : tunables 54 27 8 : slabdata 26 26 108 skbuff_head_cache 79303 79335 256 15 1 : tunables 120 60 8 : slabdata 5289 5289 60 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 142 208 448 8 1 : tunables 54 27 8 : slabdata 26 26 91 skbuff_head_cache 79238 79335 256 15 1 : tunables 120 60 8 : slabdata 5289 5289 30 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 57 192 448 8 1 : tunables 54 27 8 : slabdata 22 24 47 skbuff_head_cache 79104 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 53 192 448 8 1 : tunables 54 27 8 : slabdata 22 24 47 skbuff_head_cache 79115 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 3 96 448 8 1 : tunables 54 27 8 : slabdata 3 12 0 skbuff_head_cache 79067 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 1 8 448 8 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 79221 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 7 8 448 8 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 79054 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 Here it is for the same dd command without oflag=direct: :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 4 8 448 8 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 79153 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 216 216 448 8 1 : tunables 54 27 8 : slabdata 27 27 135 skbuff_head_cache 79305 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 205 216 448 8 1 : tunables 54 27 8 : slabdata 27 27 162 skbuff_head_cache 79301 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 204 216 448 8 1 : tunables 54 27 8 : slabdata 27 27 108 skbuff_head_cache 79264 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 60 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 193 216 448 8 1 : tunables 54 27 8 : slabdata 27 27 92 skbuff_head_cache 79215 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 256 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 162 skbuff_head_cache 79281 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 60 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 255 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 162 skbuff_head_cache 79289 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 120 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 200 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 145 skbuff_head_cache 79213 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 30 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 189 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 108 skbuff_head_cache 79272 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 60 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 178 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 81 skbuff_head_cache 79258 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 123 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 64 skbuff_head_cache 79224 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 108 256 448 8 1 : tunables 54 27 8 : slabdata 32 32 64 skbuff_head_cache 79271 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 74 248 448 8 1 : tunables 54 27 8 : slabdata 31 31 18 skbuff_head_cache 79234 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 61 248 448 8 1 : tunables 54 27 8 : slabdata 29 31 18 skbuff_head_cache 79197 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 39 224 448 8 1 : tunables 54 27 8 : slabdata 26 28 9 skbuff_head_cache 79189 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 28 224 448 8 1 : tunables 54 27 8 : slabdata 22 28 9 skbuff_head_cache 79211 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 21 160 448 8 1 : tunables 54 27 8 : slabdata 18 20 4 skbuff_head_cache 79172 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 19 160 448 8 1 : tunables 54 27 8 : slabdata 17 20 4 skbuff_head_cache 79151 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 60 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 15 136 448 8 1 : tunables 54 27 8 : slabdata 15 17 2 skbuff_head_cache 79092 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 30 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 14 136 448 8 1 : tunables 54 27 8 : slabdata 14 17 2 skbuff_head_cache 79083 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 11 104 448 8 1 : tunables 54 27 8 : slabdata 11 13 1 skbuff_head_cache 79131 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 9 104 448 8 1 : tunables 54 27 8 : slabdata 9 13 1 skbuff_head_cache 79112 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0:~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 6 64 448 8 1 : tunables 54 27 8 : slabdata 6 8 0 skbuff_head_cache 79084 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 :~# cat /proc/slabinfo | grep skbuff skbuff_fclone_cache 4 64 448 8 1 : tunables 54 27 8 : slabdata 4 8 0 skbuff_head_cache 79075 79305 256 15 1 : tunables 120 60 8 : slabdata 5287 5287 0 It looks like all is well with respect to these stats. There is still a considerable rise in buffers without direct I/O (28,732 to 440,080 and back down to 34,044). With direct I/O, the rise is small but there are still additional buffers when dd is finished (34,044 to 36,382). Perhaps it is expected to have unreleased buffers after the operation and they could be coming from various other places outside of libfc/libfcoe/fcoe. It appears that things go haywire when a large number of buffers is required: dd if=/dev/zero of=/dev/sde1 bs=4M count=1000 (~4GB, goes through without any noticeable problems) dd if=/dev/zero of=/dev/sde1 bs=512 count=10000000 (~5GB, buffer count goes over 5 million, free memory goes from ~6B to ~500MB) dd if=/dev/zero of=/dev/sde1 bs=512 count=100000000 (~50GB, Reducing can_queue to 512, then increased can_queue to 1024, then Buffer I/O errors on device sde1 for chunks of blocks) My original mkfs.ext3 command was on a 2TB partition. Thanks, Tristan _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
