One more note... I added printk to fc_frame_alloc and fc_frame_free to show the number of frames allocated vs freed.
Without Direct I/O: Number of allocated frames is much greater than number freed. With Direct I/O: They are identical When going through the code, it seemed as though not all frames are freed via fc_frame_free. Thanks, Tristan -----Original Message----- From: Joe Eykholt (jeykholt) Sent: Tuesday, July 27, 2010 4:44 PM To: Tristan Cheever (tcheever) Cc: [email protected]; Walter Song (wasong) Subject: Re: [Open-FCoE] Huge memory usage with mkfs.ext3 over FCoE On 7/27/10 3:46 PM, Tristan Cheever (tcheever) wrote: > [CC'd John Fastabend at Intel] > > > > Hi, > > > > I'm not sure what the typical response time is on this mailing list but > I have not received a response yet. Assistance would be much > appreciated as this is a high priority issue for our product. > > > > We have been working with John Fastabend on various DCB/FCoE hurdles and > he has now referred me to this mailer for help. > > > > Our most recent debugging shows that the millions of buffers allocated > by the FC/FCoE drivers are roughly equivalent to the number of FC frames > allocated. It appears that not all of the frame/skbuff copies get > freed. Is the memory really tied up or could it just be in use by the page cache? It seems reasonable to me that the pages recently written by mkfs.ext3 would remain in the cache (clean) for awhile, but would be reusable. However, if you are just looking at the free page count from vmstat, one might wonder if there was a memory leak. Does a write using 'dd' and oflag=direct also result in lots of memory tied up? I don't see problems like this in doing direct I/O and I don't think doing cached I/O would cause a leak. I've been using 2.6.33.5 and other recent kernels. Cheers, Joe > Thanks, > > Tristan > > > > From: Tristan Cheever (tcheever) > Sent: Friday, July 23, 2010 12:13 PM > To: '[email protected]' > Cc: Walter Song (wasong) > Subject: Huge memory usage with mkfs.ext3 over FCoE > > > > Hi, > > > > I have been developing an FCoE solution and have come across an > interesting problem. When running mkfs.ext3 on a target connected via > FCoE, my machine goes from 6GB of available RAM to 70MB in less than a > minute. When checking the output of "free" from the bash prompt, I see > between 1 and 2 million buffers allocated. None of the fcoe process > show user space memory usage in "top". The same mkfs.ext3 utility does > not cause this problem when using a target connected via an external SAS > wide port. > > > > Is this a known issue or has anyone experienced it? > > > > Here is the connection "diagram": > > > > Host (my machine)<--FCoE--> N5K Switch<--FC--> FC Storage > > > > Here are details of my machine: > > > > * Montavista Linux 5.0 > > * 2.6.33 kernel (from kernel.org, with a few minor patches from > us) > > * ixgbe 2.0.44-k2 (from the 2.6.33 kernel, our patch attached) > > * dcbd 0.9.21 > > * fcoe-utils 2.6.33 > > > > Please let me know what other information you may need. > > > > Thanks, > > Tristan > > > > > > _______________________________________________ > devel mailing list > [email protected] > http://www.open-fcoe.org/mailman/listinfo/devel _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
