Hi Joe, Thank you very much for responding quickly. I see there are many threads to address.
Cache is definitely a possibility. I attempted setting /proc/sys/vm/drop_caches to 1 but it did not help. I am not aware of other ways to force the kernel to flush caches right away. In my testing I was checking the MemFree, Buffers and Cached levels from /proc/meminfo. The memory drops, anywhere from 2-6GB (starting with ~6GB available). The buffers go into the millions and the cached level stays about the same. When running dd without direct I/O, the buffers increase dramatically as well (>10,000/sec with initial burst of over 1 million). I suppose this is expected given the mkfs test results. The buffer count increases dramatically slower (<100/sec) when using dd with direct I/O. So this suggests a caching issue? Why would I not see an increase in the cache level from /proc/meminfo? Thanks, Tristan -----Original Message----- From: Joe Eykholt (jeykholt) Sent: Tuesday, July 27, 2010 4:44 PM To: Tristan Cheever (tcheever) Cc: [email protected]; Walter Song (wasong) Subject: Re: [Open-FCoE] Huge memory usage with mkfs.ext3 over FCoE On 7/27/10 3:46 PM, Tristan Cheever (tcheever) wrote: > [CC'd John Fastabend at Intel] > > > > Hi, > > > > I'm not sure what the typical response time is on this mailing list but > I have not received a response yet. Assistance would be much > appreciated as this is a high priority issue for our product. > > > > We have been working with John Fastabend on various DCB/FCoE hurdles and > he has now referred me to this mailer for help. > > > > Our most recent debugging shows that the millions of buffers allocated > by the FC/FCoE drivers are roughly equivalent to the number of FC frames > allocated. It appears that not all of the frame/skbuff copies get > freed. Is the memory really tied up or could it just be in use by the page cache? It seems reasonable to me that the pages recently written by mkfs.ext3 would remain in the cache (clean) for awhile, but would be reusable. However, if you are just looking at the free page count from vmstat, one might wonder if there was a memory leak. Does a write using 'dd' and oflag=direct also result in lots of memory tied up? I don't see problems like this in doing direct I/O and I don't think doing cached I/O would cause a leak. I've been using 2.6.33.5 and other recent kernels. Cheers, Joe > Thanks, > > Tristan > > > > From: Tristan Cheever (tcheever) > Sent: Friday, July 23, 2010 12:13 PM > To: '[email protected]' > Cc: Walter Song (wasong) > Subject: Huge memory usage with mkfs.ext3 over FCoE > > > > Hi, > > > > I have been developing an FCoE solution and have come across an > interesting problem. When running mkfs.ext3 on a target connected via > FCoE, my machine goes from 6GB of available RAM to 70MB in less than a > minute. When checking the output of "free" from the bash prompt, I see > between 1 and 2 million buffers allocated. None of the fcoe process > show user space memory usage in "top". The same mkfs.ext3 utility does > not cause this problem when using a target connected via an external SAS > wide port. > > > > Is this a known issue or has anyone experienced it? > > > > Here is the connection "diagram": > > > > Host (my machine)<--FCoE--> N5K Switch<--FC--> FC Storage > > > > Here are details of my machine: > > > > * Montavista Linux 5.0 > > * 2.6.33 kernel (from kernel.org, with a few minor patches from > us) > > * ixgbe 2.0.44-k2 (from the 2.6.33 kernel, our patch attached) > > * dcbd 0.9.21 > > * fcoe-utils 2.6.33 > > > > Please let me know what other information you may need. > > > > Thanks, > > Tristan > > > > > > _______________________________________________ > devel mailing list > [email protected] > http://www.open-fcoe.org/mailman/listinfo/devel _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
