On Mon, Mar 2, 2009 at 1:16 PM, guy keren <c...@actcom.co.il> wrote: > > if you don't care about the speed of copying the data and of slowing down > the disk media consderably during this copying, try to change the copy > program, so it will use the O_DIRECT flag when opening the file (and then > you'll need to make sure the buffers you pass to write() are aligned to disk > sector size, as well as being in disk sector size multiples). this way, > you'll bypass the buffer cache altogether. > > maybe someone else will come up with a better response. >
choo, nice to chat with you again. Still have an appetite to red heads :) ? Thank you for your answer. While O_DIRECT is a valid way and probably will solve the problem of crashing, I would not want to patch wget/sftp/whatever. The list is long, I have better things to do mean while, instead of burning in hell myself re-patching all the user space with each new wget/OpenSSH/whatever release. I still hope to find some magic kernel tuning variable which would say: don't grab more than half/quoter of physical memory. This way the main application would not burn in hell, just because there is transfer of data to flash storage progressing in the background. ~baum > --guy > > Alexander Indenbaum wrote: >> >> Guys/Girls/Dudes, >> >> I'm working with embedded Linux box powered by 2.4 kernel ( I know :) >> we are in process of switching to 2.6 ) with 1G physical memory, no >> swap defined and some slow flash storage mounted. >> >> We noticed that if we copy large media file ( 1.4G ) to flash storage >> filesystem, using wget/sftp/whatever kernel grabs almost all the >> available physical memory and buffers are not released even after >> transfer is completed and sync; sync; sync is issued :). It >> constitutes a problem - applications have no enough memory to run - >> crash and burn in hell. See bellow memory consumption numbers (top >> output): >> >> Before test start: Mem: 97296K used, 807096K free, 0K shrd, 1732K >> buff, 54712K cached >> During/After test: Mem: 897908K used, 6484K free, 0K shrd, 2872K buff, >> 835088K cached >> >> It worth mentioning that probably we see this issue since network IO >> rate is higher than storage IO rate. I’m not sure which kernel >> subsystem is memory grabber: VFS buffer cache, storage driver maybe >> something else? Bottom line is we don’t want to let the kernel to be >> so memory hungry and it’s desirable to set some reasonable limit on >> memory usage and thus work around this problem. >> >> What are your recommendations? >> >> ~baum >> >> _______________________________________________ >> Linux-il mailing list >> Linux-il@cs.huji.ac.il >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il _______________________________________________ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
