Re: Cat on (RAM) steroids
Well, the real reason is that the buffers in the kernel are allocated as DMA memory. So it's not just a matter of getting hold of a lot of memory, but a lot of memory which is continuous in its physical space. Or more precisely, a lot of continuous buffers which are fairly large. One could, of course, copy the data in the smaller DMA buffers to larger buffers by virtue of some bottom-half interrupt handler (i.e. a tasklet), but I think that's a safe way to stop your code from making it to the kernel tree. Besides, I have a faint memory of a limitation on the total RAM allocatable inside the kernel. Was it 512MB? Has this limitation vanished? Eli Oron Peled wrote: I fail to see why the kernel driver would be more limited in RAM allocation than the utility you want? After all, RAM is RAM if you have enough for the application to use (you asked for non-pageable memory), than why can't the kernel driver allocate it just the same and be done with it? Bye, -- Web: http://www.billauer.co.il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cat on (RAM) steroids
On Sun, Feb 5, 2012 at 10:41 AM, Eli Billauer e...@billauer.co.il wrote: Besides, I have a faint memory of a limitation on the total RAM allocatable inside the kernel. Was it 512MB? Has this limitation vanished? You can reserve more memory for kernel side processing using echo size_in_kb /proc/sys/vm/min_free_kbytes Keep in mind that raising this value will have adverse effect on user-mode applications. - Gilboa ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Cat on (RAM) steroids
Hi all, I need a simple command-line program, which works as a plain FIFO stream buffer with a huge RAM. Something I can do: $ fatcat -b 256M /dev/datasource | ./my_shaky_data_sink The idea is that fatcat reads data whenever available and stores it to non-swappable RAM. It then pushes the data to stdout. So it's just like good old cat, only with a potentially large tummy. Rationale: The (kernel) device /dev/datasource has limited RAM it can allocate in kernel space. If data arrives at a high constant rate (say, 100 MB/sec), the kernel buffers (4 MB total?) overflow rather quickly if not constantly drained. A real-life data sink may write the data to a disk with sufficient *average* rate, but with breaks every now and then to mind its own business. So if data is loaded into a huge RAM array (what is 256 MB these days?) by a high-priority thread (dare I say real-time?), there's a good chance the kernel buffers are emptied quick enough. It's rather straightforward to write a program doing this with POSIX threads, but why bother if it's already available? After all, I can't be the first one in the world with this need. My own search lead me at best to cstream, which doesn't look like it's tailored for solving realtime issues. Ideas, anyone? Eli -- Web: http://www.billauer.co.il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cat on (RAM) steroids
On Sat, Feb 4, 2012 at 4:27 PM, Eli Billauer e...@billauer.co.il wrote: Hi all, I need a simple command-line program, which works as a plain FIFO stream buffer with a huge RAM. Something I can do: $ fatcat -b 256M /dev/datasource | ./my_shaky_data_sink The idea is that fatcat reads data whenever available and stores it to non-swappable RAM. It then pushes the data to stdout. So it's just like good old cat, only with a potentially large tummy. I've used pv or buffer for such purposes. Baruch ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cat on (RAM) steroids
Thanks for that one. yum install buffer. How simple. So I tried it out: $ dd if=/dev/zero bs=1M count=256 /dev/null 256+0 records in 256+0 records out 268435456 bytes (268 MB) copied, 0.0258608 s, 10.4 GB/s Original throughput is 10.4 GB/sec. Looks not so bad. $ dd if=/dev/zero bs=1M count=256 | cat /dev/null 256+0 records in 256+0 records out 268435456 bytes (268 MB) copied, 0.199799 s, 1.3 GB/s The regular cat is somewhat slower, then. Now let's try buffer: $ dd if=/dev/zero bs=1M count=256 | buffer -m 256m /dev/null Cannot handle that many blocks, aborting! Hmmm... The default buffer size is too small, so asking for 256MB didn't work well. Let's ask for 128kByte buffers: $ dd if=/dev/zero bs=1M count=256 | buffer -s 128k -m 256m /dev/null 256+0 records in 256+0 records out 268435456 bytes (268 MB) copied, 0.379354 s, 708 MB/s On one hand, it's not all that impressive that the rate went down. But it's still way above the 100 MB/sec needed. On the other hand, going $ dd if=/dev/zero bs=1M count=256 | buffer -s 128k -m 256m | hexdump -C /dev/null gives exactly the same result, except that command prompt returns after a second or so, which is the time it took for hexdump to consume the data. So I suppose we have a winner. Thanks again. Eli Baruch Even wrote: On Sat, Feb 4, 2012 at 4:27 PM, Eli Billauer e...@billauer.co.il mailto:e...@billauer.co.il wrote: Hi all, I need a simple command-line program, which works as a plain FIFO stream buffer with a huge RAM. Something I can do: $ fatcat -b 256M /dev/datasource | ./my_shaky_data_sink The idea is that fatcat reads data whenever available and stores it to non-swappable RAM. It then pushes the data to stdout. So it's just like good old cat, only with a potentially large tummy. I've used pv or buffer for such purposes. Baruch -- Web: http://www.billauer.co.il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cat on (RAM) steroids
On Saturday, 4 בFebruary 2012 16:27:27 Eli Billauer wrote: ... Rationale: The (kernel) device /dev/datasource has limited RAM it can allocate in kernel space. ... So if data is loaded into a huge RAM array (what is 256 MB these days?) I fail to see why the kernel driver would be more limited in RAM allocation than the utility you want? After all, RAM is RAM if you have enough for the application to use (you asked for non-pageable memory), than why can't the kernel driver allocate it just the same and be done with it? Bye, -- Oron Peled Voice: +972-4-8228492 o...@actcom.co.il http://users.actcom.co.il/~oron Linux: Because a PC is a terrible thing to waste. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
