On Friday 13 November 2009 17:18:17 Edmund R. MacKenty wrote:
> On Friday 13 November 2009 17:14, McKown, John wrote:
> >I think you're right. He was worried that instead of his program just
> > reading the file(s), the I/O would be: (1) "cat" reading the files from
> > disk; (2)writing the contents to the pipe and (3) his program reading the
> > pipe. Or about 3x the I/O. But pipes are not hardened to disk (ignoring
> > paging?), so it is more like a VIO dataset in z/OS (VIO datasets are
> > written to memory only).
>
> I was curious to see what the overhad is, so I made five 1GB files:
>
> # for f in one two three four five; do \
> dd if=/dev/urandom of=$f bs=1M count=1024; done
>
> and then tried running a simple tool that would just read through them all
> to get a base time:
>
> # time od one two three four five > /dev/null
> real 21m13.009s
> user 19m52.603s
> sys 0m11.557s
>
> Then I tried the named pipe approach:
>
> # mknod pipe p
> # time (cat one two three four five > pipe & od -c pipe > /dev/null)
> real 58m19.154s
> user 56m56.490s
> sys 0m20.361s
>
> Now this is just on a laptop, and a very crude measurement, but it sure
> looks like there's a bit of overhead in them thar named pipes and cat!
> - MacK.
If I remember correctly, the originator of this thread was attempting to
process LOTS of LITTLE files, so it might be interesting to try it that
way...
==========================
19:03:14 turr...@pinto$ concatTest
Making 200 test files...
10+0 records in
10+0 records out
10240 bytes (10 kB) copied, 0.00760478 s, 1.3 MB/s
:
10+0 records in
10+0 records out
10240 bytes (10 kB) copied, 0.00420962 s, 2.4 MB/s
Processing by simple catenation...
real 0m0.498s
user 0m0.388s
sys 0m0.028s
Processing by named pipe...
real 0m1.571s
user 0m1.436s
sys 0m0.028s
Removing test files...
Done.
==============================
Seems like there is quite a bit of overhead with a pipe...
==============================
19:03:22 turr...@pinto$ cat ~/bin/concatTest
#!/bin/bash
echo
echo "Making 200 test files..."
for f in f{1..200}
do
dd if=/dev/urandom of=${f} bs=1K count=10
done
echo
echo "Processing by simple catenation..."
time od f{1..200} >/dev/null
echo
echo "Processing by named pipe..."
mknod pipe p
time (cat f{1..200} >pipe & od -c pipe >/dev/null)
echo
echo "Removing test files..."
rm f{1..200}
echo
echo "Done."
echo
================================
Leslie
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
