Re: Poor read() performance, and I can't profile it
On Wednesday 11 June 2008, Chuck Swiger wrote: If your data files are small enough to fit into 2GB of address space, try using mmap() and then treat the file(s) as an array of records or memoblocks or whatever, and let the VM system deal with paging in the parts of the file you need. Otherwise, don't fread() 1 record at a time, read in at least a (VM page / sizeof(record)) number of records at a time into a bigger buffer, and then process that in RAM rather than trying to fseek in little increments. During a marathon session last night, I did just that. I changed the sequential reads in the outer file to fread many records at a time. Then I switched to mmap() for the random-access file. The results were much better, with good CPU usage and only 3 times the wall clock runtime: [EMAIL PROTECTED] date; time /tmp/cdbf /tmp/invoice.dbf /dev/null; date Thu Jun 12 13:56:49 CDT 2008 /tmp/cdbf /tmp/invoice.dbf /dev/null 29.00s user 11.16s system 56% cpu 1:11.03 total Thu Jun 12 13:58:00 CDT 2008 [EMAIL PROTECTED] date; time /tmp/cdbf ~pgsql/data/frodumps/xbase/invoice.dbf invid ln /dev/null; date Thu Jun 12 14:10:57 CDT 2008 /tmp/cdbf ~pgsql/data/frodumps/xbase/invoice.dbf invid ln /dev/null 38.14s user 6.21s system 23% cpu 3:05.13 total Thu Jun 12 14:14:02 CDT 2008 Also, if you're malloc'ing and freeing buf memohead with every iteration of the loop, you're just thrashing the malloc system; instead, allocate your buffers once before the loop, and reuse them (zeroize or copy new data over the previous results) instead. Also done. I'd gotten some technical advice from Slashdot (which speaks volumes for my clueless, granted) that made it sound like a good idea. I changed almost all the mallocs into static buffers. I'm still offering that shell account to anyone who wants to take a peek. :-) -- Kirk Strauser signature.asc Description: This is a digitally signed message part.
Re: Poor read() performance, and I can't profile it
On Thursday 05 June 2008, Kirk Strauser wrote:
I was testing the same software on my desktop PC when I noticed that it
ran *much* faster, and found that it was spending only about 1% as much
time in the kernel on Linux as it was on FreeBSD.
I'm almost ready to give up on this. I've gone as far as completely rewriting
the
original C++ program into straightforward C, and still the performance is
terrible on
FreeBSD versus Linux.
On Linux:
$ time ./cdbf /tmp/invoice.dbf /dev/null
./cdbf /tmp/invoice.dbf /dev/null 42.65s user 20.09s system 71% cpu 1:28.15
total
On FreeBSD:
Also note that on the FreeBSD machine, I have enough RAM that to buffer the
entire
file, and in practice gstat shows that the drives are idle for subsequent runs
after
the first one.
Right now my code looks a lot like:
for(recordnum = 0; recordnum recordcount; recordnum++) {
buf = malloc(recordlength);
fread(buf, recordlength, 1, dbffile);
/* Do stuff with buf */
memoblock = getmemoblock(buf);
/* Skip to the requested block if we're not already there */
if(memoblock != currentmemofileblock) {
currentmemofileblock = memoblock;
fseek(memofile, currentmemofileblock * memoblocksize, SEEK_SET);
}
memohead = malloc(memoblocksize);
fread(memohead, memoblocksize, 1, memofile);
currentmemofileblock++;
/* Do stuff with memohead */
free(memohead);
free(buf);
}
...where recordlength == 13 in this one case. Given that the whole file is
buffered in
RAM, the small reads shouldn't make a difference, should they? I've played
with
setvbuf() and it shaves off a few percent of runtime, but nothing to write home
about.
Now, memofile gets quite a lot of seeks. Again, that shouldn't make too much
of a
difference if it's already buffered in RAM, should it? setvbuf() on that file
that
gets lots of random access actually made performance worse.
What else can I do to make my code run as well on FreeBSD as it does on a much
wimpier
Linux machine? I'm almost to the point of throwing in the towel and making a
Linux
server to do nothing more than run this one program if I can't FreeBSD's
performance
more on parity, and I honestly never thought I'd be considering that.
I'll gladly give shell access with my code and sample data files if anyone is
interested in testing it.
--
Kirk Strauser
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
On Wednesday 11 June 2008, Kirk Strauser wrote: On Linux: $ time ./cdbf /tmp/invoice.dbf /dev/null ./cdbf /tmp/invoice.dbf /dev/null 42.65s user 20.09s system 71% cpu 1:28.15 total On FreeBSD: Oops! I left that out: $ time /tmp/cdbf /var/tmp/invoice.dbf /dev/null /tmp/cdbf /var/tmp/invoice.dbf /dev/null 59.15s user 11.93s system 36% cpu 3:14.53 total Again, Linux is on a boring Dell workstation, FreeBSD is on a far faster Dell server 15K RPM SCSI drives (even if they don't come into play once the data files are buffered). -- Kirk Strauser ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
On Jun 11, 2008, at 12:42 PM, Kirk Strauser wrote:
I'm almost ready to give up on this. I've gone as far as completely
rewriting the
original C++ program into straightforward C, and still the
performance is terrible on
FreeBSD versus Linux.
On Linux, GNU libc buffers file data much more extensively than
FreeBSD's libc does. It means that doing things like reading a dozen
bytes or so at a time is not intolerably slow on the former system,
but that doesn't mean that it's a great idea either.
If your data files are small enough to fit into 2GB of address space,
try using mmap() and then treat the file(s) as an array of records or
memoblocks or whatever, and let the VM system deal with paging in the
parts of the file you need. Otherwise, don't fread() 1 record at a
time, read in at least a (VM page / sizeof(record)) number of records
at a time into a bigger buffer, and then process that in RAM rather
than trying to fseek in little increments.
(This is the opposite of calling setvbuf() to set the I/O buffer to,
say, 13 bytes...)
Also, if you're malloc'ing and freeing buf memohead with every
iteration of the loop, you're just thrashing the malloc system;
instead, allocate your buffers once before the loop, and reuse them
(zeroize or copy new data over the previous results) instead.
Regards,
--
-Chuck
Also note that on the FreeBSD machine, I have enough RAM that to
buffer the entire
file, and in practice gstat shows that the drives are idle for
subsequent runs after
the first one.
Right now my code looks a lot like:
for(recordnum = 0; recordnum recordcount; recordnum++) {
buf = malloc(recordlength);
fread(buf, recordlength, 1, dbffile);
/* Do stuff with buf */
memoblock = getmemoblock(buf);
/* Skip to the requested block if we're not already there */
if(memoblock != currentmemofileblock) {
currentmemofileblock = memoblock;
fseek(memofile, currentmemofileblock * memoblocksize, SEEK_SET);
}
memohead = malloc(memoblocksize);
fread(memohead, memoblocksize, 1, memofile);
currentmemofileblock++;
/* Do stuff with memohead */
free(memohead);
free(buf);
}
...where recordlength == 13 in this one case. Given that the whole
file is buffered in
RAM, the small reads shouldn't make a difference, should they? I've
played with
setvbuf() and it shaves off a few percent of runtime, but nothing to
write home about.
Now, memofile gets quite a lot of seeks. Again, that shouldn't make
too much of a
difference if it's already buffered in RAM, should it? setvbuf() on
that file that
gets lots of random access actually made performance worse.
What else can I do to make my code run as well on FreeBSD as it does
on a much wimpier
Linux machine? I'm almost to the point of throwing in the towel and
making a Linux
server to do nothing more than run this one program if I can't
FreeBSD's performance
more on parity, and I honestly never thought I'd be considering that.
I'll gladly give shell access with my code and sample data files if
anyone is
interested in testing it.
--
Kirk Strauser
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
Kirk Strauser wrote: Kris Kennaway wrote: I don't understand what you meant by It's also doing a lot of lseek()s to what is likely the current position anyway (example: seek to 0x00, read 16 bytes, seek to 0x10, etc.). then. I just meant that 16 was a smaller number than 4096 to use in an example. :-) But anyway, it looks like I was wrong. Each record in this test file is 144 bytes long, but instead of reading 144 bytes, it's reading 4096 bytes then seeking backward 3952 (4096-144) bytes to the start of the next record. For instance: 99823 dumprecspg CALL lseek(0x3,0x1c8,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) 99823 dumprecspg CALL lseek(0x3,0x258,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) 99823 dumprecspg CALL lseek(0x3,0x2e8,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) 99823 dumprecspg CALL lseek(0x3,0x378,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) Now, I know this is suboptimal. My code is a patch on another, longer-established project that I wasn't a part of, and I probably can't do a lot about it without a pretty major rewrite. Still, I can't believe the same code is *so* much faster on Linux. I'd also swear that this is a regression and that it used to run much faster on the same FreeBSD machine back when it was running 6.x, but I never bothered to benchmark it then because it didn't seem to be an issue. Can you confirm or provide a code sample? What does strace show on Linux? Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Poor read() performance, and I can't profile it
I'm running a command (dumprecspg from my XBaseToPg project) on a FreeBSD 7 server. I've noticed that throughput on that program is a lot lower than I would have expected, and further investigation found it spending most of its time in the kernel, presumably in read() [1]. I was testing the same software on my desktop PC when I noticed that it ran *much* faster, and found that it was spending only about 1% as much time in the kernel on Linux as it was on FreeBSD. I ran a quick-and-dirty comparison of the same software on two different machines, the FreeBSD server being by far the more powerful of the two. I ran the same command on both machines from various filesystems (to rule out differences in drive performance), and posted the output of zsh's time command for the fastest run in each setting. The results are below. Any ideas what could be causing this horrible performance? I'm willing to try just about anything. FreeBSD on a Dell Poweredge 1600SC server: 7-STABLE from 2008-03-09 2x 2.4GHz P4 Xeon 3GB RAM Changes to /etc/make.conf: CPUTYPE?=pentium4 Kernel config: include GENERIC ident JAIL1 options PMAP_SHPGPERPROC=301 nooptionSCHED_4BSD option SCHED_ULE root : Fujitsu 36GB, 10k RPM Best time: 6.37s user 9.68s system 99% cpu 16.068 total /tmp : tmpfs Best time: 6.29s user 10.88s system 99% cpu 17.194 total /fast : 4 Seagate Cheetah 36GB, 15k RPM SCSI320 drives in RAID-0 with gstripe, 128KB stripe size with kern.geom.stripe.fast enabled and stripe.fast_failed=0 Best time: 6.60s user 9.46s system 99% cpu 16.088 total Conclusion: Since gstat showed all drives as idle through most of all the tests, it looks like the rest is running entirely from buffers. Linux on a Dell Dimension 4600 desktop: Ubuntu 8.04 2.4GHz P4 1GB RAM root: WD 250GB SATA Best time: 7.60s user 0.92s system 97% cpu 8.722 total Conclusion: I don't know if there's an equivalent to gstat in Linux, but the system overhead is about one-hundredth as much as in FreeBSD. [1] I can't run gprof on FreeBSD because if I build the binary with -pg, then it segfaults on startup: $ gdb /tmp/xbase/bin/dumprecspg /tmp/dumprecspg.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd... Core was generated by `dumprecspg'. Program terminated with signal 11, Segmentation fault. Reading symbols from /tmp/xbase/lib/libxbase64.so.1.0...done. Loaded symbols for /tmp/xbase/lib/libxbase64.so.1.0 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0807110c in main (ac=Cannot access memory at address 0x18 ) at dumprecspg.cpp:63 63 int main(int ac,char** av) (gdb) -- Kirk Strauser ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
Kirk Strauser wrote: I'm running a command (dumprecspg from my XBaseToPg project) on a FreeBSD 7 server. I've noticed that throughput on that program is a lot lower than I would have expected, and further investigation found it spending most of its time in the kernel, presumably in read() [1]. I was testing the same software on my desktop PC when I noticed that it ran *much* faster, and found that it was spending only about 1% as much time in the kernel on Linux as it was on FreeBSD. I ran a quick-and-dirty comparison of the same software on two different machines, the FreeBSD server being by far the more powerful of the two. I ran the same command on both machines from various filesystems (to rule out differences in drive performance), and posted the output of zsh's time command for the fastest run in each setting. The results are below. Any ideas what could be causing this horrible performance? I'm willing to try just about anything. ktrace(1) and check for the buffer size in use. It is probably too small. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
On Thursday 05 June 2008, Kris Kennaway wrote: Kirk Strauser wrote: ktrace(1) and check for the buffer size in use. It is probably too small. Kris It seems to be doing a lot of read()s with 4096-byte buffers. Is that what you mean? It's also doing a lot of lseek()s to what is likely the current position anyway (example: seek to 0x00, read 16 bytes, seek to 0x10, etc.). Would that make a difference, or should that be a NOP? -- Kirk Strauser ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
Kirk Strauser wrote: On Thursday 05 June 2008, Kris Kennaway wrote: Kirk Strauser wrote: ktrace(1) and check for the buffer size in use. It is probably too small. Kris It seems to be doing a lot of read()s with 4096-byte buffers. Is that what you mean? It's also doing a lot of lseek()s to what is likely the current position anyway (example: seek to 0x00, read 16 bytes, seek to 0x10, etc.). Would that make a difference, or should that be a NOP? No, if it's reading in 16 byte units it will explain the terrible performance. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
Kris Kennaway wrote: No, if it's reading in 16 byte units it will explain the terrible performance. No, it's actually doing 4096-byte reads. That was just an example of what I meant. Since I wrote that, though, I wrote a program to do 1,000,000 seeks to position 0, and it ran immeasurably fast. I'm guessing that lseek() is optimized to not do anything if you ask it to move to the position you're already at. Any other thoughts? There definitely aren't any setbuf() calls, and no matter what it still takes 100 times more kernel overhead on Linux than FreeBSD. Speaking of which, I think my next experiment will be to try the Linux binaries on FreeBSD and see if it behaves similarly. -- Kirk Strauser ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
Kirk Strauser wrote: Kris Kennaway wrote: No, if it's reading in 16 byte units it will explain the terrible performance. No, it's actually doing 4096-byte reads. That was just an example of what I meant. I don't understand what you meant by It's also doing a lot of lseek()s to what is likely the current position anyway (example: seek to 0x00, read 16 bytes, seek to 0x10, etc.). then. Please show a typical part of the ktrace output. Kris Since I wrote that, though, I wrote a program to do 1,000,000 seeks to position 0, and it ran immeasurably fast. I'm guessing that lseek() is optimized to not do anything if you ask it to move to the position you're already at. Any other thoughts? There definitely aren't any setbuf() calls, and no matter what it still takes 100 times more kernel overhead on Linux than FreeBSD. Speaking of which, I think my next experiment will be to try the Linux binaries on FreeBSD and see if it behaves similarly. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Poor read() performance, and I can't profile it
Kris Kennaway wrote: I don't understand what you meant by It's also doing a lot of lseek()s to what is likely the current position anyway (example: seek to 0x00, read 16 bytes, seek to 0x10, etc.). then. I just meant that 16 was a smaller number than 4096 to use in an example. :-) But anyway, it looks like I was wrong. Each record in this test file is 144 bytes long, but instead of reading 144 bytes, it's reading 4096 bytes then seeking backward 3952 (4096-144) bytes to the start of the next record. For instance: 99823 dumprecspg CALL lseek(0x3,0x1c8,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) 99823 dumprecspg CALL lseek(0x3,0x258,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) 99823 dumprecspg CALL lseek(0x3,0x2e8,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) 99823 dumprecspg CALL lseek(0x3,0x378,SEEK_SET,0) 99823 dumprecspg CALL read(0x3,0x8106000,0x1000) Now, I know this is suboptimal. My code is a patch on another, longer-established project that I wasn't a part of, and I probably can't do a lot about it without a pretty major rewrite. Still, I can't believe the same code is *so* much faster on Linux. I'd also swear that this is a regression and that it used to run much faster on the same FreeBSD machine back when it was running 6.x, but I never bothered to benchmark it then because it didn't seem to be an issue. -- Kirk Strauser ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
