RE: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
FSYNC:257tps O_DSYNC: 333tps Just(?) 30% faster, -:( First of all, if you ask me, that is one hell of an improvement :-) Of course -:) But tfsync tests were more promising -:) Probably we should update XLogWrite to write() more than 1 block, but Tom should apply his patches first (btw, did you implement "log file size" condition for checkpoints, Tom?). Vadim ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
"Mikheev, Vadim" [EMAIL PROTECTED] writes: Probably we should update XLogWrite to write() more than 1 block, but Tom should apply his patches first (btw, did you implement "log file size" condition for checkpoints, Tom?). Yes I did. There's a variable now to specify a checkpoint every N log segments --- I figured that was good enough resolution, and it allowed the test to be made only when we're rolling over to a new segment, so it's not in a time-critical path. If you're happy with what I did so far, I'll go ahead and commit. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
Denis Perchine [EMAIL PROTECTED] writes: On Saturday 10 March 2001 08:41, Tom Lane wrote: More numbers, these from a Powerbook G3 laptop running Linux 2.2: Eeegghhh. Sorry... But where did you get O_DSYNC on Linux? bits/fcntl.h: # define O_DSYNC O_SYNC Hm, must be. Okay, so those two sets of numbers should be taken as fsync() and O_SYNC respectively. Still the conclusion seems pretty clear: the open() options are way more efficient than calling fsync() separately. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
Ok, I've made changes in xlog.c and run tests: Could you send me your diffs? Sorry, Monday only. Vadim ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
"Mikheev, Vadim" [EMAIL PROTECTED] writes: Tom, could you run this test for different block sizes? Up to 32*8k? You mean changing the amount written per write(), while holding the total file size constant, right? Yes. Currently XLogWrite writes 8k blocks one by one. From what I've seen on Solaris we can use O_DSYNC there without changing XLogWrite to write() more than 1 block (if 1 block is available for writing). But on other platforms write(BLOCKS_TO_WRITE * 8k) + fsync() probably will be faster than BLOCKS_TO_WRITE * write(8k) (for file opened with O_DSYNC) if BLOCKS_TO_WRITE 1. I just wonder with what BLOCKS_TO_WRITE we'll see same times for both approaches. Okay, I changed the program to char zbuffer[8192 * BLOCKS]; (all else the same) and on HPUX 10.20 I get $ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=1 tfsync.c $ time a.out real1m18.48s user0m0.04s sys 0m34.69s $ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=4 tfsync.c $ time a.out real0m35.10s user0m0.01s sys 0m9.08s $ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=8 tfsync.c $ time a.out real0m29.75s user0m0.01s sys 0m5.23s $ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=32 tfsync.c $ time a.out real0m22.77s user0m0.01s sys 0m1.80s $ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=64 tfsync.c $ time a.out real0m22.08s user0m0.01s sys 0m1.25s $ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=1 tfsync.c $ time a.out real0m20.64s user0m0.02s sys 0m0.67s $ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=4 tfsync.c $ time a.out real0m20.72s user0m0.01s sys 0m0.57s $ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=32 tfsync.c $ time a.out real0m20.59s user0m0.01s sys 0m0.61s $ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=64 tfsync.c $ time a.out real0m20.86s user0m0.01s sys 0m0.69s So I also see that there is no benefit to writing more than one block at a time with ODSYNC. And even at half a meg per write, DSYNC is slower than ODSYNC with 8K per write! Note the fairly high system-time consumption for DSYNC, too. I think this is not so much a matter of a really good ODSYNC implementation, as a really bad DSYNC one ... regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
RE: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
$ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=1 tfsync.c ^^^ You should use -DUSE_OSYNC to test O_SYNC. So you've tested N * write() + fsync(), exactly what I've asked -:) So I also see that there is no benefit to writing more than one block at a time with ODSYNC. And even at half a meg per write, DSYNC is slower than ODSYNC with 8K per write! Note the fairly high system-time consumption for DSYNC, too. I think this is not so much a matter of a really good ODSYNC implementation, as a really bad DSYNC one ... So seems we can use O_DSYNC without losing log write performance comparing with write() + fsync. Though, we didn't tested write() + fdatasync() yet... Vadim ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
More numbers, these from a Powerbook G3 laptop running Linux 2.2: [tgl@g3 tmp]$ uname -a Linux g3 2.2.18-4hpmac #1 Thu Dec 21 15:16:15 MST 2000 ppc unknown [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=1 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m32.418s user0m0.020s sys 0m14.020s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=4 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m10.894s user0m0.000s sys 0m4.030s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=8 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m7.211s user0m0.000s sys 0m2.200s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=32 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m4.441s user0m0.020s sys 0m0.870s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=64 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m4.488s user0m0.000s sys 0m0.640s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=1 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m3.725s user0m0.000s sys 0m0.310s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=4 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m3.785s user0m0.000s sys 0m0.290s [tgl@g3 tmp]$ gcc -Wall -O -DINIT_WRITE -DUSE_ODSYNC -DBLOCKS=64 tfsync.c [tgl@g3 tmp]$ time ./a.out real0m3.753s user0m0.010s sys 0m0.300s Starting to look like we should just use ODSYNC where available, and forget about dumping more per write ... regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
RE: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
Starting to look like we should just use ODSYNC where available, and forget about dumping more per write ... I'll run these tests on RedHat 7.0 tomorrow. Vadim ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
"Mikheev, Vadim" [EMAIL PROTECTED] writes: $ gcc -Wall -O -DINIT_WRITE -DUSE_DSYNC -DBLOCKS=1 tfsync.c ^^^ You should use -DUSE_OSYNC to test O_SYNC. Ooops ... let's hear it for cut-and-paste, and for sharp-eyed readers! Just for completeness, here are the results for O_SYNC: $ gcc -Wall -O -DINIT_WRITE -DUSE_OSYNC -DBLOCKS=1 tfsync.c $ time a.out real0m43.44s user0m0.02s sys 0m0.74s $ gcc -Wall -O -DINIT_WRITE -DUSE_OSYNC -DBLOCKS=4 tfsync.c $ time a.out real0m26.38s user0m0.01s sys 0m0.59s $ gcc -Wall -O -DINIT_WRITE -DUSE_OSYNC -DBLOCKS=8 tfsync.c $ time a.out real0m23.86s user0m0.01s sys 0m0.59s $ gcc -Wall -O -DINIT_WRITE -DUSE_OSYNC -DBLOCKS=64 tfsync.c $ time a.out real0m22.93s user0m0.01s sys 0m0.66s Better than fsync(), but still not up to O_DSYNC. So seems we can use O_DSYNC without losing log write performance comparing with write() + fsync. Though, we didn't tested write() + fdatasync() yet... Good point, we should check fdatasync() too --- although I have no machines where it's different from fsync(). regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: AW: AW: AW: [HACKERS] WAL does not recover gracefully from ou t-of -dis k-sp ace
"Mikheev, Vadim" [EMAIL PROTECTED] writes: Ok, I've made changes in xlog.c and run tests: Could you send me your diffs? regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
