Re: [HACKERS] Background writer process

2003-11-19 Thread Zeugswetter Andreas SB SD
1. Open WAL files with O_SYNC|O_DIRECT or O_SYNC(Not sure if Without grouping WAL writes that does not fly. Iff however such grouping is implemented that should deliver optimal performance. I don't think flushing WAL to the OS early (before a tx commits) is necessary, since writing 8k or

Re: [HACKERS] Background writer process

2003-11-18 Thread Shridhar Daithankar
Bruce Momjian wrote: Shridhar Daithankar wrote: On Friday 14 November 2003 22:10, Jan Wieck wrote: Shridhar Daithankar wrote: On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the

Re: [HACKERS] Background writer process

2003-11-18 Thread Zeugswetter Andreas SB SD
If the background writer uses fsync, it can write and allow the buffer to be reused and fsync later, while if we use O_SYNC, we have to wait for the O_SYNC write to happen before reusing the buffer; that will be slower. You can forget O_SYNC for datafiles for now. There would simply be too

Re: [HACKERS] Background writer process

2003-11-18 Thread Shridhar Daithankar
Zeugswetter Andreas SB SD wrote: 1. Open WAL files with O_SYNC|O_DIRECT or O_SYNC(Not sure if Without grouping WAL writes that does not fly. Iff however such grouping is implemented that should deliver optimal performance. I don't think flushing WAL to the OS early (before a tx commits) is

Re: [HACKERS] Background writer process

2003-11-17 Thread Shridhar Daithankar
On Friday 14 November 2003 22:10, Jan Wieck wrote: Shridhar Daithankar wrote: On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be

Re: [HACKERS] Background writer process

2003-11-17 Thread Zeugswetter Andreas SB SD
1. Open WAL files with O_SYNC|O_DIRECT or O_SYNC(Not sure if Without grouping WAL writes that does not fly. Iff however such grouping is implemented that should deliver optimal performance. I don't think flushing WAL to the OS early (before a tx commits) is necessary, since writing 8k or

Re: [HACKERS] Background writer process

2003-11-17 Thread Bruce Momjian
Shridhar Daithankar wrote: On Friday 14 November 2003 22:10, Jan Wieck wrote: Shridhar Daithankar wrote: On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much

Re: [HACKERS] Background writer process

2003-11-14 Thread Jan Wieck
Shridhar Daithankar wrote: On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after all, this does not change

Re: [HACKERS] Background writer process

2003-11-14 Thread Bruce Momjian
Shridhar Daithankar wrote: On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after all, this does not

Re: [HACKERS] Background writer process

2003-11-14 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Shridhar Daithankar wrote: Having fsync for regular data files and sync for WAL segment a comfortable compramise? Or this is going to use fsync for all of them. I think we still need sync() for WAL because sometimes backends are going to have to

[HACKERS] Background writer process

2003-11-13 Thread Jan Wieck
The attached diff is another attempt for distributing the write IO. It is a separate background process much like the checkpointer. It's purpose is to keep the number of dirty blocks in the buffer cache at a reasonable level and try that the buffers returned by the strategy for replacement are

Re: [HACKERS] Background writer process

2003-11-13 Thread Kurt Roeckx
On Thu, Nov 13, 2003 at 04:35:31PM -0500, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after all, this does not change the real

Re: [HACKERS] Background writer process

2003-11-13 Thread Bruce Momjian
Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 04:35:31PM -0500, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after all, this does not

Re: [HACKERS] Background writer process

2003-11-13 Thread Jan Wieck
Bruce Momjian wrote: Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 04:35:31PM -0500, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after

Re: [HACKERS] Background writer process

2003-11-13 Thread Bruce Momjian
Jan Wieck wrote: Bruce Momjian wrote: Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 04:35:31PM -0500, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be

Re: [HACKERS] Background writer process

2003-11-13 Thread Kurt Roeckx
On Thu, Nov 13, 2003 at 05:39:32PM -0500, Bruce Momjian wrote: Jan Wieck wrote: Bruce Momjian wrote: He found that write() itself didn't encourage the kernel to write the buffers to disk fast enough. I think the final solution will be to use fsync or O_SYNC. write() alone doesn't

Re: [HACKERS] Background writer process

2003-11-13 Thread Jan Wieck
Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 05:39:32PM -0500, Bruce Momjian wrote: Jan Wieck wrote: Bruce Momjian wrote: He found that write() itself didn't encourage the kernel to write the buffers to disk fast enough. I think the final solution will be to use fsync or O_SYNC. write()

Re: [HACKERS] Background writer process

2003-11-13 Thread Bruce Momjian
Kurt Roeckx wrote: On Thu, Nov 13, 2003 at 05:39:32PM -0500, Bruce Momjian wrote: Jan Wieck wrote: Bruce Momjian wrote: He found that write() itself didn't encourage the kernel to write the buffers to disk fast enough. I think the final solution will be to use fsync or O_SYNC.

Re: [HACKERS] Background writer process

2003-11-13 Thread Bruce Momjian
Jan Wieck wrote: That is all right and as said, how often, how much and how forced we do the IO can all be configurable and as flexible as people see fit. But whether you use sync(), fsync(), fdatasync(), O_SYNC, O_DSYNC or posix_fadvise(), somewhere you have to do the write(). And that

Re: [HACKERS] Background writer process

2003-11-13 Thread Shridhar Daithankar
On Friday 14 November 2003 03:05, Jan Wieck wrote: For sure the sync() needs to be replaced by the discussed fsync() of recently written files. And I think the algorithm how much and how often to flush can be significantly improved. But after all, this does not change the real checkpointing at