Re: [HACKERS] Where Can I Find The Code Segment For WAL Control?
[Please use reply to all so the list is CC-d] Charlie, I guess what you're after is to make sure the WAL buffers are shipped to the stand-by at the same time as they are committed to disk. In any other case your desire to have the stand-by EXACTLY in sync with the primary server will not gonna work. But that would mean that the communication to the stand-by will become a performance bottleneck, as all transactions are only finished after the WAL records for them are synced to the disk. So if you want your stand-by completely in sync with your primary, you will want that the transactions finish only after their WAL records are pushed to the stand-by too... and then if the communication to the stand-by fails, all your transactions will wait after it, possibly causing the primary to stop working properly. So now you have another point of failure, and instead of making the setup safer, you make it unsafer. What I want to say is that it is likely not feasible to keep the stand-by completely in sync. In practice it is enough to keep the standby NEARLY in sync with the primary server. That means you will ship the WAL records asynchronously, i.e. after they are written to the disk, and in a separate thread. What I'm after is to have a thread which starts streaming the current WAL file, and keeps streaming it as it grows. I'm not completely sure how I'll implement that, but I guess it will need to do a loop and transfer whatever records are available, and then sleep a few seconds if it reaches the end. It must be prepared to stumble upon partially written WAL records, and sleep on those too. On the stand-by end, the current partial WAL will not be used unless the stand-by is fired up... So I'm after a solution which makes sure the stand-by is as up to date as possible, with a few seconds allowed gap in normal operation, and possibly more if the communication channel has bandwidth problems and the server is very busy. Usually if the server crashes, than there are worse problems than the few seconds/minutes worth of lost transactions. To name one, if the server crashes you will have for sure at least a few minutes of downtime. At least for our application, downtime in a busy period is actually worse than the lost data (that we can recover from other logs)... Cheers, Csaba. On Sun, 2006-03-12 at 02:50, 王宝兵 wrote: Csaba: Firstly I must thank you for your help.Some of our designs are identical except the following: - create a standby manager program which only needs to know how to Access the primary server in order to create the standby (by connecting To it through normal data base connections and using the above mentioned Functions to stream the files); In my opinion,if we create a standby manager program and run it as a daemon process,it will check the state of the WAL files of the Principal every few seconds.But there is a risk for data lost.For an instance,if the Principal has flushed its log buffer to the disk and the dirty data are also flushed immediately,but the standby manager program is running in its interval.Then the Principal fails.In this situation,the Principal has updated its database but the log segment hasn't been sent to the Mirror,because the time point for the standby manager program to check the WAL files hasn't come.And then these data are lost. I think this situation will happen very probably in a big cooperation and it s very serious. Perhaps I have misunderstood your opinion.If that,I apologize. Charlie Wang ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Where Can I Find The Code Segment For WAL Control?
Charlie, I'm currently working on a similar solution (it's true I'm only in the thinking phase). I don't have too much time to spend on it (~few hours per week, on the train during commuting), so it's not progressing too fast... Nevertheless, I would like to do a first proof-of-concept version in a contrib module with minimal coding involved. I don't have time right now to explain all the details how I would like to do it, but in essence I want to: - create a few functions which expose the WAL files and the data base files as streams; - use the archive_command framework to control when the WAL files can be recycled, in combination with a WAL subscription system to make sure the standby won't be missing WAL files (with safeguards so the primary system will not go out of WAL file system space if the standby can't keep up with the WAL stream); - create a standby manager program which only needs to know how to access the primary server in order to create the standby (by connecting to it through normal data base connections and using the above mentioned functions to stream the files); These are just the main ideas, the devil is in the details. As per answering your question, a possible starting point would be to take a look at: src/backend/postmaster/pg_arch.c This one deals with archiving WAL files, so it could give you some insight in how you can deal with the WAL files. Or just search the source tree for 'WAL' ... I guess there's quite a few places in the code where WAL is involved. Next week I'll post more detailed design (some charts I did to better understand the idea, and some more text to add details), if anybody is interested in co-working on it, otherwise I'll just keep working in my own pace until it gets done... Cheers, Csaba. On Fri, 2006-03-10 at 07:34, 王宝兵 wrote: I am now trying to develop the PG to support real-time backup.My architecture is somehow similar to the Database Mirroring technology of SQL Server 2005.The server end of the system is consisted of two DB servers one is the Principal server,the other is the Mirror server.whenever the Principal flushes its log buffer to the local log file,it must send the content of its buffer to the Mirror simultaneously.After the mirror receives the buffer,it write to its own log file and send a response message to the Principal,then the Mirror redo/undo its log.By this way, we can guarantee the database instances of the two servers identical. But now I encounter a problem.I don't know where the functions to control log buffer are.Which code segment may I refer to? I have just participated the PG project for a short time,and I will appreciate your help very much! Look forward to you all! Charlie Wang ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Where Can I Find The Code Segment For WAL Control?
I am now trying to develop the PG to support real-time backup.My architecture is somehow similar to the Database Mirroring technology of SQL Server 2005.The server end of the system is consisted of two DB servers one is the Principal server,the other is the Mirror server.whenever the Principal flushes its log buffer to the local log file,it must send the content of its buffer to the Mirror simultaneously.After the mirror receives the buffer,it write to its own log file and send a response message to the Principal,then the Mirror redo/undo its log.By this way, we can guarantee the database instances of the two servers identical. But now I encounter a problem.I don't know where the functions to control log buffer are.Which code segment may I refer to? I have just participated the PG project for a short time,and I will appreciate your help very much! Look forward to you all! Charlie Wang ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
