Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Sat, Jan 5, 2013 at 11:11 PM, Magnus Hagander mag...@hagander.net wrote: On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut pete...@gmx.net wrote: On 1/3/13 12:30 PM, Robert Haas wrote: On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander mag...@hagander.net wrote: Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. It does seem like a hack. Couldn't this be implemented with a backend switch instead? It definitely is a bit of a hack. I assume by backend switch you mean guc, right? If so, no, not easily so. Because it's the archiver process that does the deleting. The process which deletes the old WAL files is the checkpointer. The checkpointer can access to the shared memory and know the location of the WAL record which has been already replicated to the standby. ISTM it's not difficult to implement the logic which pg_retainxlog provides into the checkpointer. How about just changing the checkpointer so that it checks whether the WAL file to delete has been already not only archived but also replicated if GUC flag is enabled? Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On 2013-01-26 02:21:00 +0900, Fujii Masao wrote: On Sat, Jan 5, 2013 at 11:11 PM, Magnus Hagander mag...@hagander.net wrote: On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut pete...@gmx.net wrote: On 1/3/13 12:30 PM, Robert Haas wrote: On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander mag...@hagander.net wrote: Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. It does seem like a hack. Couldn't this be implemented with a backend switch instead? It definitely is a bit of a hack. I assume by backend switch you mean guc, right? If so, no, not easily so. Because it's the archiver process that does the deleting. The process which deletes the old WAL files is the checkpointer. The checkpointer can access to the shared memory and know the location of the WAL record which has been already replicated to the standby. ISTM it's not difficult to implement the logic which pg_retainxlog provides into the checkpointer. How about just changing the checkpointer so that it checks whether the WAL file to delete has been already not only archived but also replicated if GUC flag is enabled? The problem with that is that to implement it robustly we would need persistent state about the replicas. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On 1/25/13 12:24 PM, Andres Freund wrote: On 2013-01-26 02:21:00 +0900, Fujii Masao wrote: The process which deletes the old WAL files is the checkpointer. The checkpointer can access to the shared memory and know the location of the WAL record which has been already replicated to the standby. ISTM it's not difficult to implement the logic which pg_retainxlog provides into the checkpointer. How about just changing the checkpointer so that it checks whether the WAL file to delete has been already not only archived but also replicated if GUC flag is enabled? That makes sense. The problem with that is that to implement it robustly we would need persistent state about the replicas. Well, pg_retainxlog kind of handwaves around that. If you use it in the default mode, it assumes that the pg_receivexlog entries in pg_stat_replication are that state. And then it says, if you use other kinds of clients, you need to keep track of that state yourself. But that seems to assume that pg_receivexlog never disconnects (thus losing its entries from pg_stat_replication). (pg_receivexlog is designed to automatically reconnect on connection loss, so this possibility was obviously thought about.) So it seems to me this just doesn't work (this way). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
After reviewing this, it appears to me that this is really just a very verbose version of archive_command = 'sleep $initialsleep; while test $(psql -AtX -c select pg_xlogfile_name(something) $$%f$$ collate \C\;) = t; sleep $sleep; done' I think it might be better to just document this as an example. I don't quite see the overhead of maintaining another tool justified. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut pete...@gmx.net wrote: After reviewing this, it appears to me that this is really just a very verbose version of archive_command = 'sleep $initialsleep; while test $(psql -AtX -c select pg_xlogfile_name(something) $$%f$$ collate \C\;) = t; sleep $sleep; done' I think it might be better to just document this as an example. I don't quite see the overhead of maintaining another tool justified. Well, obviously I don't entirely agree ;) Yes, it's a convenience command. Like pg_standby was. And like many other commands that we maintain as part of *core*, such as createuser, vacuumdb, etc. Those can all be done with an even *simpler* command than the one you suggest above. So I don't see that as an argument why it wouldn't be useful. Also, the command you suggest above does not work on Windows. You can probably write a .BAT file to do it for you, but I'm pretty sure it's impossible to do it as an archive_command there. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
Magnus Hagander mag...@hagander.net writes: On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut pete...@gmx.net wrote: I think it might be better to just document this as an example. I don't quite see the overhead of maintaining another tool justified. Well, obviously I don't entirely agree ;) Yes, it's a convenience command. Like pg_standby was. And like many other commands that we maintain as part of *core*, such as createuser, vacuumdb, etc. Those can all be done with an even *simpler* command than the one you suggest above. So I don't see that as an argument why it wouldn't be useful. We've discussed removing a lot of those tools, too. Not breaking backwards compatibility is probably the only reason they're still there. In the case at hand, I seem to recall from upthread that we expect this'd be obsolete in a release or two. If that's true then I think a para or two of documentation is a better idea than a tool we'll be essentially condemned to keep maintaining forever. Also, the command you suggest above does not work on Windows. You can probably write a .BAT file to do it for you, but I'm pretty sure it's impossible to do it as an archive_command there. Perhaps we could whip up such a .BAT file and put it in the docs? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Thu, Jan 24, 2013 at 6:25 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut pete...@gmx.net wrote: I think it might be better to just document this as an example. I don't quite see the overhead of maintaining another tool justified. Well, obviously I don't entirely agree ;) Yes, it's a convenience command. Like pg_standby was. And like many other commands that we maintain as part of *core*, such as createuser, vacuumdb, etc. Those can all be done with an even *simpler* command than the one you suggest above. So I don't see that as an argument why it wouldn't be useful. We've discussed removing a lot of those tools, too. Not breaking backwards compatibility is probably the only reason they're still there. In the case at hand, I seem to recall from upthread that we expect this'd be obsolete in a release or two. If that's true then I think a para or two of documentation is a better idea than a tool we'll be essentially condemned to keep maintaining forever. Not really sure there is such an expectation - any more than there was such an expectation when we initially put pg_standby in there. It would be *possible* to do it, certainly. But it's not like we have an actual plan. And AFAIK the stuff that was discussed upthread was a simplified version of it - not the full flexibility. That said, it's certainly a point that we'd have to maintain it. But I don't see why we'd have to maintain it beyond the point where we included the same functionality in core, if we did. Also, the command you suggest above does not work on Windows. You can probably write a .BAT file to do it for you, but I'm pretty sure it's impossible to do it as an archive_command there. Perhaps we could whip up such a .BAT file and put it in the docs? That would probably work, yes. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Tue, Jan 15, 2013 at 2:51 AM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jan 14, 2013 at 5:56 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. Do you have the specs of such better way? Would it be a problem to have both pg_retainxlog and the new way? Well, I think in the long term we are likely to want the master to have some kind of ability to track the positions of its slaves, even when they are disconnected. And, optionally, to retain the WAL that they need, again even when they are disconnected. If such an ability materializes, this will be moot (even as I think that pg_standby is now largely moot, at least for new installations, now that we have standby_mode=on). I agree. But just as we had pg_standby for quite a while before we got standby_mode=on, I believe we should have pg_retainxlog (or something like it) until we have something more integrated. Yep, not disagreeing. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Sat, Jan 5, 2013 at 3:11 PM, Magnus Hagander mag...@hagander.net wrote: On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut pete...@gmx.net wrote: On 1/3/13 12:30 PM, Robert Haas wrote: On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander mag...@hagander.net wrote: Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. It does seem like a hack. Couldn't this be implemented with a backend switch instead? It definitely is a bit of a hack. I assume by backend switch you mean guc, right? If so, no, not easily so. Because it's the archiver process that does the deleting. And this process does not have access to a full backend interface, e.g. the ability to run a query. We could make it look at the same data that's currently shown in pg_stat_replicatoin through shared memory, but thta would *only* work in the very most simple cases (e.g. a single pg_receivexlog and no other replication). The ability to run a custom SQL query is going to be necessary for anything a bit more advanced. Also, as a small practical matter, since this is a server-side program (since it's being used as archive_command), we shouldn't put it into the pg_basebackup directory, because that would blur the lines about what to install where, in particular for the translations. Good argument. That along with the being a hack, and the comment from Robert, means that maybe contrib/ is a better place for it, yes. Here's a version for inclusion in /contrib. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ pg_retainxlog.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. Do you have the specs of such better way? Would it be a problem to have both pg_retainxlog and the new way? Well, I think in the long term we are likely to want the master to have some kind of ability to track the positions of its slaves, even when they are disconnected. And, optionally, to retain the WAL that they need, again even when they are disconnected. If such an ability materializes, this will be moot (even as I think that pg_standby is now largely moot, at least for new installations, now that we have standby_mode=on). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Mon, Jan 14, 2013 at 5:56 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. Do you have the specs of such better way? Would it be a problem to have both pg_retainxlog and the new way? Well, I think in the long term we are likely to want the master to have some kind of ability to track the positions of its slaves, even when they are disconnected. And, optionally, to retain the WAL that they need, again even when they are disconnected. If such an ability materializes, this will be moot (even as I think that pg_standby is now largely moot, at least for new installations, now that we have standby_mode=on). I agree. But just as we had pg_standby for quite a while before we got standby_mode=on, I believe we should have pg_retainxlog (or something like it) until we have something more integrated. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut pete...@gmx.net wrote: On 1/3/13 12:30 PM, Robert Haas wrote: On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander mag...@hagander.net wrote: Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. It does seem like a hack. Couldn't this be implemented with a backend switch instead? It definitely is a bit of a hack. I assume by backend switch you mean guc, right? If so, no, not easily so. Because it's the archiver process that does the deleting. And this process does not have access to a full backend interface, e.g. the ability to run a query. We could make it look at the same data that's currently shown in pg_stat_replicatoin through shared memory, but thta would *only* work in the very most simple cases (e.g. a single pg_receivexlog and no other replication). The ability to run a custom SQL query is going to be necessary for anything a bit more advanced. Also, as a small practical matter, since this is a server-side program (since it's being used as archive_command), we shouldn't put it into the pg_basebackup directory, because that would blur the lines about what to install where, in particular for the translations. Good argument. That along with the being a hack, and the comment from Robert, means that maybe contrib/ is a better place for it, yes. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander mag...@hagander.net wrote: So, it turns out the reason I got no feedback on this tool, was that I forgot both to email about and to actually push the code to github :O So this is actually code that's almost half a year old and that I was supposed to submit for the first or second commitfest. Oops. So, the tool and a README for it right now can be found at https://github.com/mhagander/pg_retainxlog for the time being. The idea behind the tool is to plug a hole in the case when pg_receivexlog is used for log archiving, which is that you still need a blocking archive_command in order to make sure that files aren't recycled on the master. So for 9.2 you can do this with an archive_command that checks if the file has appeared properly on the slave - but that usually means you're back at requiring ssh connectivity between the machines, for example. Even though this information is actually avialable on the master... This can also be of use to pure replication scenarios, where you don't know how to tune wal_keep_segments, but using actual live feedback instead of guessing. When pg_retainxlog is used as an archive_command, it will check the pg_stat_replication view instead of checking the slave. It will then only return ok once the requested logfile has been replicated to the slave. By default it will look for a replication client named pg_receivexlog, but it supports overriding the query with anything - so you can say things like needs to have arrived on at least two replication slaves before we consider it archived. Or if used instead of wal_keep_segmnets, needs to have arrived at *all* replication slaves. Is this a tool that people would like to see included in the general toolchain? If so, I'll reformat it to work in the general build environment and submit it for the last commitfest. (comments on the code itself are of course also welcome) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers +1 to this concept, however it may be implemented. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On 1/3/13 12:30 PM, Robert Haas wrote: On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander mag...@hagander.net wrote: Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. It does seem like a hack. Couldn't this be implemented with a backend switch instead? Also, as a small practical matter, since this is a server-side program (since it's being used as archive_command), we shouldn't put it into the pg_basebackup directory, because that would blur the lines about what to install where, in particular for the translations. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
Robert Haas robertmh...@gmail.com writes: Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. Do you have the specs of such better way? Would it be a problem to have both pg_retainxlog and the new way? -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander mag...@hagander.net wrote: So, it turns out the reason I got no feedback on this tool, was that I forgot both to email about and to actually push the code to github :O So this is actually code that's almost half a year old and that I was supposed to submit for the first or second commitfest. Oops. So, the tool and a README for it right now can be found at https://github.com/mhagander/pg_retainxlog for the time being. The idea behind the tool is to plug a hole in the case when pg_receivexlog is used for log archiving, which is that you still need a blocking archive_command in order to make sure that files aren't recycled on the master. So for 9.2 you can do this with an archive_command that checks if the file has appeared properly on the slave - but that usually means you're back at requiring ssh connectivity between the machines, for example. Even though this information is actually avialable on the master... This can also be of use to pure replication scenarios, where you don't know how to tune wal_keep_segments, but using actual live feedback instead of guessing. When pg_retainxlog is used as an archive_command, it will check the pg_stat_replication view instead of checking the slave. It will then only return ok once the requested logfile has been replicated to the slave. By default it will look for a replication client named pg_receivexlog, but it supports overriding the query with anything - so you can say things like needs to have arrived on at least two replication slaves before we consider it archived. Or if used instead of wal_keep_segmnets, needs to have arrived at *all* replication slaves. Is this a tool that people would like to see included in the general toolchain? If so, I'll reformat it to work in the general build environment and submit it for the last commitfest. (comments on the code itself are of course also welcome) I would tend to vote for putting this in contrib rather than src/bin. But apart from that I have no objection to the idea. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Thu, Jan 3, 2013 at 3:13 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander mag...@hagander.net wrote: So, it turns out the reason I got no feedback on this tool, was that I forgot both to email about and to actually push the code to github :O So this is actually code that's almost half a year old and that I was supposed to submit for the first or second commitfest. Oops. So, the tool and a README for it right now can be found at https://github.com/mhagander/pg_retainxlog for the time being. The idea behind the tool is to plug a hole in the case when pg_receivexlog is used for log archiving, which is that you still need a blocking archive_command in order to make sure that files aren't recycled on the master. So for 9.2 you can do this with an archive_command that checks if the file has appeared properly on the slave - but that usually means you're back at requiring ssh connectivity between the machines, for example. Even though this information is actually avialable on the master... This can also be of use to pure replication scenarios, where you don't know how to tune wal_keep_segments, but using actual live feedback instead of guessing. When pg_retainxlog is used as an archive_command, it will check the pg_stat_replication view instead of checking the slave. It will then only return ok once the requested logfile has been replicated to the slave. By default it will look for a replication client named pg_receivexlog, but it supports overriding the query with anything - so you can say things like needs to have arrived on at least two replication slaves before we consider it archived. Or if used instead of wal_keep_segmnets, needs to have arrived at *all* replication slaves. Is this a tool that people would like to see included in the general toolchain? If so, I'll reformat it to work in the general build environment and submit it for the last commitfest. (comments on the code itself are of course also welcome) I would tend to vote for putting this in contrib rather than src/bin. But apart from that I have no objection to the idea. Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander mag...@hagander.net wrote: Any particular reason? It goes pretty tightly together with pg_receivexlog, which is why I'd prefer putting it alongside that one. But if you have a good argument against it, I can change my mind :) Mostly that it seems like a hack, and I suspect we may come up with a better way to do this in the future. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_retainxlog for inclusion in 9.3?
So, it turns out the reason I got no feedback on this tool, was that I forgot both to email about and to actually push the code to github :O So this is actually code that's almost half a year old and that I was supposed to submit for the first or second commitfest. Oops. So, the tool and a README for it right now can be found at https://github.com/mhagander/pg_retainxlog for the time being. The idea behind the tool is to plug a hole in the case when pg_receivexlog is used for log archiving, which is that you still need a blocking archive_command in order to make sure that files aren't recycled on the master. So for 9.2 you can do this with an archive_command that checks if the file has appeared properly on the slave - but that usually means you're back at requiring ssh connectivity between the machines, for example. Even though this information is actually avialable on the master... This can also be of use to pure replication scenarios, where you don't know how to tune wal_keep_segments, but using actual live feedback instead of guessing. When pg_retainxlog is used as an archive_command, it will check the pg_stat_replication view instead of checking the slave. It will then only return ok once the requested logfile has been replicated to the slave. By default it will look for a replication client named pg_receivexlog, but it supports overriding the query with anything - so you can say things like needs to have arrived on at least two replication slaves before we consider it archived. Or if used instead of wal_keep_segmnets, needs to have arrived at *all* replication slaves. Is this a tool that people would like to see included in the general toolchain? If so, I'll reformat it to work in the general build environment and submit it for the last commitfest. (comments on the code itself are of course also welcome) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
On 01/01/2013 04:10 PM, Magnus Hagander wrote: So, it turns out the reason I got no feedback on this tool, was that I forgot both to email about and to actually push the code to github :O So this is actually code that's almost half a year old and that I was supposed to submit for the first or second commitfest. Oops. So, the tool and a README for it right now can be found at https://github.com/mhagander/pg_retainxlog for the time being. The idea behind the tool is to plug a hole in the case when pg_receivexlog is used for log archiving, which is that you still need a blocking archive_command in order to make sure that files aren't recycled on the master. So for 9.2 you can do this with an archive_command that checks if the file has appeared properly on the slave - but that usually means you're back at requiring ssh connectivity between the machines, for example. Even though this information is actually avialable on the master... This can also be of use to pure replication scenarios, where you don't know how to tune wal_keep_segments, but using actual live feedback instead of guessing. When pg_retainxlog is used as an archive_command, it will check the pg_stat_replication view instead of checking the slave. It will then only return ok once the requested logfile has been replicated to the slave. By default it will look for a replication client named pg_receivexlog, but it supports overriding the query with anything - so you can say things like needs to have arrived on at least two replication slaves before we consider it archived. Or if used instead of wal_keep_segmnets, needs to have arrived at *all* replication slaves. Is this a tool that people would like to see included in the general toolchain? If so, I'll reformat it to work in the general build environment and submit it for the last commitfest. +1 Hannu Krosing -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_retainxlog for inclusion in 9.3?
Magnus Hagander mag...@hagander.net writes: Is this a tool that people would like to see included in the general toolchain? If so, I'll reformat it to work in the general build environment and submit it for the last commitfest. Please do. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
