Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
On 12/8/13 3:08 PM, MauMau wrote: #5 is output when the DBA shuts down the replication standby server. #6 is output when the DBA shuts down the server if he is using any custom background worker. These messages are always output. What I'm seeing as a problem is that FATAL messages are output in a normal situation, which worries the DBA, and those messages don't help the DBA with anything. They merely worry the DBA. While worry is something to be avoided not logging messages is not the correct solution. On the project side looking for ways and places to better communicate to the user is worthwhile in the absence of adding additional complexity to the logging system. The user/DBA, though, is expected to learn how the proces works, ideally in a testing environment where worry is much less a problem, and understand that some of what they are seeing are client-oriented messages that they should be aware of but that do not indicate server-level issues by themselves. I see, It might makes sense to make the DBA learn how the system works, so that more DBAs can solve problems by themselves. However, I wonder how those messages (just stopping server process by admin request) are useful for the DBA. If they are useful, why don't other background processes (e.g. archiver, writer, walsender, etc.) output such a message when stopping? For information, #5 and #6 are not client-oriented. If we want to keep the messages, I think LOG or DEBUG would be appropriate. How about altering ereport(FATAL, ...) to ereport(LOG, ...); proc_exit(1)? IMNSHO, this isn't a worry thing, it's a complete mis-categorization of what's happening. Do we output FATAL when you disconnect from the database? Something that is happening *by design* should not be considered as fatal. I'm not saying we shouldn't log it at all, but it's silly to call it fatal just because the process is going away *when we did something that needs to make it go away*. What makes it worse is what happens if you do something like pg_terminate_backend() on one of those? That *should* create a fatal error (or at least be treated the same way we treat any other use of pg_terminate_backend()). So in this case I'm betting the real problem is that an orderly shutdown is calling pg_terminate_backend() or the equivalent. While that's convenient from a code standpoint, it's inaccurate from an API standpoint. -- Jim C. Nasby, Data Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
On 12/06/2013 03:02 AM, Josh Berkus wrote: Heck, I'd be happy just to have a class of messages which specifically means OMG, there's something wrong with the server, that is, a flag for messages which only occur when PostgreSQL encounters a bug, data corrpution, or platform error. Right now, I have to suss those out by regex. +10 That's what I really need to see in the logs, too. Using SQLState might be reasonable, but I'd be concerned we'd find cases where the same standard-specififed SQLState should be sent to the *client*, despite different underlying causes for the server of different levels of DBA concern. I'd rather not suppress logging of user-level errors, etc; they're important too. It's just necessary to be able to separate them from internal errors that should never happen when the system is operating correctly in order to do effective log-based alerting. After all, I *don't* want to get an SMS whenever the deadlock detector kicks in and someone's created a database in de_DE so the string pattern match doesn't kick in. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
From: David Johnston pol...@yahoo.com 5. FATAL: terminating walreceiver process due to administrator command 6. FATAL: terminating background worker \%s\ due to administrator command 5 and 6: I don't fully understand when they would happen but likely fall into the same the DBA should know what is going on with their server and confirm any startup/shutdown activity it is involved with. They might be better categorized NOTICE level if they were in response to a administrator action, versus in response to a crashed process, but even for the user-initiated situation making sure they hit the log but using FATAL is totally understandable and IMO desirable. #5 is output when the DBA shuts down the replication standby server. #6 is output when the DBA shuts down the server if he is using any custom background worker. These messages are always output. What I'm seeing as a problem is that FATAL messages are output in a normal situation, which worries the DBA, and those messages don't help the DBA with anything. They merely worry the DBA. While worry is something to be avoided not logging messages is not the correct solution. On the project side looking for ways and places to better communicate to the user is worthwhile in the absence of adding additional complexity to the logging system. The user/DBA, though, is expected to learn how the proces works, ideally in a testing environment where worry is much less a problem, and understand that some of what they are seeing are client-oriented messages that they should be aware of but that do not indicate server-level issues by themselves. I see, It might makes sense to make the DBA learn how the system works, so that more DBAs can solve problems by themselves. However, I wonder how those messages (just stopping server process by admin request) are useful for the DBA. If they are useful, why don't other background processes (e.g. archiver, writer, walsender, etc.) output such a message when stopping? For information, #5 and #6 are not client-oriented. If we want to keep the messages, I think LOG or DEBUG would be appropriate. How about altering ereport(FATAL, ...) to ereport(LOG, ...); proc_exit(1)? Regards MauMau -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
From: David Johnston pol...@yahoo.com 5. FATAL: terminating walreceiver process due to administrator command 6. FATAL: terminating background worker \%s\ due to administrator command 5 and 6: I don't fully understand when they would happen but likely fall into the same the DBA should know what is going on with their server and confirm any startup/shutdown activity it is involved with. They might be better categorized NOTICE level if they were in response to a administrator action, versus in response to a crashed process, but even for the user-initiated situation making sure they hit the log but using FATAL is totally understandable and IMO desirable. #5 is output when the DBA shuts down the replication standby server. #6 is output when the DBA shuts down the server if he is using any custom background worker. These messages are always output. What I'm seeing as a problem is that FATAL messages are output in a normal situation, which worries the DBA, and those messages don't help the DBA with anything. They merely worry the DBA. Regards MauMau -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
From: David Johnston pol...@yahoo.com PITR/Failover is not generally that frequent an occurrence but I will agree that these events are likely common during such. Maybe PITR/Failover mode can output something in the logs to alleviate user angst about these frequent events? I'm doubting that a totally separate mechanism can be used for this mode but instead of looking for things to remove how about adding some additional coddling to the logs and the beginning and end of the mode change? Yes, those messages are not output frequently, because they are only output during planned or unplanned downtime. But frequency does not matter here. Imagine the DBA's heart. They need to perform maintenance or recovery operation in a hurry and wish to not encounter any trouble. They would panic if something goes trouble when he expects to see no trouble. The FATAL messages merely make him worried. The extra FATAL messages can be a problem for support staff. What do you feel if the DBA asks you for help when some emergency trouble occurs during recovery, with a few important messages buried in lots of unnecessary FATAL ones? Regards MauMau -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
MauMau wrote From: David Johnston lt; polobo@ gt; 5. FATAL: terminating walreceiver process due to administrator command 6. FATAL: terminating background worker \%s\ due to administrator command 5 and 6: I don't fully understand when they would happen but likely fall into the same the DBA should know what is going on with their server and confirm any startup/shutdown activity it is involved with. They might be better categorized NOTICE level if they were in response to a administrator action, versus in response to a crashed process, but even for the user-initiated situation making sure they hit the log but using FATAL is totally understandable and IMO desirable. #5 is output when the DBA shuts down the replication standby server. #6 is output when the DBA shuts down the server if he is using any custom background worker. These messages are always output. What I'm seeing as a problem is that FATAL messages are output in a normal situation, which worries the DBA, and those messages don't help the DBA with anything. They merely worry the DBA. While worry is something to be avoided not logging messages is not the correct solution. On the project side looking for ways and places to better communicate to the user is worthwhile in the absence of adding additional complexity to the logging system. The user/DBA, though, is expected to learn how the proces works, ideally in a testing environment where worry is much less a problem, and understand that some of what they are seeing are client-oriented messages that they should be aware of but that do not indicate server-level issues by themselves. They should probably run the log through a filter (a template of which the project should provide) that takes the raw log and filters out those things that, relative to the role and context, are really better classified as NOTICE even if the log-level is shown as FATAL. Raw logs should be verbose so that tools can have more data with which to make decisions. You can always hide what is provided but you can never show was has been omitted. I get that people wil scan raw logs but targeting that lowest-denominator isn't necessarily the best thing to do. Instead, getting those people on board with better practices and showing (and providing) them helpful tools should be the goal. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/RFC-Shouldn-t-we-remove-annoying-FATAL-messages-from-server-log-tp5781899p5782267.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
From: Josh Berkus j...@agliodbs.com Heck, I'd be happy just to have a class of messages which specifically means OMG, there's something wrong with the server, that is, a flag for messages which only occur when PostgreSQL encounters a bug, data corrpution, or platform error. Right now, I have to suss those out by regex. What are the issues of using SQLSTATE XXnnn as a filter? Regards MauMau -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
From: Alvaro Herrera alvhe...@2ndquadrant.com There was also the idea that this would be driven off SQLSTATE but this seems pretty unwieldy to me. You are referring to this long discussion, don't you? http://www.postgresql.org/message-id/19791.1335902...@sss.pgh.pa.us I've read it before, and liked the SQLSTATE-based approach. It seems like properly assigned SQLSTATEs can be used as message IDs, and pairs of SQLSTATE and its user action might be utilized to provide sophisticated database administration GUI. That discussion sounds interesting, and I want to take more time to consider. But what do you think of my original suggestion to easily solve the current issue? I'd like to remove the current annoying problem first before spending much time for more excited infrastructure. Regards MauMau -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
MauMau maumau...@gmail.com writes: That discussion sounds interesting, and I want to take more time to consider. But what do you think of my original suggestion to easily solve the current issue? I'd like to remove the current annoying problem first before spending much time for more excited infrastructure. There is no enthusiasm for a quick-hack solution here, and most people don't actually agree with your proposal that these errors should never get logged. So no, that is not happening. You can hack your local copy that way if you like of course, but it's not getting committed. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
From: Tom Lane t...@sss.pgh.pa.us There is no enthusiasm for a quick-hack solution here, and most people don't actually agree with your proposal that these errors should never get logged. So no, that is not happening. You can hack your local copy that way if you like of course, but it's not getting committed. Oh, I may have misunderstood your previous comments. I got the impression that you and others regard those messages (except too many clients) as unnecessary in server log. 1. FATAL: the database system is starting up 2. FATAL: the database system is shutting down 3. FATAL: the database system is in recovery mode 5. FATAL: terminating walreceiver process due to administrator command 6. FATAL: terminating background worker \%s\ due to administrator command Could you tell me why these are necessary in server log? I guess like this. Am I correct? * #1 through #3 are necessary for the DBA to investigate and explain to the end user why he cannot connect to the database. * #4 and #5 are unnecessary for the DBA. I can't find out any reason why these are useful for the DBA. Regards MauMau -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
MauMau wrote From: Tom Lane lt; tgl@.pa gt; There is no enthusiasm for a quick-hack solution here, and most people don't actually agree with your proposal that these errors should never get logged. So no, that is not happening. You can hack your local copy that way if you like of course, but it's not getting committed. Oh, I may have misunderstood your previous comments. I got the impression that you and others regard those messages (except too many clients) as unnecessary in server log. 1. FATAL: the database system is starting up 2. FATAL: the database system is shutting down 3. FATAL: the database system is in recovery mode 5. FATAL: terminating walreceiver process due to administrator command 6. FATAL: terminating background worker \%s\ due to administrator command Could you tell me why these are necessary in server log? I guess like this. Am I correct? * #1 through #3 are necessary for the DBA to investigate and explain to the end user why he cannot connect to the database. * #4 and #5 are unnecessary for the DBA. I can't find out any reason why these are useful for the DBA. For me 1-3 are unusual events in production situations and so knowing when they occur, and confirming they occurred for a good reason, is a key job of the DBA. 5 and 6: I don't fully understand when they would happen but likely fall into the same the DBA should know what is going on with their server and confirm any startup/shutdown activity it is involved with. They might be better categorized NOTICE level if they were in response to a administrator action, versus in response to a crashed process, but even for the user-initiated situation making sure they hit the log but using FATAL is totally understandable and IMO desirable. I'd ask in what situations are these messages occurring so frequently that they are becoming noise instead of useful data? Sorry if I missed your use-case explanation up-thread. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/RFC-Shouldn-t-we-remove-annoying-FATAL-messages-from-server-log-tp5781899p5782234.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
David Johnston wrote MauMau wrote From: Tom Lane lt; tgl@.pa gt; There is no enthusiasm for a quick-hack solution here, and most people don't actually agree with your proposal that these errors should never get logged. So no, that is not happening. You can hack your local copy that way if you like of course, but it's not getting committed. Oh, I may have misunderstood your previous comments. I got the impression that you and others regard those messages (except too many clients) as unnecessary in server log. 1. FATAL: the database system is starting up 2. FATAL: the database system is shutting down 3. FATAL: the database system is in recovery mode 5. FATAL: terminating walreceiver process due to administrator command 6. FATAL: terminating background worker \%s\ due to administrator command Could you tell me why these are necessary in server log? I guess like this. Am I correct? * #1 through #3 are necessary for the DBA to investigate and explain to the end user why he cannot connect to the database. * #4 and #5 are unnecessary for the DBA. I can't find out any reason why these are useful for the DBA. For me 1-3 are unusual events in production situations and so knowing when they occur, and confirming they occurred for a good reason, is a key job of the DBA. 5 and 6: I don't fully understand when they would happen but likely fall into the same the DBA should know what is going on with their server and confirm any startup/shutdown activity it is involved with. They might be better categorized NOTICE level if they were in response to a administrator action, versus in response to a crashed process, but even for the user-initiated situation making sure they hit the log but using FATAL is totally understandable and IMO desirable. I'd ask in what situations are these messages occurring so frequently that they are becoming noise instead of useful data? Sorry if I missed your use-case explanation up-thread. David J. Went and scanned the thread: PITR/Failover is not generally that frequent an occurrence but I will agree that these events are likely common during such. Maybe PITR/Failover mode can output something in the logs to alleviate user angst about these frequent events? I'm doubting that a totally separate mechanism can be used for this mode but instead of looking for things to remove how about adding some additional coddling to the logs and the beginning and end of the mode change? Thought provoking only as I have not actually been a user of said feature. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/RFC-Shouldn-t-we-remove-annoying-FATAL-messages-from-server-log-tp5781899p5782235.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
MauMau wrote From: Tom Lane lt; tgl@.pa gt; There is no enthusiasm for a quick-hack solution here, and most people don't actually agree with your proposal that these errors should never get logged. So no, that is not happening. You can hack your local copy that way if you like of course, but it's not getting committed. Oh, I may have misunderstood your previous comments. I got the impression that you and others regard those messages (except too many clients) as unnecessary in server log. 1. FATAL: the database system is starting up How about altering the message to tone down the severity by a half-step... FATAL: (probably) not! - the database system is starting up David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/RFC-Shouldn-t-we-remove-annoying-FATAL-messages-from-server-log-tp5781899p5782236.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
Tom Lane-2 wrote MauMau lt; maumau307@ gt; writes: Shouldn't we lower the severity or avoiding those messages to server log? No. They are FATAL so far as the individual session is concerned. Possibly some documentation effort is needed here, but I don't think any change in the code behavior would be an improvement. 1. FATAL: the database system is starting up 2. FATAL: the database system is shutting down 3. FATAL: the database system is in recovery mode 4. FATAL: sorry, too many clients already Report these as FATAL to the client because the client wants to know the reason. But don't output them to server log because they are not necessary for DBAs (4 is subtle.) The notion that a DBA should not be allowed to find out how often #4 is happening is insane. Agreed #4 is definitely DBA territory. ISTM that instituting some level of categorization for messages would be helpful. Then logging and reporting frameworks would be able to identify and segregate the logs in whatever way they and the configuration deems appropriate. FATAL: [LOGON] too many clients already I'd make the category output disabled by default for a long while then eventually enabled by default but leave the ability to disable. Calls that do not supply a category get [N/A] output in category mode. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/RFC-Shouldn-t-we-remove-annoying-FATAL-messages-from-server-log-tp5781899p5781925.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
* David Johnston (pol...@yahoo.com) wrote: ISTM that instituting some level of categorization for messages would be helpful. Then logging and reporting frameworks would be able to identify and segregate the logs in whatever way they and the configuration deems appropriate. I've wanted to do that and have even discussed it with folks in the past, the trick is finding enough toit's, which is difficult when you start to look at the size of the task... Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
On 12/05/2013 10:21 AM, Stephen Frost wrote: * David Johnston (pol...@yahoo.com) wrote: ISTM that instituting some level of categorization for messages would be helpful. Then logging and reporting frameworks would be able to identify and segregate the logs in whatever way they and the configuration deems appropriate. I've wanted to do that and have even discussed it with folks in the past, the trick is finding enough toit's, which is difficult when you start to look at the size of the task... But ... if we set a firm policy on this, then we could gradually clean up the error messages piecemeal over the next couple of major versions. We could also make sure that any new features complied with the categorization policy. Right now, how to categorize errors is up to each individual patch author, which means that things are all over the place, and get worse with each new feature added. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
Josh Berkus j...@agliodbs.com writes: On 12/05/2013 10:21 AM, Stephen Frost wrote: But ... if we set a firm policy on this, then we could gradually clean up the error messages piecemeal over the next couple of major versions. We could also make sure that any new features complied with the categorization policy. Right now, how to categorize errors is up to each individual patch author, which means that things are all over the place, and get worse with each new feature added. I don't think there's that much randomness in is-it-an-ERROR-or-not. What I believe Stephen is talking about is a classification that simply doesn't exist today, namely something around how likely is the message to be of interest to a DBA as opposed to the client application. We currently compose messages almost entirely with the client in mind, and that's as it should be. But we could use some new decoration that's more DBA-oriented to help decide what goes into the postmaster log. Before we could get very far we'd need a better understanding than we have of what cases a DBA might be interested in. To take the specific example that started this thread, there wouldn't be a lot of value IMO in a classification like connection failure messages. I think the OP is probably right that those are often uninteresting --- but as I mentioned, too many clients might become interesting if he's wondering whether he needs to enlarge max_connections. Or password failure cases might become interesting if he starts to suspect breakin attempts. So I'd want to see a design that credibly covers those sorts of needs before we put any large effort into code changes. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
On 12/05/2013 10:46 AM, Tom Lane wrote: Before we could get very far we'd need a better understanding than we have of what cases a DBA might be interested in. To take the specific example that started this thread, there wouldn't be a lot of value IMO in a classification like connection failure messages. I think the OP is probably right that those are often uninteresting --- but as I mentioned, too many clients might become interesting if he's wondering whether he needs to enlarge max_connections. Or password failure cases might become interesting if he starts to suspect breakin attempts. So I'd want to see a design that credibly covers those sorts of needs before we put any large effort into code changes. Heck, I'd be happy just to have a class of messages which specifically means OMG, there's something wrong with the server, that is, a flag for messages which only occur when PostgreSQL encounters a bug, data corrpution, or platform error. Right now, I have to suss those out by regex. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?
Josh Berkus wrote: Heck, I'd be happy just to have a class of messages which specifically means OMG, there's something wrong with the server, that is, a flag for messages which only occur when PostgreSQL encounters a bug, data corrpution, or platform error. Right now, I have to suss those out by regex. My proposal was to have something separate from message severity (criticality). So the problems would continue to be reported as FATAL, ERROR or WARNING, but if they are just the result of something the user did wrong, then they get marked as non critical, but if there, say, a failure to flush xlog (which currently results in an ERROR), we could flag it as critical. Grepping the log for critical messages, regardless of severity, would result in actual action items for the DBA, without having to grep things out by regex. Otherwise you have to come up with a lot of messages each keeping the current behavior of abort the current transaction or not, or terminate the process or not. There was also the idea that this would be driven off SQLSTATE but this seems pretty unwieldy to me. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
