Re: [HACKERS] bgworker sigusr1 handler

2013-04-10 Thread Michael Paquier
On Wed, Apr 10, 2013 at 12:15 PM, Robert Haas robertmh...@gmail.com wrote: Just for fun, I implemented a toy background worker tonight using the new bgworker framework. Generally, it went well, and I'm pleased with the design of the new facility. However, I did notice one oddity. I

Re: [HACKERS] page 1 of relation global/11787 was uninitialized

2013-04-10 Thread Albe Laurenz
Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Afaik we don't have any debugging utility to dump the pg_filenode.map contents? Hardly need one ... od -t d4 $PGDATA/global/pg_filenode.map is readable enough, though it does leave you still having to map the numeric OIDs back to

Re: [HACKERS] lwlock contention with SSI

2013-04-10 Thread Dan Ports
On Tue, Apr 09, 2013 at 07:49:51PM -0400, Robert Haas wrote: These locks are all SSI-related and they're all really hot. Lock 28 is SerializableXactHashLock and lock 29 is SerializableFinishedListLock; both are acquired an order of magnitude more often than any non-SSI lock, and cause two

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Ants Aasma
On Wed, Apr 10, 2013 at 4:36 AM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-04-09 at 05:35 +0300, Ants Aasma wrote: And here you go. I decided to be verbose with the comments as it's easier to delete a comment to write one. I also left in a huge jumble of macros to calculate the contents

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Simon Riggs
On 10 April 2013 09:01, Ants Aasma a...@cybertec.at wrote: Using SIMD for WAL is not a requirement at all; I just thought it might be a nice benefit for non-checksum-enabled users in some later release. I think we should first deal with using it for page checksums and if future versions

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Amit Kapila
Sent: Wednesday, April 10, 2013 1:49 PM Dang Minh Huong wrote: To: Amit Kapila Subject: Re: [BUGS] replication_timeout not effective On Wednesday, April 10, 2013 1:49 PM Hi, Thank you for your soon reply. I'm trying to set the network timeout related parameters to terminate it. # i've

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Samrat Revagade
it's one of the reasons why a fresh base backup is required when starting old master as new standby? If yes, I agree with you. I've often heard the complaints about a backup when restarting new standby. That's really big problem. I think Fujii Masao is on the same page. In case of syncrep the

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Dang Minh Huong
Hi Amit, Thank you for your consideration. My project not allows to use 9.2 or 9.3. In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. So if it is solved in 9.3 i think there is a way to terminate it. I hope it is fixed in 9.1 soon Regards, 2013/04/10 18:33、Amit Kapila

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Samrat Revagade
(5) *The master then forces a write of the data page related to this transaction.* *Sorry, this is incorrect. Whenever the master writes the data page it checks that the WAL record is written in standby till that LSN. * * * While master is waiting to force a write (point 5) for this data page,

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Ants Aasma
On Wed, Apr 10, 2013 at 12:25 PM, Simon Riggs si...@2ndquadrant.com wrote: On 10 April 2013 09:01, Ants Aasma a...@cybertec.at wrote: Using SIMD for WAL is not a requirement at all; I just thought it might be a nice benefit for non-checksum-enabled users in some later release. I think we

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Amit Kapila
On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: (5) The master then forces a write of the data page related to this transaction. Sorry, this is incorrect. Whenever the master writes the data page it checks that the WAL record is written in standby till that LSN.  While master is

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Kyotaro HORIGUCHI
Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong kakalo...@gmail.com wrote: In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. So if it is solved in 9.3 i think there is a way to terminate it. I hope it is fixed in 9.1 soon Hmm. He said that, But in my

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Andres Freund
On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong kakalo...@gmail.com wrote: In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. So if it is solved in 9.3 i think there is a way to terminate it. I hope it

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Tom Lane
Amit Kapila amit.kap...@huawei.com writes: On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: Sorry, this is incorrect. Streaming replication continuous, master is not waiting, whenever the master writes the data page it checks that the WAL record is written in standby till that LSN.

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Andres Freund
On 2013-04-10 10:10:31 -0400, Tom Lane wrote: Amit Kapila amit.kap...@huawei.com writes: On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: Sorry, this is incorrect. Streaming replication continuous, master is not waiting, whenever the master writes the data page it checks that the

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Bruce Momjian
On Wed, Apr 10, 2013 at 01:15:12PM +0300, Ants Aasma wrote: This work needs to happen now, since once the checksum algorithm is set we won't easily be able to change it. The page checksum algorithm needs to be decided now, but WAL CRCs and full page writes can be changed in 9.4 and don't

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Shaun Thomas
On 04/10/2013 09:10 AM, Tom Lane wrote: IOW, I wouldn't consider skipping the rsync even if I had a feature like this. Totally. Out in the field, we consider the old database corrupt the moment we fail over. There is literally no way to verify the safety of any data along the broken chain,

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Dang Minh Huong
Thanks all, (2013/04/10 22:55), Andres Freund wrote: On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong kakalo...@gmail.com wrote: In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. So if it is solved in 9.3 i

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Andres Freund
On 2013-04-10 23:37:44 +0900, Dang Minh Huong wrote: Thanks all, (2013/04/10 22:55), Andres Freund wrote: On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong kakalo...@gmail.com wrote: In 9.3, it sounds replication_timeout is

[HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread
Hi all, I'd like to introduce myself to the dev community. I am Shuai Fan, a student from Dalian University of Technology, DLUT , for short, China. And I am interested in working with PostgreSQL project in GSOC2013. I'm interested in the idea Rewrite (add) pg_dump and pg_restore

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Dang Minh Huong
2013/04/10 23:44、Andres Freund and...@2ndquadrant.com のメッセージ: On 2013-04-10 23:37:44 +0900, Dang Minh Huong wrote: Thanks all, (2013/04/10 22:55), Andres Freund wrote: On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong

Re: [HACKERS] Problem with background worker

2013-04-10 Thread Alvaro Herrera
Marc Cousin escribió: On 20/03/2013 16:33, Alvaro Herrera wrote: Ah. The reason for this problem is that the statement start time (which also sets the transaction start time, when it's the first statement) is set by postgres.c, not the transaction-control functions in xact.c. So you'd need

Re: [HACKERS] Behaviour of bgworker with SIGHUP

2013-04-10 Thread Alvaro Herrera
Guillaume Lelarge wrote: worker_spi.naptime is the naptime between two checks. worker_spi.total_workers is the number of workers to launch at postmaster start time. The first one can change with a sighup, the last one obviously needs a restart. Many thanks. Pushed as

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Fujii Masao
On Wed, Apr 10, 2013 at 11:26 PM, Shaun Thomas stho...@optionshouse.com wrote: On 04/10/2013 09:10 AM, Tom Lane wrote: IOW, I wouldn't consider skipping the rsync even if I had a feature like this. Totally. Out in the field, we consider the old database corrupt the moment we fail over.

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Shaun Thomas
On 04/10/2013 11:40 AM, Fujii Masao wrote: Strange. If this is really true, shared disk failover solution is fundamentally broken because the standby needs to start up with the shared corrupted database at the failover. How so? Shared disk doesn't use replication. The point I was trying to

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Fujii Masao
On Wed, Apr 10, 2013 at 11:16 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-10 10:10:31 -0400, Tom Lane wrote: Amit Kapila amit.kap...@huawei.com writes: On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: Sorry, this is incorrect. Streaming replication continuous,

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Fujii Masao
On Thu, Apr 11, 2013 at 1:44 AM, Shaun Thomas stho...@optionshouse.com wrote: On 04/10/2013 11:40 AM, Fujii Masao wrote: Strange. If this is really true, shared disk failover solution is fundamentally broken because the standby needs to start up with the shared corrupted database at the

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Ants Aasma
On Wed, Apr 10, 2013 at 7:44 PM, Shaun Thomas stho...@optionshouse.com wrote: On 04/10/2013 11:40 AM, Fujii Masao wrote: Strange. If this is really true, shared disk failover solution is fundamentally broken because the standby needs to start up with the shared corrupted database at the

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Tom Lane
Ants Aasma a...@cybertec.at writes: We already rely on WAL-before-data to ensure correct recovery. What is proposed here is to slightly redefine it to require WAL to be replicated before it is considered to be flushed. This ensures that no data page on disk differs from the WAL that the slave

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Jeff Davis
On Wed, 2013-04-10 at 11:01 +0300, Ants Aasma wrote: I think we should first deal with using it for page checksums and if future versions want to reuse some of the code for WAL checksums then we can rearrange the code. Sounds good to me, although I expect we at least want any assembly to be in

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Boszormenyi Zoltan
2013-04-10 18:46 keltezéssel, Fujii Masao írta: On Wed, Apr 10, 2013 at 11:16 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-10 10:10:31 -0400, Tom Lane wrote: Amit Kapila amit.kap...@huawei.com writes: On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: Sorry, this is

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Andres Freund
On 2013-04-10 20:39:25 +0200, Boszormenyi Zoltan wrote: 2013-04-10 18:46 keltezéssel, Fujii Masao írta: On Wed, Apr 10, 2013 at 11:16 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-10 10:10:31 -0400, Tom Lane wrote: Amit Kapila amit.kap...@huawei.com writes: On Wednesday, April

[HACKERS] Re: [COMMITTERS] pgsql: Get rid of USE_WIDE_UPPER_LOWER dependency in trigram constructi

2013-04-10 Thread Stefan Kaltenbrunner
On 04/08/2013 10:11 AM, Dimitri Fontaine wrote: Tom Lane t...@sss.pgh.pa.us writes: If there is anybody still using Postgres on machines without wcstombs() or towlower(), and they have non-ASCII data indexed by pg_trgm, they'll need to REINDEX those indexes after pg_upgrade to 9.3, else

Re: [HACKERS] bgworker sigusr1 handler

2013-04-10 Thread Alvaro Herrera
Robert Haas escribió: Just for fun, I implemented a toy background worker tonight using the new bgworker framework. Generally, it went well, and I'm pleased with the design of the new facility. Thanks. However, I did notice one oddity. I initialized the worker flags like this:

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-04-10 Thread Alvaro Herrera
Michael Paquier escribió: Hi all, While playing with custom background workers, I noticed that postmaster does not notify its registered bgworkers if it receives SIGHUP, so you have to send a SIGHUP directly to the bgworker process to notify it. Signal handling is correctly done for SIGQUIT

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-04-10 Thread Alvaro Herrera
Michael Paquier escribió: Hi all, Please find attached a simple example of bgworker that logs a message each time a SIGTERM or SIGHUP signal is received by it: - hello signal: processed SIGHUP when SIGHUP is handled by my example - hello signal: processed SIGTERM when SIGTERM is handled by

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Simon Riggs
On 10 April 2013 11:15, Ants Aasma a...@cybertec.at wrote: * We might even be able to calculate CRC32 checksum for normal WAL records, and use Ants' checksum for full page writes (only). So checking WAL checksum would then be to confirm header passes CRC32 and then re-check the Ants

Re: [HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread Peter Eisentraut
On 4/10/13 10:54 AM, ˧ wrote: I'm interested in the idea Rewrite (add) pg_dump and pg_restore utilities as libraries (.so, .dll .dylib). The pg_dump code is a giant mess, and refactoring it as a library is perhaps not a project for a new hacker. Independent of that, I think the first

Re: [HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread Alvaro Herrera
Peter Eisentraut wrote: I think the main uses cases mentioned in connection with this idea are usually in the direction of finer-grained control over what gets dumped and how. But making pg_dump into a library would not necessarily address that. There's also the matter of embedding pg_dump

Re: [HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread Hannu Krosing
On 04/10/2013 11:02 PM, Alvaro Herrera wrote: Peter Eisentraut wrote: I think the main uses cases mentioned in connection with this idea are usually in the direction of finer-grained control over what gets dumped and how. But making pg_dump into a library would not necessarily address that.

Re: [HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread Alvaro Herrera
Hannu Krosing wrote: On 04/10/2013 11:02 PM, Alvaro Herrera wrote: Peter Eisentraut wrote: I think the main uses cases mentioned in connection with this idea are usually in the direction of finer-grained control over what gets dumped and how. But making pg_dump into a library would not

Re: [HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Hannu Krosing wrote: Natural solution to this seems to move most of pg_dump functionality into backend as functions, so we have pg_dump_xxx() for everything we want to dump plus a topological sort function for getting the objects in right order.

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-10 Thread Robert Haas
On Sat, Apr 6, 2013 at 10:44 AM, Andres Freund and...@2ndquadrant.com wrote: I feel pretty strongly that we shouldn't add any such complications to XLogInsert() itself, its complicated enough already and it should be made simpler, not more complicated. +1, emphatically. XLogInsert is a really

Re: [HACKERS] [sepgsql 3/3] Add db_procedure:execute permission checks

2013-04-10 Thread Alvaro Herrera
Kohei KaiGai wrote: This patch adds sepgsql support for permission checks almost equivalent to the existing FUNCTION EXECUTE privilege. While skimming this patch I noticed that you're using getObjectDescription() as the audit_name of objects. This may be a bit unstable, for example consider

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-04-10 Thread Michael Paquier
Thanks for committing the fix! On Thu, Apr 11, 2013 at 4:11 AM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Michael Paquier escribió: Hi all, Please find attached a simple example of bgworker that logs a message each time a SIGTERM or SIGHUP signal is received by it: - hello

[HACKERS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Gurjeet Singh
If I'm reading the code right [1], this GUC does not actually *synchronize* the scans, but instead just makes sure that a new scan starts from a block that was reported by some other backend performing a scan on the same relation. Since the backends scanning the relation may be processing the

Re: [HACKERS] [DOCS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Tom Lane
Gurjeet Singh gurj...@singh.im writes: If I'm reading the code right [1], this GUC does not actually *synchronize* the scans, but instead just makes sure that a new scan starts from a block that was reported by some other backend performing a scan on the same relation. Well, that's the only

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Jeff Davis
On Wed, 2013-04-10 at 20:17 +0100, Simon Riggs wrote: OK, so we have a single combined calculate a checksum for a block function. That uses Jeff's zeroing trick and Ants' bulk-oriented performance optimization. For buffer checksums we simply calculate for the block. Sounds good. For

[HACKERS] ObjectClass/ObjectType mixup

2013-04-10 Thread Peter Eisentraut
src/backend/catalog/dependency.c:213: EventTriggerSupportsObjectType(getObjectClass(thisobj))) src/backend/commands/event_trigger.c:1014: Assert(EventTriggerSupportsObjectType(getObjectClass(object))); getObjectClass() returns type ObjectClass, but

Re: [HACKERS] [DOCS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Gurjeet Singh
On Wed, Apr 10, 2013 at 11:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: Gurjeet Singh gurj...@singh.im writes: If I'm reading the code right [1], this GUC does not actually *synchronize* the scans, but instead just makes sure that a new scan starts from a block that was reported by some

Re: [HACKERS] [DOCS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Tom Lane
Gurjeet Singh gurj...@singh.im writes: On Wed, Apr 10, 2013 at 11:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: The point you're missing is that the synchronization is self-enforcing: Let's consider a pathological case where a scan is performed by a user controlled cursor, whose scan speed

Re: [HACKERS] [GSOC] questions about idea rewrite pg_dump as library

2013-04-10 Thread Pavel Golub
Hello, 帅. You wrote: 帅 Hi all, 帅 I'd like to introduce myself to the dev community. I am Shuai 帅 Fan, a student from Dalian University of Technology, DLUT , for 帅 short, China. And I am interested in working with PostgreSQL project in GSOC2013. 帅 I'm interested in the idea Rewrite