Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 07:09, Tom Lane wrote: There has been regular griping in this list about our dependence on SysV shared memory, but not so much about SysV semaphores, even though the latter cause their fair share of issues; as seen for example in buildfarm member spoonbill's recent string of

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Sandro Santilli
On Sat, May 26, 2012 at 01:24:26PM +0200, Florian Pflug wrote: On May26, 2012, at 12:40 , Simon Riggs wrote: On 25 May 2012 17:34, Tom Lane t...@sss.pgh.pa.us wrote: I assume that the geos::util::Interrupt::request() call sets a flag somewhere that's going to be periodically checked in

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Gurjeet Singh
On Wed, Jun 6, 2012 at 1:55 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 6, 2012 at 10:38 AM, Tom Lane t...@sss.pgh.pa.us wrote: Well, that's what I wanted to discuss before Honza starts coding. It's not obvious that there are any use-cases for more than two. It's also not

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 6 June 2012 20:08, Tom Lane t...@sss.pgh.pa.us wrote: In commit 18fb9d8d21a28caddb72c7ffbdd7b96d52ff9724, Simon modified the rule for when to skip checkpoints on the grounds that not enough activity has happened since the last one.  However, that commit left the comment block about it in a

Re: [HACKERS] Time for pgindent run?

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Robert Haas wrote: On Tue, Jun 5, 2012 at 10:25 AM, Bruce Momjian br...@momjian.usjavascript:; wrote: On Tue, Jun 05, 2012 at 10:21:14AM -0400, Tom Lane wrote: Bruce Momjian br...@momjian.us javascript:; writes: Is everyone ready for me to run pgindent? We

[HACKERS] SP-GIST docs and examples

2012-06-07 Thread Simon Riggs
I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST simply suggest people read the code, which is way below our normal standard of documentation. CREATE INDEX contains no examples involving SP-GIST indexes. It seems likely that

Re: [HACKERS] \conninfo and SSL

2012-06-07 Thread Magnus Hagander
On Wednesday, June 6, 2012, Alvaro Herrera wrote: Excerpts from Robert Haas's message of mié jun 06 14:45:46 -0400 2012: On Sun, Jun 3, 2012 at 5:30 AM, Alastair Turner b...@ctrlf5.co.zajavascript:; wrote: A one-line change adds the SSL info on its own line like -- You are

Re: [HACKERS] pg_receivexlog and feedback message

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Fujii Masao wrote: On Thu, Jun 7, 2012 at 5:05 AM, Magnus Hagander mag...@hagander.netjavascript:; wrote: On Wed, Jun 6, 2012 at 8:26 PM, Fujii Masao masao.fu...@gmail.comjavascript:; wrote: On Tue, Jun 5, 2012 at 11:44 PM, Magnus Hagander

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Simon Riggs wrote: I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST simply suggest people read the code, which is way below our normal standard of documentation. It's the same we have for normal

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Florian Pflug
On Jun7, 2012, at 10:20 , Sandro Santilli wrote: On Sat, May 26, 2012 at 01:24:26PM +0200, Florian Pflug wrote: On May26, 2012, at 12:40 , Simon Riggs wrote: On 25 May 2012 17:34, Tom Lane t...@sss.pgh.pa.us wrote: I assume that the geos::util::Interrupt::request() call sets a flag somewhere

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Sandro Santilli
On Thu, Jun 07, 2012 at 12:00:09PM +0200, Florian Pflug wrote: On Jun7, 2012, at 10:20 , Sandro Santilli wrote: In that case I can understand Tom's advice about providing a callback, and then I would only need to perform the events flushing part of from within the callback, and only for

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Florian Pflug
On Jun7, 2012, at 12:08 , Sandro Santilli wrote: On Thu, Jun 07, 2012 at 12:00:09PM +0200, Florian Pflug wrote: On Jun7, 2012, at 10:20 , Sandro Santilli wrote: In that case I can understand Tom's advice about providing a callback, and then I would only need to perform the events flushing

Re: [HACKERS] pg_receivexlog and feedback message

2012-06-07 Thread Fujii Masao
On Thu, Jun 7, 2012 at 6:25 PM, Magnus Hagander mag...@hagander.net wrote: On Thursday, June 7, 2012, Fujii Masao wrote: On Thu, Jun 7, 2012 at 5:05 AM, Magnus Hagander mag...@hagander.net wrote: On Wed, Jun 6, 2012 at 8:26 PM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jun 5, 2012

Re: [HACKERS] pg_receivexlog and feedback message

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Fujii Masao wrote: On Thu, Jun 7, 2012 at 6:25 PM, Magnus Hagander mag...@hagander.net wrote: On Thursday, June 7, 2012, Fujii Masao wrote: On Thu, Jun 7, 2012 at 5:05 AM, Magnus Hagander mag...@hagander.net wrote: On Wed, Jun 6, 2012 at 8:26 PM, Fujii

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 5:19 AM, Simon Riggs si...@2ndquadrant.com wrote: I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST simply suggest people read the code, which is way below our normal standard of documentation. I

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Thom Brown
On 7 June 2012 12:52, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 7, 2012 at 5:19 AM, Simon Riggs si...@2ndquadrant.com wrote: I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST simply suggest people read the code,

Re: [HACKERS] creating objects in pg_catalog

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 5:21 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: rhaas=# create table pg_catalog.tom (a int); ERROR:  permission denied to create pg_catalog.tom The offending error check is in heap_create(), and based on what you're saying here it

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Oleg Bartunov
On Thu, 7 Jun 2012, Thom Brown wrote: On 7 June 2012 12:52, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 7, 2012 at 5:19 AM, Simon Riggs si...@2ndquadrant.com wrote: I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Sergey Koposov
On Tue, 5 Jun 2012, Simon Riggs wrote: Sounds less good and we'd need reasonable proof it actually did anything useful without being dangerous. Doing an initial unlocked test speeds things up another 2.69 fold (on top of 3.55 for your patch) for me, with 1GB of shared buffers.  That seems

Re: [HACKERS] Early hint bit setting

2012-06-07 Thread Merlin Moncure
On Wed, Jun 6, 2012 at 5:41 PM, Jim Nasby j...@nasby.net wrote: On 5/30/12 4:42 PM, Ants Aasma wrote: I was thinking about what is the earliest time where we could set hint bits. This would be just after the commit has been made visible. Except that's only true when there are no other

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 3:07 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. For a short while I thought there would be an issue with heap_delete and IOS because the deleting transaction can commit without any barriers happening on the IOS side. But that only seems to be possible with non

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 03:20:50 PM Robert Haas wrote: On Wed, Jun 6, 2012 at 3:07 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. For a short while I thought there would be an issue with heap_delete and IOS because the deleting transaction can commit without any barriers happening

[HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
When I worked on the XLogInsert scaling patch, it became apparent that some changes to the WAL format would make it a lot easier. So for 9.3, I'd like to do some refactoring: 1. Use a 64-bit integer instead of the two-variable log/seg representation, for identifying a WAL segment. This has no

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Sergey Koposov kopo...@ast.cam.ac.uk writes: On Tue, 5 Jun 2012, Simon Riggs wrote: Sounds less good and we'd need reasonable proof it actually did anything useful without being dangerous. Doing an initial unlocked test speeds things up another 2.69 fold (on top of 3.55 for your patch) for

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: there is no guarantee that we'll manage to reach a database state that is consistent with data already flushed out to disk during the last checkpoint. Robert Haas robertmh...@gmail.com wrote: I know of real customers who would have suffered real data

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 6:46 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Actually, it looks like there is an extremely simple way to handle this, which is to move the call of LogStandbySnapshot (which generates the WAL record in question) to before the checkpoint's REDO pointer is set, but

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 07.06.2012 07:09, Tom Lane wrote: It strikes me that we have recently put together an independent but just about equivalent waiting mechanism in the form of latches. And not only that, but there's already a latch for each

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: When I worked on the XLogInsert scaling patch, it became apparent that some changes to the WAL format would make it a lot easier. So for 9.3, I'd like to do some refactoring: 1. Use a 64-bit integer instead of the two-variable

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund and...@2ndquadrant.com wrote: Well, one, commits are irrelevant; the page ceases to be all-visible as soon as the delete happens. Its not irrelevant for the code as it stands if non-mvcc snapshots were allowed. Unless I miss something, even

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: When I worked on the XLogInsert scaling patch, it became apparent that some changes to the WAL format would make it a lot easier. So for 9.3, I'd like to do some refactoring: 1. Use a 64-bit integer instead of the two-variable

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 04:27:32 PM Robert Haas wrote: On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund and...@2ndquadrant.com wrote: Proposed patch attached. This adds some more comments in various places, and implements your suggestion of retesting the visibility-map bit when we detect

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 10:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: I would be more inclined to look into OS-specific primitives such as futexes on Linux.  (No idea whether those actually would be suitable, just pointing out that they exist.)  Our semaphore-based API was always both

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 17:18, Andres Freund wrote: On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: 3. Move the only field, xl_rem_len, from the continuation record header straight to the xlog page header, eliminating XLogContRecord altogether. This makes it easier to calculate in

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Honza Horak
On 06/06/2012 04:50 PM, Andres Freund wrote: On Wednesday, June 06, 2012 04:38:42 PM Tom Lane wrote: You might think we should design this exactly like the TCP-socket multiple-listen-addresses case, ie just have a config variable containing a list of directory names. The sticking point there

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Jun 6, 2012 at 6:46 PM, Tom Lane t...@sss.pgh.pa.us wrote: If we don't like that, I can think of a couple of other ways to get there, but they have their own downsides: * Instead of trying to detect after-the-fact whether any concurrent WAL

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost certainly set back the other work that we have brewing for 9.3. So this is a

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Tom Lane
Honza Horak hho...@redhat.com writes: On 06/06/2012 04:50 PM, Andres Freund wrote: I wonder if the whole issue doesn't require libpq to also try multiple hardcoded socket locations. I guess so. I don't really want to go there. Some use cases have been shown in this thread for having a

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
Hi, On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: Honza Horak hho...@redhat.com writes: On 06/06/2012 04:50 PM, Andres Freund wrote: I wonder if the whole issue doesn't require libpq to also try multiple hardcoded socket locations. I guess so. I don't really want to go there.

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:59, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane t...@sss.pgh.pa.us wrote: there is no guarantee that we'll manage to reach a database state that is consistent with data already flushed out to disk during the last checkpoint. Robert Haas robertmh...@gmail.com

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Magnus Hagander
On Thu, Jun 7, 2012 at 5:56 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: These changes will help the XLogInsert scaling patch ...and as I'm

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:02:12 PM Magnus Hagander wrote: On Thu, Jun 7, 2012 at 5:56 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:04 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, June 07, 2012 04:27:32 PM Robert Haas wrote: On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund and...@2ndquadrant.com wrote: Proposed patch attached.  This adds some more comments in various places, and

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 18:51, Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost certainly set back the other

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:57 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: Honza Horak hho...@redhat.com writes: On 06/06/2012 04:50 PM, Andres Freund wrote: I wonder if the whole issue doesn't require libpq to also try multiple

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:08:34 PM Robert Haas wrote: On Thu, Jun 7, 2012 at 11:04 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, June 07, 2012 04:27:32 PM Robert Haas wrote: On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund and...@2ndquadrant.com wrote: Proposed patch

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: Honza Horak hho...@redhat.com writes: On 06/06/2012 04:50 PM, Andres Freund wrote: I wonder if the whole issue doesn't require libpq to also try multiple hardcoded socket locations. I don't

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:51 AM, Simon Riggs si...@2ndquadrant.com wrote: So this is a very large curve ball you're throwing there. This is not exactly unexpected. At least the first two of these items were previous discussed in the context of the XLOG scaling patch, many months ago. It

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:20:32 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: Honza Horak hho...@redhat.com writes: On 06/06/2012 04:50 PM, Andres Freund wrote: I wonder if the whole issue doesn't require libpq

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 12:19 PM, Andres Freund and...@2ndquadrant.com wrote: You could do a visibilitymap_pin outside, but I don't really see the point as the page is already locked. There might be some slight benefit in doing so in multi_insert but that would be more complicated. And of

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:56, Tom Lane t...@sss.pgh.pa.us wrote: Sergey Koposov kopo...@ast.cam.ac.uk writes: On Tue, 5 Jun 2012, Simon Riggs wrote: Sounds less good and we'd need reasonable proof it actually did anything useful without being dangerous. Doing an initial unlocked test speeds things

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Robert Haas robertmh...@gmail.com wrote: I know of real customers who would have suffered real data loss had this code been present in the server version they were using. If that is the concern, then its a one line fix to add the missing clog flush.

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:23:58 PM Robert Haas wrote: On Thu, Jun 7, 2012 at 12:19 PM, Andres Freund and...@2ndquadrant.com wrote: You could do a visibilitymap_pin outside, but I don't really see the point as the page is already locked. There might be some slight benefit in doing so in

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 7 June 2012 14:56, Tom Lane t...@sss.pgh.pa.us wrote: Say what?  That's a performance result and proves not a damn thing about safety. Of course not. Based on the rationale explained in the code comments in the patch, it seems like a reasonable

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 05:35:11 PM Heikki Linnakangas wrote: On 07.06.2012 17:18, Andres Freund wrote: On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: 3. Move the only field, xl_rem_len, from the continuation record header straight to the xlog page header, eliminating

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Updated patch attached. The comments need a pass of copy-editing, eg here and here: + * so somebody else could be change the bit just after we look at it. In fact, ^^^ + * got cleared after we checked it

Re: [HACKERS] page is not marked all-visible warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 12:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Updated patch attached. The comments need a pass of copy-editing, eg here and here: + * so somebody else could be change the bit just after we look at it.  In fact,                

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:27, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Robert Haas robertmh...@gmail.com wrote: I know of real customers who would have suffered real data loss had this code been present in the server version they were using. If that is the

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: dance. If the record can be smeared over two pages there is no point in storing it aligned. I think this is not true. The value of requiring alignment is that you can read the record-length field without first having to copy it somewhere. In

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:34, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 7 June 2012 14:56, Tom Lane t...@sss.pgh.pa.us wrote: Say what?  That's a performance result and proves not a damn thing about safety. Of course not. Based on the rationale explained in

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:53:58 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: dance. If the record can be smeared over two pages there is no point in storing it aligned. I think this is not true. The value of requiring alignment is that you can read the

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs si...@2ndquadrant.com wrote: Clearly, delaying checkpoint indefinitely would be a high risk choice. But they won't be delayed indefinitely, since changes cause WAL records to be written and that would soon cause another checkpoint. But that's

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:12, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 07.06.2012 18:51, Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: These changes will help the XLogInsert scaling patch ...and as I'm sure you're

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 7 June 2012 17:27, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: If that is the concern, then its a one line fix to add the missing clog flush. To where, and what performance impact will that have? To the point

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 07:03:32 PM Simon Riggs wrote: On 7 June 2012 17:12, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 07.06.2012 18:51, Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: These changes

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Anything changing filenames will break every HA config anybody has anywhere. It will impact our scripts related to backup and archiving, but I think we're talking about two or three staff days to cover it in our shop. We should definitely make sure

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: ... It's better to have a few unnecessary checkpoints than to risk losing somebody's data, especially since the unnecessary checkpoints only happen with wal_level=hot_standby, but the data loss risk exists for everyone. Yeah, that's another point

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Stefan Kaltenbrunner
On 06/07/2012 06:09 AM, Tom Lane wrote: There has been regular griping in this list about our dependence on SysV shared memory, but not so much about SysV semaphores, even though the latter cause their fair share of issues; as seen for example in buildfarm member spoonbill's recent string of

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: But if you're just using regexp matching against pathnames, your tool will be just fine. Do your tools actually rely on the occasional absence of a file in what would otherwise be the usual sequence of files? To save snapshot backups for the long

Re: [HACKERS] Early hint bit setting

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 6:41 PM, Jim Nasby j...@nasby.net wrote: Except that's only true when there are no other transactions running. That's been one of the big sticking points about trying to proactively set hint bits; in a real system you're not going to gain very much unless you wait a

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 1:40 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: But if you're just using regexp matching against pathnames, your tool will be just fine.  Do your tools actually rely on the occasional absence of a file in what would

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 1:15 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Simon Riggs si...@2ndquadrant.com wrote: Anything changing filenames will break every HA config anybody has anywhere. It will impact our scripts related to backup and archiving, but I think we're talking about

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-06-07 Thread Simon Riggs
On 6 June 2012 20:11, Andres Freund and...@2ndquadrant.com wrote: On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: Hi, On Monday, May 28, 2012 07:11:53 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Does anybody have a better idea than to either call

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Anything changing filenames will break every HA config anybody has anywhere. This seems like nonsense to me. How many external scripts are likely to know that we skip the FF page? There might be some, but not many. So you can pretty much kiss

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 19:52, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Anything changing filenames will break every HA config anybody has anywhere. This seems like nonsense to me.  How many external scripts are likely to know that we skip the FF page?  There might

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 7 June 2012 19:52, Tom Lane t...@sss.pgh.pa.us wrote: This seems like nonsense to me.  How many external scripts are likely to know that we skip the FF page?  There might be some, but not many. If that is the only change in filenames, then all is

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 7 June 2012 17:34, Tom Lane t...@sss.pgh.pa.us wrote: Oh, I must be confused about which patch we are talking about --- I thought this was in reference to some of the WIP ideas that were being thrown about with respect to using lock-free access

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 08:41:23 PM Simon Riggs wrote: On 6 June 2012 20:11, Andres Freund and...@2ndquadrant.com wrote: On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: Hi, On Monday, May 28, 2012 07:11:53 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes:

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: On 30 May 2012 12:10, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Also, I wonder if DropRelFileNodeBuffers() could scan the pool without grabbing the spinlocks on every buffer? It could do an unlocked test first, and only grab the

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
I wrote: Simon Riggs si...@2ndquadrant.com writes: Both of these, as attached up thread. Simon's patch - dropallforks.v1.patch Jeff's patch - DropRelFileNodeBuffers_unlock_v1.patch (needs a little tidyup) OK, will take a look. I didn't like dropallforks.v1.patch at all as presented, for

[HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Tom Lane
I am pleased to announce that Kevin Grittner has accepted the core committee's invitation to become our newest committer. As you all know, Kevin's done a good deal of work on the project over the past couple of years. We judged that he has the requisite skills, dedication to the project, and a

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Michael Glaesemann
On Jun 7, 2012, at 18:15, Tom Lane wrote: I am pleased to announce that Kevin Grittner has accepted the core committee's invitation to become our newest committer. As you all know, Kevin's done a good deal of work on the project over the past couple of years. We judged that he has the

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Thom Brown
On 7 June 2012 23:15, Tom Lane t...@sss.pgh.pa.us wrote: I am pleased to announce that Kevin Grittner has accepted the core committee's invitation to become our newest committer.  As you all know, Kevin's done a good deal of work on the project over the past couple of years.  We judged that he

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Peter Geoghegan
On 7 June 2012 23:15, Tom Lane t...@sss.pgh.pa.us wrote:  Please join me in welcoming him aboard. Congratulations, Kevin. -- Peter Geoghegan       http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] WIP: parameterized function scan

2012-06-07 Thread Antonin Houska
On 05/24/2012 12:46 AM, Tom Lane wrote: Well, it's not per spec: what you did accepts queries that are invalid per spec and are very likely to be errors rather than intentional invocations of the LATERAL facility. This might be all right for I think I saw queries where function is joined with

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Peter Geoghegan
On 7 June 2012 23:40, Peter Geoghegan pe...@2ndquadrant.com wrote: On 7 June 2012 23:15, Tom Lane t...@sss.pgh.pa.us wrote:  Please join me in welcoming him aboard. Congratulations, Kevin. Idle thought for the web team: Now might be a good time to take down the blurb on .org in which Kevin

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Jaime Casanova
On Thu, Jun 7, 2012 at 5:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Please join me in welcoming him aboard. Congratulations Kevin -- Jaime Casanova         www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación -- Sent via pgsql-hackers mailing list

[HACKERS] log_newpage header comment

2012-06-07 Thread Robert Haas
It seems that in implementing ginbuildempty(), I falsified the first note in the header comment for log_newpage(): * Note: all current callers build pages in private memory and write them * directly to smgr, rather than using bufmgr. Therefore there is no need * to pass a buffer ID to

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 7:09 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Thu, Jun 7, 2012 at 5:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Please join me in welcoming him aboard. Congratulations Kevin +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-06-07 Thread Simon Riggs
On 7 June 2012 21:08, Andres Freund and...@2ndquadrant.com wrote: Moved the wakeup to a logical place outside a critical section. Hm. I don't really like the way you implemented that. While it reduces the likelihood quite a bit it will still miss wakeups if an XLogInsert pushes out the data

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Simon Riggs
On 7 June 2012 22:54, Tom Lane t...@sss.pgh.pa.us wrote: I thought it would be a lot safer and probably a little bit quicker if we just split DropRelFileNodeBuffers into two routines, one for the specific-fork case and one for the all-forks case; and then the same for its main caller

Re: [HACKERS] log_newpage header comment

2012-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: It seems that in implementing ginbuildempty(), I falsified the first note in the header comment for log_newpage(): * Note: all current callers build pages in private memory and write them * directly to smgr, rather than using bufmgr. Therefore

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Sergey Koposov
On Thu, 7 Jun 2012, Tom Lane wrote: I extended the patch to also cover DropDatabaseBuffers, FlushRelationBuffers, and FlushDatabaseBuffers, which have got the exact same type of full-pool scan loop, and committed it. Thanks everybody for the patches/commits. Sergey

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 7 June 2012 18:03, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs si...@2ndquadrant.com wrote: Clearly, delaying checkpoint indefinitely would be a high risk choice. But they won't be delayed indefinitely, since changes cause WAL records to be written

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Peter Geoghegan
On 7 June 2012 18:03, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs si...@2ndquadrant.com wrote: Clearly, delaying checkpoint indefinitely would be a high risk choice. But they won't be delayed indefinitely, since changes cause WAL records to be written

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Peter Geoghegan pe...@2ndquadrant.com writes: On 7 June 2012 18:03, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs si...@2ndquadrant.com wrote: Clearly, delaying checkpoint indefinitely would be a high risk choice. But they won't be delayed indefinitely,