Re: [HACKERS] What exactly is our CRC algorithm?

2014-10-08 Thread Heikki Linnakangas
On 10/09/2014 01:23 AM, Gavin Flower wrote: On 09/10/14 10:13, Andres Freund wrote: If we're switching to a saner computation, we should imo also switch to a better polynom - CRC-32C has better error detection capabilities than CRC32 and is available in hardware. As we're paying the price pf bre

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread Heikki Linnakangas
On 10/09/2014 07:47 AM, furu...@pm.nttdata.co.jp wrote: If we remove --fsync-interval, resoponse time to user will not be delay. Although, fsync will be executed multiple times in a short period. And there is no way to solve the problem without --fsync-interval, what should we do about it? I'm

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 10:49 PM, Simon Riggs wrote: >> Do you really expect me to do major work on some aspect of the syntax >> like that, given, as you say, that nobody explicitly agreed with me >> (and only you disagreed with me)? The only remark I heard on that was >> from you (you'd prefer to

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Michael Paquier
On Wed, Oct 8, 2014 at 10:00 PM, Michael Paquier wrote: > The additional process at promotion sounds like a good idea, I'll try to > get a patch done tomorrow. This would result as well in removing the > XLogArchiveForceDone stuff. Either way, not that I have been able to > reproduce the problem

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Simon Riggs
On 8 October 2014 23:24, Peter Geoghegan wrote: >> Also, it would be useful to hear that your're going to do something >> about the references to rows using conflicting(), since nobody has >> agreed with you there. Or hopefully even that you've listened and >> implemented something differently al

Re: [HACKERS] Escaping from blocked send() reprised.

2014-10-08 Thread Kyotaro HORIGUCHI
Hello, simplly inhibit set retry flag when ProcDiePending in my_sock_write seems enough. But it returns with SSL_ERROR_SYSCALL not SSL_ERROR_WANT_WRITE so I modified the patch 4 as the attached patch. Finally, the attached patch works as expected with Andres's patch 1-3. regards, -- Kyotaro Ho

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread furuyao
> What set of options would you pass if you want to use it as a > synchronous standby? And if you don't? Could we just have a single > >> "--synchronous" > flag, instead of -F and --reply-fsync? > >>> > >>> If you set "synchronous_commit" as "remote_write", the options would > >> be

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Amit Kapila
On Thu, Oct 9, 2014 at 4:02 AM, Andres Freund wrote: > > > /* > > + * Arrange to remove a dynamic shared memory mapping at cleanup time. > > + * > > + * dsm_keep_mapping() can be used to preserve a mapping for the entire > > + * lifetime of a process; this function reverses that decision, making

Re: [HACKERS] Deferring some AtStart* allocations?

2014-10-08 Thread Amit Kapila
On Wed, Oct 8, 2014 at 11:22 PM, Robert Haas wrote: > > On Sun, Jun 29, 2014 at 9:12 PM, Tom Lane wrote: > > Meh. Even "SELECT 1" is going to be doing *far* more pallocs than that to > > get through raw parsing, parse analysis, planning, and execution startup. > > If you can find a few hundred p

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-10-08 Thread Etsuro Fujita
(2014/10/08 22:51), Fujii Masao wrote: On Wed, Sep 24, 2014 at 11:10 AM, Etsuro Fujita wrote: On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita wrote: PENDING_LIST_CLEANUP_SIZE and work_mem, for this setting. Woul

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 7:04 PM, Marti Raudsepp wrote: > Oops, I missed "for each row" here. Note that I have an open item for what to do about statement level triggers here: https://wiki.postgresql.org/wiki/UPSERT I'm not satisfied with the current behavior. It needs more thought. But I thin

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Marti Raudsepp
On Thu, Oct 9, 2014 at 4:56 AM, Marti Raudsepp wrote: > create trigger ev1 before insert on evt_type execute procedure ins(); > create trigger ev2 before update on evt_type execute procedure upd(); > create trigger ev3 after insert on evt_type execute procedure ins(); > create trigger ev4 after up

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Marti Raudsepp
On Thu, Oct 9, 2014 at 4:25 AM, Peter Geoghegan wrote: > On Wed, Oct 8, 2014 at 6:12 PM, Marti Raudsepp wrote: >> Skipping >> BEFORE UPDATE entirely seems to violate POLA. > Good thing that the patch doesn't do that, then. I clearly documented > this in a few places, including: > http://postgres

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 6:12 PM, Marti Raudsepp wrote: > Oh, one more consideration: I believe you will run into the same issue > if you want to implement BEFORE UPDATE triggers in any form. Skipping > BEFORE UPDATE entirely seems to violate POLA. Good thing that the patch doesn't do that, then. I

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Marti Raudsepp
On Thu, Oct 9, 2014 at 3:47 AM, Peter Geoghegan wrote: > On Wed, Oct 8, 2014 at 5:37 PM, Marti Raudsepp wrote: >> Only in case the trigger changes *key* columns necessary for atomicity >> (i.e. from the WITHIN index). Other columns are fair game. The >> restriction seems justifiable to me: it's u

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 5:37 PM, Marti Raudsepp wrote: > Only in case the trigger changes *key* columns necessary for atomicity > (i.e. from the WITHIN index). Other columns are fair game. The > restriction seems justifiable to me: it's unreasonable to be atomic > with respect to values that change

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Marti Raudsepp
On Thu, Oct 9, 2014 at 2:46 AM, Peter Geoghegan wrote: >> Indeed, the current behavior breaks even the canonical "keep track of >> how many posts are in a thread" trigger example > use an AFTER trigger for this kind of thing, and all of these > issues go away. In the latest patches from CommitFes

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-08 Thread Fabrízio de Royes Mello
On Wed, Oct 8, 2014 at 2:55 AM, David Fetter wrote: > > On Wed, Oct 08, 2014 at 12:41:46AM -0300, Fabrízio de Royes Mello wrote: > > On Wed, Oct 8, 2014 at 12:36 AM, Josh Berkus wrote: > > > > > > On 10/07/2014 09:44 AM, Fabrízio de Royes Mello wrote: > > > > We can think in a mechanism to create

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 2:04 PM, Andres Freund wrote: > So, what makes it work for me (among other unrelated stuff) seems to be > the following in .gdbinit, defineing away some things that gdb doesn't > handle: > macro define __builtin_offsetof(T, F) ((int) &(((T *) 0)->F)) > macro define __extensi

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 4:30 PM, Marti Raudsepp wrote: > On Thu, Oct 9, 2014 at 1:51 AM, Peter Geoghegan wrote: >> On Wed, Oct 8, 2014 at 2:01 PM, Kevin Grittner wrote: >>> Although the last go-around does suggest that there is at least one >>> point of difference on the semantics. You seem to w

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-08 Thread Ian Barwick
On 14/10/09 7:06, Stephen Frost wrote: > * Fabrízio de Royes Mello (fabriziome...@gmail.com) wrote: >> On Tue, Oct 7, 2014 at 1:24 PM, Simon Riggs wrote: >>> I hope we can get pgAudit in as a module for 9.5. I also hope that it >>> will stimulate the requirements/funding of further work in this ar

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Marti Raudsepp
On Thu, Oct 9, 2014 at 1:51 AM, Peter Geoghegan wrote: > On Wed, Oct 8, 2014 at 2:01 PM, Kevin Grittner wrote: >> Although the last go-around does suggest that there is at least one >> point of difference on the semantics. You seem to want to fire the >> BEFORE INSERT triggers before determining

Re: [HACKERS] Add CREATE support to event triggers

2014-10-08 Thread Michael Paquier
On Thu, Oct 9, 2014 at 6:26 AM, Alvaro Herrera wrote: > About patch 1, which I just pushed, there were two reviews: > > Andres Freund wrote: > > On 2014-09-25 18:59:31 -0300, Alvaro Herrera wrote: > > > diff --git a/src/include/utils/ruleutils.h > b/src/include/utils/ruleutils.h > > > new file mo

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Michael Paquier
On Wed, Oct 8, 2014 at 11:38 PM, Heikki Linnakangas wrote: > On 10/08/2014 04:59 PM, Fujii Masao wrote: > >> On Wed, Oct 8, 2014 at 6:54 PM, Heikki Linnakangas >> wrote: >> >>> Instead of creating any .done files during recovery, we could scan >>> pg_xlog >>> at promotion, and create a .done fil

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 2:01 PM, Kevin Grittner wrote: > Although the last go-around does suggest that there is at least one > point of difference on the semantics. You seem to want to fire the > BEFORE INSERT triggers before determining whether this will be an > INSERT or an UPDATE. That seems l

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
On 2014-10-08 15:03:01 -0400, Robert Haas wrote: > #1 - Just committed, per discussion last week. > #2 - No comments, possibly because it's pretty boring. fine with me. > #3 - Most people seem happy with v2, modulo the above renaming request. If you do it without the error handling stuff, I'm go

Re: [HACKERS] delta relations in AFTER triggers

2014-10-08 Thread Marti Raudsepp
On Thu, Sep 4, 2014 at 12:14 AM, Kevin Grittner wrote: >> Did I miss something? > > Apparently. I did a search on the document and counted and got 101 > occurrences of "transition table". > | A transient table is a named table that may come into existence > | implicitly during the evaluation of a

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
> /* > + * Arrange to remove a dynamic shared memory mapping at cleanup time. > + * > + * dsm_keep_mapping() can be used to preserve a mapping for the entire > + * lifetime of a process; this function reverses that decision, making > + * the segment owned by the current resource owner. This may b

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 2:04 PM, Simon Riggs wrote: >> While I am also happy with taking a vote, if we do so I vote against >> even this much less MERGE-like syntax. It's verbose, and makes much >> less sense when the mechanism is driven by would-be duplicate key >> violations rather than an outer

Re: [HACKERS] What exactly is our CRC algorithm?

2014-10-08 Thread Gavin Flower
On 09/10/14 10:13, Andres Freund wrote: On 2014-10-08 22:13:46 +0300, Heikki Linnakangas wrote: Hmm. So the simple, non-table driven, calculation gives the same result as using the lookup table using the reflected lookup code. That's expected; the lookup method is supposed to be the same, just f

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 2:50 PM, Kevin Grittner wrote: > What I said was in response to your assertion that your technique > would *never* generate a duplicate key error. That is strictly true: the INSERT cannot raise a unique violation error, because to do so would cause it to take the UPDATE pat

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-08 Thread Stephen Frost
* Fabrízio de Royes Mello (fabriziome...@gmail.com) wrote: > On Tue, Oct 7, 2014 at 1:24 PM, Simon Riggs wrote: > > I hope we can get pgAudit in as a module for 9.5. I also hope that it > > will stimulate the requirements/funding of further work in this area, > > rather than squash it. My feeling

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Kevin Grittner
Peter Geoghegan wrote: On Wed, Oct 8, 2014 at 1:25 AM, Heikki Linnakangas wrote: >> Instead of naming the index, you should name the columns, and >> the system can look up the index or indexes that match those >> columns. +1 That is what I have been consistently requesting instead of index na

Re: [HACKERS] Add CREATE support to event triggers

2014-10-08 Thread Alvaro Herrera
About patch 1, which I just pushed, there were two reviews: Andres Freund wrote: > On 2014-09-25 18:59:31 -0300, Alvaro Herrera wrote: > > diff --git a/src/include/utils/ruleutils.h b/src/include/utils/ruleutils.h > > new file mode 100644 > > index 000..520b066 > > --- /dev/null > > +++ b/src/

Re: [HACKERS] What exactly is our CRC algorithm?

2014-10-08 Thread Andres Freund
On 2014-10-08 22:13:46 +0300, Heikki Linnakangas wrote: > Hmm. So the simple, non-table driven, calculation gives the same result as > using the lookup table using the reflected lookup code. That's expected; the > lookup method is supposed to be the same, just faster. However, using the > "normal"

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Kevin Grittner
Peter Geoghegan wrote: > On Mon, Oct 6, 2014 at 8:35 AM, Robert Haas wrote: >> I think the problem is that it's not possible to respect the "usual >> guarantees" even at READ COMMITTED level when performing an INSERT OR >> UPDATE operation (however spelled). You may find that there's a tuple >>

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Simon Riggs
On 8 October 2014 21:16, Peter Geoghegan wrote: >> My opinion is that syntax for this should be similar to MERGE in the >> *body* of the command, rather than some completely different syntax. >> e.g. >> >>> WHEN NOT MATCHED THEN >>> INSERT >>> WHEN MATCHED THEN >>> UPDATE >> >> I'm happy that

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Kevin Grittner
Peter Geoghegan wrote: > On Tue, Oct 7, 2014 at 5:23 AM, Simon Riggs wrote: >> IIRC it wasn't agreed that we needed to identify which indexes in the >> upsert SQL statement itself, since this would be possible in other >> ways and would require programmers to know which unique constraints >> are

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Petr Jelinek
On 08/10/14 11:28, Peter Geoghegan wrote: On Wed, Oct 8, 2014 at 1:36 AM, Marti Raudsepp wrote: But the MERGE syntax, to me, strongly implies that insertion doesn't begin before determining whether a conflict exists or not. I think you're right. Another strike against the MERGE syntax, then,

Re: [HACKERS] UPSERT wiki page, and SQL MERGE syntax

2014-10-08 Thread Peter Geoghegan
On Mon, Oct 6, 2014 at 8:35 AM, Robert Haas wrote: > I think the problem is that it's not possible to respect the "usual > guarantees" even at READ COMMITTED level when performing an INSERT OR > UPDATE operation (however spelled). You may find that there's a tuple > with the same PK which is comm

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Petr Jelinek
On 08/10/14 21:03, Robert Haas wrote: On Wed, Oct 8, 2014 at 10:09 AM, Andres Freund wrote: WRT my complaint in the other subthread, I think PQsendMethods should just be renamed to PQcommMethods or similar. That'd, in combination with a /* at some point we might want to add methods for receivi

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 6:25 AM, Simon Riggs wrote: > I change my view on this, after some more thought. (Hope that helps) Great. > If we implement MERGE, I can see we may also wish to implement MERGE > CONCURRENTLY one day. That would be different to UPSERT. > > So in the future I think we will

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund wrote: > 2) Implement the wait free LW_SHARED algorithm. + * too high for workloads/locks that were locked in shared mode very s/locked/taken/? + * frequently. Often we were spinning in the (obviously exlusive) spinlock, exclusive. + * acquiration

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Andres Freund
On 2014-10-08 15:23:22 -0400, Robert Haas wrote: > On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund wrote: > > 1) Convert PGPROC->lwWaitLink into a dlist. The old code was frail and > >verbose. This also does: > > * changes the logic in LWLockRelease() to release all shared lockers > >

Re: [HACKERS] Context lenses to set/get values in json values.

2014-10-08 Thread Andrew Dunstan
On 10/08/2014 02:04 PM, Thom Brown wrote: There is work already being done on providing update operations. I've been looking out for that. Has there been a discussion on how that would look yet that you could point me to? https://github.com/erthalion/jsonbx Note that a) it's an extension

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund wrote: > 1) Convert PGPROC->lwWaitLink into a dlist. The old code was frail and >verbose. This also does: > * changes the logic in LWLockRelease() to release all shared lockers > when waking up any. This can yield some significant perform

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 12:12 PM, Peter Geoghegan wrote: > On Mon, Sep 29, 2014 at 7:21 AM, Simon Riggs wrote: >> Having said that, it would be much nicer to have a mode that allows >> you to just say the word "UPDATE" and have it copy the data into the >> correct columns, like MySQL does. That is

[HACKERS] What exactly is our CRC algorithm?

2014-10-08 Thread Heikki Linnakangas
Our CRC algorithm is a bit weird. It's supposedly CRC-32, with the same polynomial as used in Ethernet et al, but it actually is not. The comments refer to "Painless Guide to CRC Error Detection Algorithms" by Ross N. Williams [1] (http://www.ross.net/crc/download/crc_v3.txt), but I think it wa

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Peter Geoghegan
On Mon, Sep 29, 2014 at 7:21 AM, Simon Riggs wrote: > Having said that, it would be much nicer to have a mode that allows > you to just say the word "UPDATE" and have it copy the data into the > correct columns, like MySQL does. That is very intuitive, even if it > isn't very flexible. I thought

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 10:09 AM, Andres Freund wrote: > On 2014-09-10 16:53:12 -0400, Robert Haas wrote: >> diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h >> index 5da9d8d..0b8db42 100644 >> --- a/src/include/libpq/libpq.h >> +++ b/src/include/libpq/libpq.h >> @@ -37,6 +37,31 @

Re: [HACKERS] pg_dump refactor patch to remove global variables

2014-10-08 Thread Alvaro Herrera
Joachim Wieland wrote: > On Sat, Aug 30, 2014 at 11:12 PM, Peter Eisentraut wrote: > > - The forward declaration of struct _dumpOptions in pg_backup.h seems > > kind of useless. You could move some things around so that that's not > > necessary. > > Agreed, fixed. > > > - NewDumpOptions() and

Re: [HACKERS] Context lenses to set/get values in json values.

2014-10-08 Thread Thom Brown
On 8 October 2014 18:39, Andrew Dunstan wrote: > > On 10/08/2014 12:13 PM, Paweł Cesar Sanjuan Szklarz wrote: >> >> >> >> >> I don't think we need to import Mongo type notation here. But >> there is probably a good case for some functions like: >> >>json_table_agg(anyrecord) -> jso

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-10-08 14:23:44 -0300, Alvaro Herrera wrote: > Robert Haas wrote: > > On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund > > wrote: > > > I don't see that as being relevant. The difference is an instruction or > > > two - in the slow path we'll enter the kernel and sleep. This doesn't > > > mat

Re: [HACKERS] Deferring some AtStart* allocations?

2014-10-08 Thread Robert Haas
On Sun, Jun 29, 2014 at 9:12 PM, Tom Lane wrote: > Meh. Even "SELECT 1" is going to be doing *far* more pallocs than that to > get through raw parsing, parse analysis, planning, and execution startup. > If you can find a few hundred pallocs we can avoid in trivial queries, > it would get interest

Re: [HACKERS] Log notice that checkpoint is to be written on shutdown

2014-10-08 Thread Michael Banck
Hi, Am Samstag, den 04.10.2014, 15:05 -0500 schrieb Jim Nasby: > On 10/4/14, 1:21 PM, Jeff Janes wrote: > > On Thu, Oct 2, 2014 at 6:21 AM, Michael Banck wrote: > > we have seen repeatedly that users can be confused about why PostgreSQL > > is not shutting down even though they requested i

Re: [HACKERS] Context lenses to set/get values in json values.

2014-10-08 Thread Andrew Dunstan
On 10/08/2014 12:13 PM, Paweł Cesar Sanjuan Szklarz wrote: I don't think we need to import Mongo type notation here. But there is probably a good case for some functions like: json_table_agg(anyrecord) -> json which would work like json_agg() but would return an array of

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Alvaro Herrera
Robert Haas wrote: > On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund wrote: > > I don't see that as being relevant. The difference is an instruction or > > two - in the slow path we'll enter the kernel and sleep. This doesn't > > matter in comparison. > > And the code is *so* much more readable. >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-10-08 13:13:33 -0400, Robert Haas wrote: > On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund wrote: > > I don't see that as being relevant. The difference is an instruction or > > two - in the slow path we'll enter the kernel and sleep. This doesn't > > matter in comparison. > > And the code i

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund wrote: > I don't see that as being relevant. The difference is an instruction or > two - in the slow path we'll enter the kernel and sleep. This doesn't > matter in comparison. > And the code is *so* much more readable. I find the slist/dlist stuff ac

Re: [HACKERS] Context lenses to set/get values in json values.

2014-10-08 Thread Paweł Cesar Sanjuan Szklarz
On Wed, Oct 8, 2014 at 4:25 PM, Andrew Dunstan wrote: > > On 10/08/2014 04:38 AM, Paweł Cesar Sanjuan Szklarz wrote: > >> Hello. >> >> I am interested in the json type on postgresql. I would like to implement >> additional operations on the json structure that may extract/insert table >> like inf

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Heikki Linnakangas
On 10/08/2014 04:59 PM, Fujii Masao wrote: On Wed, Oct 8, 2014 at 6:54 PM, Heikki Linnakangas wrote: Instead of creating any .done files during recovery, we could scan pg_xlog at promotion, and create a .done file for every WAL segment that's present at that point. That would be more robust. An

Re: [HACKERS] Context lenses to set/get values in json values.

2014-10-08 Thread Andrew Dunstan
On 10/08/2014 04:38 AM, Paweł Cesar Sanjuan Szklarz wrote: Hello. I am interested in the json type on postgresql. I would like to implement additional operations on the json structure that may extract/insert table like information from the json tree structure. I have a implementation on javas

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
On 2014-09-10 16:53:12 -0400, Robert Haas wrote: > diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h > index 5da9d8d..0b8db42 100644 > --- a/src/include/libpq/libpq.h > +++ b/src/include/libpq/libpq.h > @@ -37,6 +37,31 @@ typedef struct > } u; > } PQArg

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
On 2014-09-29 13:38:37 -0400, Robert Haas wrote: > On Mon, Sep 29, 2014 at 12:05 PM, Stephen Frost wrote: > > Lastly, I will say that I feel it'd be good to support bi-directional > > communication as I think it'll be needed eventually, but I'm not sure > > that has to happen now. > > I agree we

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Fujii Masao
On Wed, Oct 8, 2014 at 6:54 PM, Heikki Linnakangas wrote: > On 10/08/2014 10:44 AM, Michael Paquier wrote: >> >> On Fri, Sep 19, 2014 at 1:07 AM, Jehan-Guillaume de Rorthais < >> j...@dalibo.com> wrote: >> >>> We kept the WAL files and log files for further analysis. How can we help >>> regarding

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-10-08 Thread Fujii Masao
On Wed, Sep 24, 2014 at 11:10 AM, Etsuro Fujita wrote: > (2014/09/13 2:42), Fujii Masao wrote: >> >> On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera >> wrote: >>> >>> Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita wrote: >>> >>> > PENDING_LIST_CLEANUP_SIZE

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Andres Freund
Hi, Attached you can find the next version of my LW_SHARED patchset. Now that atomics are committed, it seems like a good idea to also add their raison d'être. Since the last public version I have: * Addressed lots of Amit's comments. Thanks! * Peformed a fair amount of testing. * Rebased the cod

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Simon Riggs
On 8 October 2014 00:34, Peter Geoghegan wrote: > INSERTs see #2 win, and by a wider margin than #1 beat #2 with > UPDATEs. However, insert.sh is by design very unsympathetic towards > #1. It uses a serial primary key, so every INSERT uselessly obtains a > HW lock on the same leaf page for the du

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Simon Riggs
On 8 October 2014 01:47, Peter Geoghegan wrote: > It seems like what you're talking about here is just changing the > spelling of what I already have. I think that would be confusing to > users when the time comes to actually implement a fully-generalized > MERGE, even with the clearly distinct M

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Michael Paquier
On Wed, Oct 8, 2014 at 6:54 PM, Heikki Linnakangas wrote: > 1. Where do the FF files come from? In 9.2, FF-segments are not supposed > to created, ever. > Since this only happens with streaming replication, the FF segments are > probably being created by walreceiver. XLogWalRcvWrite is the functi

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-10-08 14:47:44 +0200, Andres Freund wrote: > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > > 5. > > LWLockWakeup() > > { > > .. > > dlist_foreach_modify(iter, (dlist_head *) &wakeup) > > { > > PGPROC *waiter = dlist_container(PGPROC, lwWaitLink, iter.cur); > > LOG_LWDEBUG("LWLockReleas

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > 2. > LWLockWakeup() > { > .. > #ifdef LWLOCK_STATS > lwstats->spin_delay_count += SpinLockAcquire(&lock->mutex); > #else > SpinLockAcquire(&lock->mutex); > #endif > .. > } > > Earlier while releasing lock, we don't count it towards LWLock stats >

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread Heikki Linnakangas
On 10/08/2014 11:47 AM, furu...@pm.nttdata.co.jp wrote: On 10/08/2014 07:23 AM, furu...@pm.nttdata.co.jp wrote: What set of options would you pass if you want to use it as a synchronous standby? And if you don't? Could we just have a single "--synchronous" flag, instead of -F and --reply-fsync

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-08 Thread Andres Freund
On 2014-10-09 00:21:44 +1300, David Rowley wrote: > Ok, so I've been hacking away at this for a couple of evenings and I think > I have a working prototype finally! Cool! > So it seems it's not quite as efficient as join removal at planning time, > but still a big win when it's possible to perfor

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-10-08 Thread David Rowley
On Tue, Oct 7, 2014 at 3:46 AM, Robert Haas wrote: > On Mon, Oct 6, 2014 at 5:57 AM, David Rowley wrote: > > Can anyone shed any light on how I might determine where the scan rel is > in > > the tree? I need to find it so I can check if the RangeTblEntry is > marked as > > skip-able. > > I think

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Marti Raudsepp
On Wed, Oct 8, 2014 at 12:28 PM, Peter Geoghegan wrote: > On Wed, Oct 8, 2014 at 1:36 AM, Marti Raudsepp wrote: >> I think there's a subtle difference in expectations too. The current >> BEFORE INSERT trigger behavior is somewhat defensible with an >> INSERT-driven syntax (though I don't like it

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Anssi Kääriäinen
On Wed, 2014-10-08 at 01:10 -0700, Peter Geoghegan wrote: > On Wed, Oct 8, 2014 at 12:41 AM, Anssi Kääriäinen > wrote: > > The MySQL documentation says that "you should try to avoid using an ON > > DUPLICATE KEY UPDATE clause on tables with multiple unique indexes"[1]. > > The proposed feature's d

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Anssi Kääriäinen
On Wed, 2014-10-08 at 02:22 -0700, Peter Geoghegan wrote: > On Wed, Oct 8, 2014 at 1:25 AM, Heikki Linnakangas > wrote: > > Instead of naming the index, you should name the columns, and the system can > > look up the index or indexes that match those columns. > > It's not totally clear that we ne

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Heikki Linnakangas
On 10/08/2014 10:44 AM, Michael Paquier wrote: On Fri, Sep 19, 2014 at 1:07 AM, Jehan-Guillaume de Rorthais < j...@dalibo.com> wrote: We kept the WAL files and log files for further analysis. How can we help regarding this issue? Commit c2f79ba has added as assumption that the WAL receiver s

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Marti Raudsepp
On Tue, Oct 7, 2014 at 2:27 PM, Marti Raudsepp wrote: > but the new approach seems > surprising: changes from BEFORE INSERT get persisted in the database, > but AFTER INSERT is not fired. I am sorry, I realize now that I misunderstood the current proposed trigger behavior, I thought what Simon Ri

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 1:36 AM, Marti Raudsepp wrote: > I think there's a subtle difference in expectations too. The current > BEFORE INSERT trigger behavior is somewhat defensible with an > INSERT-driven syntax (though I don't like it even now [1]). There is no way around it. We need to fire bef

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 1:25 AM, Heikki Linnakangas wrote: > Instead of naming the index, you should name the columns, and the system can > look up the index or indexes that match those columns. It's not totally clear that we need *any* WITHIN clause, BTW. I'm not dead set on it. It was something

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread furuyao
> On 10/08/2014 07:23 AM, furu...@pm.nttdata.co.jp wrote: > >> What set of options would you pass if you want to use it as a > >> synchronous standby? And if you don't? Could we just have a single > "--synchronous" > >> flag, instead of -F and --reply-fsync? > > > > If you set "synchronous_commit"

[HACKERS] Context lenses to set/get values in json values.

2014-10-08 Thread Paweł Cesar Sanjuan Szklarz
Hello. I am interested in the json type on postgresql. I would like to implement additional operations on the json structure that may extract/insert table like information from the json tree structure. I have a implementation on javascript that shows this type of operations. You can see examples i

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-08 Thread Marti Raudsepp
On Wed, Oct 8, 2014 at 3:47 AM, Peter Geoghegan wrote: > It seems like what you're talking about here is just changing the > spelling of what I already have. I think there's a subtle difference in expectations too. The current BEFORE INSERT trigger behavior is somewhat defensible with an INSERT-d

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Heikki Linnakangas
On 10/08/2014 11:10 AM, Peter Geoghegan wrote: The reasoning behind making the unique index specification optional is: We cannot easily cover corner cases with another syntax - unique indexes must be named directly to cover every case, and make the user's intent absolutely clear. That's not obvi

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Peter Geoghegan
On Wed, Oct 8, 2014 at 12:41 AM, Anssi Kääriäinen wrote: > The MySQL documentation says that "you should try to avoid using an ON > DUPLICATE KEY UPDATE clause on tables with multiple unique indexes"[1]. > The proposed feature's documentation has the same suggestion[2]. Still, > the feature defaul

Re: [HACKERS] BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

2014-10-08 Thread Michael Paquier
On Fri, Sep 19, 2014 at 1:07 AM, Jehan-Guillaume de Rorthais < j...@dalibo.com> wrote: > We kept the WAL files and log files for further analysis. How can we help > regarding this issue? > Commit c2f79ba has added as assumption that the WAL receiver should always enforce the create of .done files

Re: [HACKERS] Promise index tuples for UPSERT

2014-10-08 Thread Anssi Kääriäinen
On Tue, 2014-10-07 at 13:33 +0100, Simon Riggs wrote: > Is there a way of detecting that we are updating a unique constraint > column and then applying the HW locking only in that case? Or can we > only apply locking when we have multiple unique constraints on a > table? What is the use case of do

Re: [HACKERS] pg_receivexlog always handles -d option argument as connstr

2014-10-08 Thread Sawada Masahiko
Amit Kapila > On Tue, Oct 7, 2014 at 8:13 PM, Sawada Masahiko > wrote: > > > > On Tue, Oct 7, 2014 at 12:58 PM, Amit Kapila > wrote: > > > On Mon, Oct 6, 2014 at 10:23 PM, Sawada Masahiko < > sawada.m...@gmail.com > > > > > wrote: > > >> > > >> Hi all, > > >> > > >> pg_receivexlog always handles

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-10-08 Thread Heikki Linnakangas
On 10/08/2014 07:23 AM, furu...@pm.nttdata.co.jp wrote: What set of options would you pass if you want to use it as a synchronous standby? And if you don't? Could we just have a single "--synchronous" flag, instead of -F and --reply-fsync? If you set "synchronous_commit" as "remote_write", the