Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Simon Riggs
On 8 June 2018 at 19:03, Andres Freund wrote: > Hi, > > On 2018-06-08 09:23:02 +0100, Simon Riggs wrote: >> I have also found another bug which affects what we do next. >> >> For context, AEL locks are normally removed by COMMIT or ABORT. >> StandbyReleaseOldL

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Simon Riggs
On 8 June 2018 at 19:03, Andres Freund wrote: >> It seems to have affected Greg. > > As far as I can tell Greg was just theorizing? I'll wait for Greg to say whether this was an actual case that needs to be fixed or not. If not, happy to revert. -- Simon Riggs

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Simon Riggs
until it affects you, then its not. It seems to have affected Greg. The attached patch, or a later revision, needs to be backpatched to PG10 independently of the recent committed patch. I have yet to test this manually, but will do so tomorrow morning. -- Simon Riggshttp://www

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-07 Thread Simon Riggs
l case for this specific situation. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 20:27, Tom Lane wrote: > Simon Riggs writes: >> If we're going to compress the protocol, it seems sensible to remove >> extraneous information first. > > Breaking the wire protocol was nowhere in this thread. No, it wasn't. But there is another

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 19:20, Andres Freund wrote: > On 2018-06-07 11:40:48 +0100, Simon Riggs wrote: >> On 7 June 2018 at 11:29, Pavel Stehule wrote: >> >> >> Do we actually need the completion tag at all? In most cases?? >> > >> > >> > af

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 11:29, Pavel Stehule wrote: >> Do we actually need the completion tag at all? In most cases?? > > > affected rows is taken from this value on protocol level I didn't mean we should remove the number of rows. Many things rely on that. -- Simon Riggs

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
d the completion tag at all? In most cases?? Perhaps we should add a parameter to make it optional and turn it off by default, except for psql. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Possible bug in logical replication.

2018-06-07 Thread Simon Riggs
On 6 June 2018 at 17:22, Alvaro Herrera wrote: > This thread seems to have died down without any fix being proposed. > Simon, you own this open item. Thanks, will look. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Tr

Re: I'd like to discuss scaleout at PGCon

2018-06-06 Thread Simon Riggs
ted (difficult to scale) 2. you must put data in all places the client can connect to (i.e. multimaster) Perhaps there are more? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: plans for PostgreSQL 12

2018-06-04 Thread Simon Riggs
ing or not. Sounds good. I think this would need to be restricted by operator and datatype, since in general you won't know if the datatype functions need a snapshot or not. Immutable functions for the operators ought to do it, but I think that might not be enough. -- Simon Riggs

Re: I'd like to discuss scaleout at PGCon

2018-06-03 Thread Simon Riggs
ity solution. Each incoming query needs to be planned and executed from one coordinator component, then the work performed across many workers on different nodes (or just one). We could have coordinator components on each worker node, or we could have a set of coordinator nodes and a set of worker nodes. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: I'd like to discuss scaleout at PGCon

2018-06-02 Thread Simon Riggs
On 1 June 2018 at 16:56, Ashutosh Bapat wrote: > On Fri, Jun 1, 2018 at 11:10 AM, Simon Riggs wrote: >> >> Using a central coordinator also allows multi-node transaction >> control, global deadlock detection etc.. > > But that becomes an SPOF and then we have to config

Re: I'd like to discuss scaleout at PGCon

2018-06-01 Thread Simon Riggs
. For example, supporting SQL query plans that allow for redistribution of data for joins. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: I'd like to discuss scaleout at PGCon

2018-06-01 Thread Simon Riggs
"normalized" parse tree down to the foreign server. We need to > normalize the OIDs in the parse tree since those may be different > across the nodes. Passing detailed info between servers is exactly what XL does. It requires us to define a cluster, exactly as XL does. And yes,

Re: SHOW ALL does not honor pg_read_all_settings membership

2018-05-31 Thread Simon Riggs
o v10? > GetConfigOption is wrong by the way, as restrict_superuser means that > all members of the group pg_read_all_settings can read > GUC_SUPERUSER_ONLY params, and not only superusers, so the comment at > least needs a fix, the variable ought to be renamed as well. OK -- Simon Ri

Re: pg_replication_slot_advance to return NULL instead of 0/0 if slot not advanced

2018-05-31 Thread Simon Riggs
ments explaining how to access the various fields in the structure, so there was no way to check whether the code was good or not. If we add corrective code we should clarify that in comments the .h file also, as is done in XlogCtl Your points look correct to me, well spotted. I'd like to separ

Re: Undo logs

2018-05-28 Thread Simon Riggs
are quite a few design decisions there that need to be discussed, so lets crack on and discuss them please. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pg_replication_slot_advance to return NULL instead of 0/0 if slot not advanced

2018-05-25 Thread Simon Riggs
we try to move it backwards, but if that one fails we return > InvalidXLogRecPtr (which can happen in the case of concurrent activity on > the slot, I think)? In this case, maybe we should just re-check that and > raise an error appropriately? Agreed > (I haven't looked at the logical slot path, but I assume it would have > something similar in it) -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PG 11 feature count

2018-05-18 Thread Simon Riggs
t; 9.6 220 > 10 194 > 11 167 It would be useful to combine that with the CF app data showing number of patches submitted and number of features rejected. Not available for all time, but certainly goes back a few years now. -- Simon Riggs

Re: Removing unneeded self joins

2018-05-16 Thread Simon Riggs
On 16 May 2018 at 15:10, Tom Lane <t...@sss.pgh.pa.us> wrote: > Simon Riggs <si...@2ndquadrant.com> writes: >> What I would add is that I've seen cases where the extra joins do NOT >> hurt performance, so the extra CPU used to remove the join hurts more >> than th

Re: Removing unneeded self joins

2018-05-16 Thread Simon Riggs
ore than the benefit of removing it. Yes, we tried it. More advanced optimizations should only be applied when we've assessed that the likely run time is high enough to make it worth investing in further optimization. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Surjective functional indexes

2018-05-11 Thread Simon Riggs
On 11 May 2018 at 16:37, Andres Freund <and...@anarazel.de> wrote: > On 2018-05-11 14:56:12 +0200, Simon Riggs wrote: >> On 11 May 2018 at 05:32, Andres Freund <and...@anarazel.de> wrote: >> > No. Simon just claimed it's not actually a concern: >> > https:

Re: [HACKERS] Surjective functional indexes

2018-05-11 Thread Simon Riggs
am happy that Konstantin has changed his patch significantly in response to those reviews, as well as explaining why the patch is fine as it is. It's a useful patch that PostgreSQL needs, so all good. I have no problem if you want to replace this with an even better design in a later release. -- Si

Re: Indexes on partitioned tables and foreign partitions

2018-05-11 Thread Simon Riggs
On 9 May 2018 at 17:33, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, May 9, 2018 at 11:20 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >> On 9 May 2018 at 16:15, Robert Haas <robertmh...@gmail.com> wrote: >>> On Wed, May 9, 2018 at 11:14 AM, Simon

Re: Needless additional partition check in INSERT?

2018-05-09 Thread Simon Riggs
I think so too. > > Updated patch attached. Patch is good. The cause of this oversight is the lack of comments to explain the original coding, so we need to correct that in this patch, please. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
On 9 May 2018 at 16:15, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, May 9, 2018 at 11:14 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >> On 9 May 2018 at 16:10, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> Robert Haas <robertmh...@gmail.com> w

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
On 9 May 2018 at 16:10, Tom Lane <t...@sss.pgh.pa.us> wrote: > Robert Haas <robertmh...@gmail.com> writes: >> On Wed, May 9, 2018 at 9:08 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >>> Shouldn't the fix be to allow creation of indexes on foreign tabl

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
n index exists on a foreign table, why can we not just assume a unique index exists?? Why the difference? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
On 9 May 2018 at 15:26, Arseny Sher <a.s...@postgrespro.ru> wrote: > > Simon Riggs <si...@2ndquadrant.com> writes: > >> How much sense is it to have a partitioned table with a mix of local >> and foreign tables? > > Well, as much sense as fdw-based shard

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
the fix be to allow creation of indexes on foreign tables? (Maybe they would be virtual or foreign indexes??) -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-29 Thread Simon Riggs
On 28 April 2018 at 09:15, Andres Freund <and...@anarazel.de> wrote: > Hi, > > On 2018-04-28 08:25:53 -0700, Simon Riggs wrote: >> > - Use direct IO. Due to architectural performance issues in PG and the >> > fact that it'd not be applicable fo

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-29 Thread Simon Riggs
On 28 April 2018 at 08:25, Simon Riggs <si...@2ndquadrant.com> wrote: > On 27 April 2018 at 15:28, Andres Freund <and...@anarazel.de> wrote: > >> - Add a pre-checkpoint hook that checks for filesystem errors *after* >> fsyncing all the files, but *before* log

Re: Query is over 2x slower with jit=on

2018-04-18 Thread Simon Riggs
o humans before this gets too far into the wild. SSL is somewhat understandable because its not a Postgres-private term. geqo is regrettable and we really don't want any more too short/abbreviated parameter names. Think of our EOU if every GUC was a TLA. Thanks -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2018-04-18 Thread Simon Riggs
happens. The lock itself can be cancelled, so the user would also be interested in explicitly requesting a retry with a separate command/function. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-11 Thread Simon Riggs
On 11 April 2018 at 19:57, Tom Lane <t...@sss.pgh.pa.us> wrote: > Pavan Deolasee <pavan.deola...@gmail.com> writes: >> Ok. I propose attached patches, more polished this time. > > I'll take these, unless some other committer is hot to do so? Please go ahead. -- Simo

Re: pgsql: New files for MERGE

2018-04-11 Thread Simon Riggs
On 7 April 2018 at 18:45, Tom Lane <t...@sss.pgh.pa.us> wrote: > Simon Riggs <si...@2ndquadrant.com> writes: >> On 6 April 2018 at 17:22, Bruce Momjian <br...@momjian.us> wrote: >>> My point was that people didn't ask you to work harder on fixing the >>&

Re: pgsql: New files for MERGE

2018-04-07 Thread Simon Riggs
On 6 April 2018 at 17:22, Bruce Momjian <br...@momjian.us> wrote: > On Fri, Apr 6, 2018 at 09:21:54AM +0100, Simon Riggs wrote: >> On 5 April 2018 at 21:02, Bruce Momjian <br...@momjian.us> wrote: >> > Simon, you have three committers in this thread suggesti

Re: Add support for printing/reading MergeAction nodes

2018-04-06 Thread Simon Riggs
On 4 April 2018 at 18:08, Tom Lane <t...@sss.pgh.pa.us> wrote: > Simon Riggs <si...@2ndquadrant.com> writes: >> On 4 April 2018 at 17:19, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> BTW, poking around in the grammar, I notice that MergeStmt did not >>>

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-06 Thread Simon Riggs
and replication. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: New files for MERGE

2018-04-06 Thread Simon Riggs
On 5 April 2018 at 21:02, Bruce Momjian <br...@momjian.us> wrote: > On Thu, Apr 5, 2018 at 11:15:20AM +0100, Simon Riggs wrote: >> On 4 April 2018 at 21:28, Simon Riggs <si...@2ndquadrant.com> wrote: >> > On 4 April 2018 at 21:14, Andres Freund <and...@anarazel.

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 17:07, Alvaro Herrera <alvhe...@alvh.no-ip.org> wrote: > Simon Riggs wrote: >> On 5 April 2018 at 16:09, Alvaro Herrera <alvhe...@alvh.no-ip.org> wrote: >> > Quick item: parse_clause.h fails cpluspluscheck because it has a C++ >> &

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
espace’ >List **namespace); > ^ How's this as a fix? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services fnamespace.v1.patch Description: Binary data

Re: Excessive PostmasterIsAlive calls slow down WAL redo

2018-04-05 Thread Simon Riggs
y delay, so this would only > affect the case where we're actively replaying records. +1 -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
otaro%40lab.ntt.co.jp > > and his suggestion to use special macro looks better for me: > - charrelkind; > + charrelkind PG_USED_FOR_ASSERTS_ONLY; Thanks both, I already fixed that. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 13:19, Jesper Pedersen <jesper.peder...@redhat.com> wrote: > Hi, > > On 04/05/2018 08:04 AM, Simon Riggs wrote: >> >> On 5 April 2018 at 12:56, Jesper Pedersen <jesper.peder...@redhat.com> >> wrote: >>> >>> Updated for non

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 12:56, Jesper Pedersen <jesper.peder...@redhat.com> wrote: > Hi, > > On 04/05/2018 07:48 AM, Simon Riggs wrote: >>> >>> Updated version due to latest refactoring. >> >> >> Thanks for your input. Removing that seems to prevent

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 12:38, Jesper Pedersen <jesper.peder...@redhat.com> wrote: > Hi Simon and Paven, > > On 04/04/2018 08:46 AM, Jesper Pedersen wrote: >> >> On 03/30/2018 07:10 AM, Simon Riggs wrote: >>> >>> No problems found, but moving proposed comm

Re: Add support for printing/reading MergeAction nodes

2018-04-05 Thread Simon Riggs
triction correctly. I'll commit this tomorrow morning unless further comments or edits. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: New files for MERGE

2018-04-05 Thread Simon Riggs
On 4 April 2018 at 21:28, Simon Riggs <si...@2ndquadrant.com> wrote: > On 4 April 2018 at 21:14, Andres Freund <and...@anarazel.de> wrote: > >>> The normal way is to make review comments that allow change. Your >>> request for change of the parser data st

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
> > Makes sense. Done. (Now that the patch is committed, I don't know if there > would be a rethink about changing file names. May be not, just raising that > concern) My review notes suggest a file called execMerge.c. I didn't spot the filename change. I think it's important to do that because there is no executor node called Merge. That is especially confusing because there *is* an executor node called MergeAppend and we want some cognitive distance between those things. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: New files for MERGE

2018-04-04 Thread Simon Riggs
nd you've so far ignored those requests. Pavan tells me he has replied to you and is working on specific changes. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: New files for MERGE

2018-04-04 Thread Simon Riggs
on of MERGE and that is why I committed it. If it were not, I would have no hesitation in pushing back a year or more, if I knew of technical reasons to do so. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Add support for printing/reading MergeAction nodes

2018-04-04 Thread Simon Riggs
On 4 April 2018 at 18:51, Tom Lane <t...@sss.pgh.pa.us> wrote: > Simon Riggs <si...@2ndquadrant.com> writes: >> On 4 April 2018 at 17:19, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> If the MERGE patch has broken this, I'm going to push back on that >>>

Re: Add support for printing/reading MergeAction nodes

2018-04-04 Thread Simon Riggs
tion. Why would it be in RuleActionStmt? Could you explain what command you think should be supported? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PATCH] Logical decoding of TRUNCATE

2018-04-04 Thread Simon Riggs
rently handled like a DDL command, which doesn't work > well. Agreed -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PATCH] Logical decoding of TRUNCATE

2018-04-02 Thread Simon Riggs
ersion. Who has argued against it? I see that Andres has asked twice about it and been answered twice, but expressed no opinion. Petr has said he thinks that's right. Nobody else has commented. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-02 Thread Simon Riggs
which would indicate this fact - logical decoding is using it's own > snapshots it could inject the information about being inside the 2PC > decoding. Perhaps, but how do we know we've covered all the right places? We don't know what every plugin will require, do we? The plugin needs to take resp

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-02 Thread Simon Riggs
n there isn't one is unreasonable. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PATCH] Logical decoding of TRUNCATE

2018-04-02 Thread Simon Riggs
et of actions to be logically >> decoded. >> + * We could optimize this away when !RelationIsLogicallyLogged(rel) >> + * but that doesn't save much space or time. > > What you're saying isn't that you're not logging anything, but that > you're allocating the header re

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-04-01 Thread Simon Riggs
ndby. VACUUM FULL was the thing that needed to be excluded in the past because it needed an xid to move rows. Greg's concern is a good one and his noticing that we hadn't specifically excluded VACUUMs is valid, so we should exclude them. Well spotted, Greg. So although this doesn't have the

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-30 Thread Simon Riggs
On 29 March 2018 at 10:50, Simon Riggs <si...@2ndquadrant.com> wrote: > On 28 March 2018 at 12:00, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > >> v27 attached, though review changes are in >> the add-on 0005 patch. > > This all looks good now, thanks fo

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-30 Thread Simon Riggs
ree with points about XLOG_SWITCH, adding again the observation that they are no longer used for replication and if they are used as well as replication, have a bad effect on performance. I think it would be easily possible to add some more detail to the WAL stream if needed. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
CRC, only xl_prev. The existing WAL format is easy enough to repair, falsify etc. You've still got to keep your WAL data secure. So the image of a CSI-style chain-of-evidence thing happening isn't technical reality. You could change the existing format to include the previous CRC in the calculation of the current record's CRC. I'd be happy with that as an option, just as much as I'd support an option for high performance as I've proposed here. But bear in mind that including the CRC would mean that you couldn't repair data damaged by data corruption because it would mean all WAL after the corrupt page would be effectively scrambled. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
sting record (same size/boundaries) >> >> That seems very much like xl_prev. > > I don't think so, because this ignores, for example, timeline > switches, or multiple clusters accidentally sharing an archive > directory. I'm not hearing any actual technical problems. > Giv

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
On 29 March 2018 at 18:13, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > On 03/29/2018 06:42 PM, Tom Lane wrote: >> Simon Riggs <si...@2ndquadrant.com> writes: >>> I know the approach is new and surprising but I thought about it a lot >>> before pr

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
very much viable. I know the approach is new and surprising but I thought about it a lot before proposing it and I couldn't see any holes; still can't. Please give this some thought so we can get comfortable with this idea and increase performance as a result. Thanks. -- Simon Riggs

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-29 Thread Simon Riggs
to follow up on other issues that we will no doubt discover fairly soon after commit, such as additional runs by SQLsmith and more eyeballs. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] A design for amcheck heapam verification

2018-03-29 Thread Simon Riggs
y, which makes it more likely that people would just run CREATE INDEX and not bother with the check. So my initial questions are about when we would run this and making sure that is documented. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Commit fest 2017-11

2018-03-29 Thread Simon Riggs
tistics? > > It would only help for the future of course, unless somebody wants to go > back and backfill existing patches with such information (which they might > be). The focus of this is on the Committers, which seems wrong. I suggest someone does another analysis that shows how many patch reviews have been conducted by patch authors, so we can highlight people who are causing the problem yet not helping solve the problem. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-29 Thread Simon Riggs
On 29 March 2018 at 07:37, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Tue, Mar 27, 2018 at 5:00 PM, Simon Riggs <si...@2ndquadrant.com> wrote: >> >> >> In terms of further performance optimization, if there is just one >> WHEN AND condit

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-28 Thread Simon Riggs
On 28 March 2018 at 16:28, Nikhil Sontakke <nikh...@2ndquadrant.com> wrote: > Simon, 0003-Add-GID-and-replica-origin-to-two-phase-commit-abort.patch > is the exact patch that you had posted for an earlier commit. 0003 Pushed -- Simon Riggshttp://www.2ndQuadrant.com

Re: [HACKERS] Surjective functional indexes

2018-03-27 Thread Simon Riggs
On 23 March 2018 at 15:54, Simon Riggs <si...@2ndquadrant.com> wrote: > So please could you make the change? Committed, but I still think that change would be good. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-27 Thread Simon Riggs
ious WAL record. >> 2. Does the new logic in pg_rewind to search backward for a checkpoint >> work reliably, and will it be slow? > > If you have to search backwards, this breaks it. Full stop. You don't have to search backwards. We only need to locate the last checkpoint record

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
On 27 March 2018 at 11:46, Simon Riggs <si...@2ndquadrant.com> wrote: > On 27 March 2018 at 10:31, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > >> Fixed in v26. > > More comments on v26 In terms of further performance optimization, if there

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
ywhere else * Need tests for coverage of these ERROR messages Named security policy violation SELECT not allowed in MERGE INSERT... Multiple VALUES clauses not... MERGE is not supported for this... MERGE is not supported for relations with inheritance MERGE is not supported for relations with rul

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
from the source side since the > mergrSourceTargetList only includes user columns and so set_plan_refs() > complains about a system column. > > I am not sure what's the best way to handle this. May be we can add system > columns to the mergrSourceTargetList. I haven't yet found a neat way to do > that. I was saying the comment needs changing, not the code. Cool, thanks -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-27 Thread Simon Riggs
On 23 March 2018 at 15:26, Simon Riggs <si...@2ndquadrant.com> wrote: > Reviewing 0003-Add-support-for-logging-GID-in-commit-abort-WAL-reco > > Looks fine, reworked patch attached > * added changes to xact.h from patch 4 so that this is a whole, > committable patch >

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
On 26 March 2018 at 17:06, Simon Riggs <si...@2ndquadrant.com> wrote: > On 26 March 2018 at 15:39, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > That's all I can see so far. * change comment “once to” to “once” in src/include/nodes/execnodes.h * change comment “and

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
On 26 March 2018 at 23:10, Peter Geoghegan <p...@bowt.ie> wrote: > On Mon, Mar 26, 2018 at 12:17 PM, Simon Riggs <si...@2ndquadrant.com> wrote: >>> As far as I >>> know, the proposed MERGE patch has that issue an existing DML commands >>> don't; bu

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
On 26 March 2018 at 17:52, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Mar 26, 2018 at 12:16 PM, Simon Riggs <si...@2ndquadrant.com> wrote: >> On 26 March 2018 at 16:09, Robert Haas <robertmh...@gmail.com> wrote: >>> On Mon, Mar 26, 2018 at 5:53 AM

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
On 26 March 2018 at 16:09, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >> Since we now have MVCC catalog scans, all the name lookups are >> performed using the same snapshot so in the above sc

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
separate triggers are defined for MERGE" we should also state the same caveat for POLICY events That's all I can see so far. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
rees. +1 So v25 is the "commit candidate" and we can add other patches to it. Given recent bugfix/changes I don't plan to commit this tomorrow anymore. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
the same, then we could just bomb > out with an error if they aren't. Since we now have MVCC catalog scans, all the name lookups are performed using the same snapshot so in the above scenario the newly created object would be invisible to the second name lookup. So I don't see anyway for the ERROR to occur and hence no need for a cross check, for UPDATE or MERGE. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Simon Riggs
On 24 March 2018 at 12:19, Robert Haas <robertmh...@gmail.com> wrote: > On Sat, Mar 24, 2018 at 8:11 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >> On 24 March 2018 at 11:58, Robert Haas <robertmh...@gmail.com> wrote: >>> On Sat, Mar 24, 2018 at 7:42 AM

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-24 Thread Simon Riggs
. MERGE uses multiple RTEs in exactly the same way UPDATE does. I don't see a reason for specific concern wrt to MERGE. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Simon Riggs
On 24 March 2018 at 11:58, Robert Haas <robertmh...@gmail.com> wrote: > On Sat, Mar 24, 2018 at 7:42 AM, Simon Riggs <si...@2ndquadrant.com> wrote: >> I suggest we focus on the engineering. I've not discussed this patch >> with Pavan offline. > > Well then propo

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Simon Riggs
I suggest we focus on the engineering. I've not discussed this patch with Pavan offline. On 23 March 2018 at 23:32, Michael Paquier <mich...@paquier.xyz> wrote: > On Fri, Mar 23, 2018 at 11:06:48AM +0000, Simon Riggs wrote: >> Your assumption that I would commit a new patch that

Re: [HACKERS] Surjective functional indexes

2018-03-23 Thread Simon Riggs
strange now he mentions it. Above that in RelationData we have other structures that are List of OIDs, so Alvaro's proposal make sense. That would simplify the code in ProjectionIsNotChanged() by just looping over the list of projection indexes rather than the list of indexes So please could y

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-23 Thread Simon Riggs
dded changes to xact.h from patch 4 so that this is a whole, committable patch * added comments to make abort and commit structs look same Attached patch is proposed for a separate, early commit as part of this -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Suppo

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-23 Thread Simon Riggs
;>> DML statement in the parser. I have written a few tests, and so far it >>> holds up. Peter, if you have the code and you consider it important that this subfeature is in PostgreSQL, why not post the code so we can commit it? Why would we repeat what has already been done? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Simon Riggs
On 23 March 2018 at 09:22, Michael Paquier <mich...@paquier.xyz> wrote: > On Fri, Mar 23, 2018 at 09:04:55AM +0000, Simon Riggs wrote: >> So it shows clear benefit for both bulk actions and OLTP, with no >> regressions. >> >> No objection exists to the appro

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Simon Riggs
On 23 March 2018 at 08:35, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Fri, Mar 9, 2018 at 8:49 PM, Peter Eisentraut > <peter.eisentr...@2ndquadrant.com> wrote: >> >> On 2/1/18 19:21, Simon Riggs wrote: >> > If we really can't persua

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Simon Riggs
On 2 February 2018 at 02:17, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Fri, Feb 02, 2018 at 12:21:49AM +0000, Simon Riggs wrote: >> Yes, it would be about 99% of the time. > > When it comes to recovery, I don't think that 99% is a guarantee > sufficient. (W

Re: [HACKERS] Surjective functional indexes

2018-03-22 Thread Simon Riggs
s with it have been around the actual user interface to request it. Index option handling has changed (and this needs rebase!), but other than that I think we want this and am planning to commit something early next week. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-22 Thread Simon Riggs
r cleanup of related issues. If there are any items you believe are still open, please say so now or mention any other objections you have. Thanks for all of your detailed comments, -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-22 Thread Simon Riggs
vides an atomic > INSERT or UPDATE outcome in READ COMMITTED mode. > > BTW, the docs should be clear on the fact that "INSERT ... ON > CONFLICT" isn't a statement. INSERT is. ON CONFLICT is a clause. I think it would be better if you wrote a separate additional doc patch to explain all

Re: Faster inserts with mostly-monotonically increasing values

2018-03-14 Thread Simon Riggs
On 14 March 2018 at 13:33, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Wed, Mar 14, 2018 at 4:53 PM, Simon Riggs <simon.ri...@2ndquadrant.com> > wrote: >> >> On 14 March 2018 at 04:36, Pavan Deolasee <pavan.deola...@gmail.com> >>

Re: Faster inserts with mostly-monotonically increasing values

2018-03-14 Thread Simon Riggs
So basically only attempt the optimization if uncontended. This makes sense because in many cases if the block is locked it is filling up and the RHS block is more likely to change anyway. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

<    1   2   3   4   5   6   7   >