Re: [HACKERS] Optimising Foreign Key checks

2013-06-10 Thread Noah Misch
On Sun, Jun 09, 2013 at 10:51:43AM +0100, Simon Riggs wrote: On 9 June 2013 02:12, Noah Misch n...@leadboat.com wrote: On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch n...@leadboat.com wrote: Likewise; I don't see why we couldn't

Re: [HACKERS] Optimising Foreign Key checks

2013-06-10 Thread Simon Riggs
On 10 June 2013 07:06, Noah Misch n...@leadboat.com wrote: On Sun, Jun 09, 2013 at 10:51:43AM +0100, Simon Riggs wrote: On 9 June 2013 02:12, Noah Misch n...@leadboat.com wrote: On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch

Re: [HACKERS] Optimising Foreign Key checks

2013-06-10 Thread Noah Misch
On Mon, Jun 10, 2013 at 09:05:40AM +0100, Simon Riggs wrote: Your earlier comments argue that it is OK to make an early check. The above seems to argue the opposite, not sure. I'll attempt to summarize. If we execute a traditional error-throwing FK check any earlier than we execute it today,

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Simon Riggs
On 9 June 2013 02:12, Noah Misch n...@leadboat.com wrote: On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch n...@leadboat.com wrote: Likewise; I don't see why we couldn't perform an optimistic check ASAP and schedule a final

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Andres Freund
On 2013-06-01 09:41:13 +0100, Simon Riggs wrote: FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here: Another idea would be to optimize away the row level locks if we have

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Greg Stark
On Sun, Jun 9, 2013 at 10:51 AM, Simon Riggs si...@2ndquadrant.com wrote: AFAICS there are weird cases where changing the way FKs execute will change the way complex trigger applications will execute. I don't see a way to avoid that other than do nothing. Currently, we execute the checks

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Simon Riggs
On 9 June 2013 14:59, Greg Stark st...@mit.edu wrote: On Sun, Jun 9, 2013 at 10:51 AM, Simon Riggs si...@2ndquadrant.com wrote: AFAICS there are weird cases where changing the way FKs execute will change the way complex trigger applications will execute. I don't see a way to avoid that other

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: For clarity the 4 problems are 1. SQL execution overhead 2. Memory usage 3. Memory scrolling 4. Locking overhead, specifically FPWs and WAL records from FK checks

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Simon Riggs
On 8 June 2013 15:30, Noah Misch n...@leadboat.com wrote: On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: For clarity the 4 problems are 1. SQL execution overhead 2. Memory usage 3. Memory scrolling 4. Locking

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 09:39:14PM +0100, Simon Riggs wrote: On 8 June 2013 15:30, Noah Misch n...@leadboat.com wrote: On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: 2. Don't store FK events in the after trigger queue at all, but apply them as we go. That solves problems2 and

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Robert Haas
On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch n...@leadboat.com wrote: This does appear to specify FK timing semantics like PostgreSQL gives today. Namely, it does not permit a FK-induced error when later actions of the query that prompted the check could possibly remedy the violation. Yeah.

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch n...@leadboat.com wrote: Likewise; I don't see why we couldn't perform an optimistic check ASAP and schedule a final after-statement check when an early check fails. That changes

Re: [HACKERS] Optimising Foreign Key checks

2013-06-05 Thread Greg Stark
On Sat, Jun 1, 2013 at 9:41 AM, Simon Riggs si...@2ndquadrant.com wrote: COMMIT; The inserts into order_line repeatedly execute checks against the same ordid. Deferring and then de-duplicating the checks would optimise the transaction. Proposal: De-duplicate multiple checks against same

Re: [HACKERS] Optimising Foreign Key checks

2013-06-05 Thread Hannu Krosing
On 06/05/2013 11:37 AM, Greg Stark wrote: On Sat, Jun 1, 2013 at 9:41 AM, Simon Riggs si...@2ndquadrant.com wrote: COMMIT; The inserts into order_line repeatedly execute checks against the same ordid. Deferring and then de-duplicating the checks would optimise the transaction. Proposal:

Re: [HACKERS] Optimising Foreign Key checks

2013-06-04 Thread Simon Riggs
On 4 June 2013 01:54, Noah Misch n...@leadboat.com wrote: On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: For clarity the 4 problems are 1. SQL execution overhead 2. Memory usage 3. Memory scrolling 4. Locking overhead, specifically FPWs and WAL records from FK checks probably

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Jim Nasby
On 6/2/13 4:45 AM, Simon Riggs wrote: Will this add too much cost where it doesn't help? I don't know what to predict there. There's the obvious case of trivial transactions with no more than one referential integrity check per FK, but there's also the case of a transaction with many FK checks

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Simon Riggs
On 3 June 2013 19:41, Jim Nasby j...@nasby.net wrote: On 6/2/13 4:45 AM, Simon Riggs wrote: Will this add too much cost where it doesn't help? I don't know what to predict there. There's the obvious case of trivial transactions with no more than one referential integrity check per FK, but

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Noah Misch
On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: For clarity the 4 problems are 1. SQL execution overhead 2. Memory usage 3. Memory scrolling 4. Locking overhead, specifically FPWs and WAL records from FK checks probably in that order or thereabouts. The above is why I went

Re: [HACKERS] Optimising Foreign Key checks

2013-06-02 Thread Simon Riggs
On 1 June 2013 21:27, Noah Misch n...@leadboat.com wrote: On Sat, Jun 01, 2013 at 09:41:13AM +0100, Simon Riggs wrote: FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here:

[HACKERS] Optimising Foreign Key checks

2013-06-01 Thread Simon Riggs
FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here: 1. Use Case: Bulk loading COPY pgbench_accounts; -- references pgbench_branches with many repeated values Proposal:

Re: [HACKERS] Optimising Foreign Key checks

2013-06-01 Thread Noah Misch
On Sat, Jun 01, 2013 at 09:41:13AM +0100, Simon Riggs wrote: FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here: 1. Use Case: Bulk loading COPY pgbench_accounts; --