[HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Simon Riggs
On 30 July 2015 at 12:26, Greg Stark st...@mit.edu wrote: On Thu, Jul 30, 2015 at 12:09 PM, Heikki Linnakangas hlinn...@iki.fi wrote: True, you need a heap to hold the next tuple from each tape in the merge step. To avoid exceeding work_mem, you'd need to push some tuples from the

Re: [HACKERS] [PATCH] Microvacuum for gist.

2015-07-30 Thread Alexander Korotkov
Hi! On Thu, Jul 30, 2015 at 2:51 PM, Anastasia Lubennikova lubennikov...@gmail.com wrote: I have written microvacuum support for gist access method. Briefly microvacuum includes two steps: 1. When search tells us that the tuple is invisible to all transactions it is marked LP_DEAD and page

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Tomas Vondra
Hi, On 07/30/2015 10:21 AM, Heikki Linnakangas wrote: On 05/25/2015 11:43 PM, Tomas Vondra wrote: There are 6 files attached, but only 0002-0006 are actually part of the multivariate statistics patch itself. All of these patches are huge. In order to review this in a reasonable amount of

Re: [HACKERS] 9.5a1 BUG FIX: pgbench negative latencies

2015-07-30 Thread Heikki Linnakangas
On 07/27/2015 03:43 PM, Fabien COELHO wrote: Under 9.5a1 pgbench -r negative latencies are reported on meta commands, probably as an oversight of 84f0ea3f. This patch ensures that now is reset on each loop inside doCustom. Applied, thanks! - Heikki -- Sent via pgsql-hackers mailing list

Re: [HACKERS] [COMMITTERS] pgsql: Row-Level Security Policies (RLS)

2015-07-30 Thread Dean Rasheed
On 30 July 2015 at 01:35, Joe Conway m...@joeconway.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2015 02:21 AM, Dean Rasheed wrote: While going through this, I spotted another issue --- in a DML query with additional non-target relations, such as UPDATE t1 .. FROM t2

[HACKERS] [PATCH] Microvacuum for gist.

2015-07-30 Thread Anastasia Lubennikova
Hi, I have written microvacuum support for gist access method. Briefly microvacuum includes two steps: 1. When search tells us that the tuple is invisible to all transactions it is marked LP_DEAD and page is marked as has dead tuples, 2. Then, when insert touches full page which has dead tuples

[HACKERS] 64-bit XIDs again

2015-07-30 Thread Alexander Korotkov
Hackers, I know there were already couple of threads about 64bit XIDs. http://www.postgresql.org/message-id/42cccfe9.9040...@intellilink.co.jp http://www.postgresql.org/message-id/4f6c0e13.3080...@wiesinger.com I read them carefully, but I didn't find all the arguments for 64bit XIDs mentioned.

[HACKERS] [BUG] CustomScan-custom_plans are not copied

2015-07-30 Thread Kouhei Kaigai
Hello, The attached patch fixes up my careless misses when custom_plans field was added. Please apply this fix. Thanks, -- NEC Business Creation Division / PG-Strom Project KaiGai Kohei kai...@ak.jp.nec.com pgsql-v9.5-fix-copy-custom-scan.patch Description:

Re: [HACKERS] creating extension including dependencies

2015-07-30 Thread Petr Jelinek
On 2015-07-27 15:18, Michael Paquier wrote: On Sun, Jul 26, 2015 at 1:01 AM, Petr Jelinek wrote: Yes that's what I meant by the change of checking order in the explanation above. I did that because I thought code would be more complicated otherwise, but apparently I was stupid... +In

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 05:57 PM, Alexander Korotkov wrote: On Thu, Jul 30, 2015 at 5:24 PM, Heikki Linnakangas hlinn...@iki.fi wrote: I think we should move to 64-bit XIDs in in-memory structs snapshots, proc array etc. And expand clog to handle 64-bit XIDs. But keep the xmin/xmax fields on heap pages

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-30 Thread Heikki Linnakangas
On 07/29/2015 09:35 PM, Andres Freund wrote: On 2015-07-29 20:23:24 +0300, Heikki Linnakangas wrote: Backend A has called LWLockWaitForVar(X) on a lock, and is now waiting on it. The lock holder releases the lock, and wakes up A. But before A wakes up and sees that the lock is free, another

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Joshua D. Drake
On 07/30/2015 08:04 AM, Simon Riggs wrote: There is a big downside to expanding xmin/xmax to 64 bits: it takes space. More space means more memory needed for caching, more memory bandwidth, more I/O, etc. My feeling is that the overhead will recede in time. Having a nice, simple

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/30/2015 07:14 AM, Simon Riggs wrote: On 30 July 2015 at 14:26, Alexander Korotkov a.korot...@postgrespro.ru Any thoughts? Do you think 64-bit XIDs worth it? The problem of freezing is painful, but not impossible, which is why we have

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Alexander Korotkov
On Thu, Jul 30, 2015 at 5:24 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples would be created with

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Simon Riggs
On 30 July 2015 at 14:26, Alexander Korotkov a.korot...@postgrespro.ru wrote: Any thoughts? Do you think 64-bit XIDs worth it? The problem of freezing is painful, but not impossible, which is why we have held out so long. The problem of very long lived snapshots is coming closer at the same

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples would be created with 64-bit XIDs. We can use free bits in t_infomask2 to distinguish old and new

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Simon Riggs
On 30 July 2015 at 15:24, Heikki Linnakangas hlinn...@iki.fi wrote: On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples would be created with 64-bit

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-30 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Kevin Grittner wrote: If you run `make installcheck` against a cluster with default_transaction_isolation = 'repeatable read' you get one failure: I am tempted to say that we should just disallow to run vacuum on a table containing a brin

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Tom Lane
Gavin Flower gavinflo...@archidevsys.co.nz writes: On 31/07/15 02:24, Heikki Linnakangas wrote: There is a big downside to expanding xmin/xmax to 64 bits: it takes space. More space means more memory needed for caching, more memory bandwidth, more I/O, etc. I think having a special case to

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Josh Berkus
On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: I think we should move to 64-bit XIDs in in-memory structs snapshots, proc array etc. And expand clog to handle 64-bit XIDs. But keep the xmin/xmax fields on heap pages at 32-bits, and add an epoch-like field to the page header so that

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Gavin Flower
On 31/07/15 02:24, Heikki Linnakangas wrote: On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples would be created with 64-bit XIDs. We can use free bits in

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Tomas Vondra
Hi, On 07/30/2015 06:58 PM, Heikki Linnakangas wrote: The problem with a threshold is that around that threshold, even a small change in the data set can drastically change the produced estimates. For example, imagine that we know from the stats that zip code implies city. But then someone

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-30 Thread Alvaro Herrera
Kevin Grittner wrote: If you run `make installcheck` against a cluster with default_transaction_isolation = 'repeatable read' you get one failure: I am tempted to say that we should just disallow to run vacuum on a table containing a brin index in a transaction-snapshot transaction. It is

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-07-30 Thread Robert Haas
On Sun, Jul 19, 2015 at 4:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: On 07/17/2015 04:36 PM, Jim Nasby wrote: I'm guessing it'd be really ugly/hard to support at least this GUC being multi-line? Mind you, multi-line GUCs would be useful otherwise, but we

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Arthur Silva
On Thu, Jul 30, 2015 at 5:31 PM, Gavin Flower gavinflo...@archidevsys.co.nz wrote: On 31/07/15 02:24, Heikki Linnakangas wrote: On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with

Re: [HACKERS] Improving test coverage of extensions with pg_dump

2015-07-30 Thread Michael Paquier
On Fri, Jul 31, 2015 at 6:41 AM, Andreas Karlsson andr...@proxel.se wrote: The comment is good, but what I personally do not like is that the path to the test suite is non-obvious and not self explanatory I would not expect src/test/tables_fk/t/ to test pg_dump and extensions. I find it to

Re: [HACKERS] [PATCH] Microvacuum for gist.

2015-07-30 Thread Jim Nasby
On 7/30/15 7:33 AM, Alexander Korotkov wrote: 2) Generating notices for every dead tuple would be too noisy. I suggest to replace notice with one of debug levels. + elog(NOTICE, gistkillitems. Mark Item Dead offnum %hd, blkno %d, offnum, BufferGetBlockNumber(buffer)); Even that seems like

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Peter Geoghegan
On Thu, Jul 30, 2015 at 12:00 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Hmm. You don't really need to merge the in-memory array into the tape, as you know that all the tuples in the in-memory must come after the tuples already on the tape. You can just return all the tuples from the tape

Re: [HACKERS] TAP tests are badly named

2015-07-30 Thread Andrew Dunstan
On 07/30/2015 12:40 PM, Andrew Dunstan wrote: We should describe test sets by what they test, not by how they test. TAP is a testing tool/protocol. The current set of tests we have test the programs in src/bin, and we should really name the test set by a name that reflects that, rather than

[HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-07-30 Thread Josh Berkus
Hackers, In guc.c, the maximum for wal_buffers is INT_MAX. However, wal_buffers is actually measured in 8KB buffers, not in bytes. This means that users are able to set wal_buffers 2GB. When the database is started, this can cause a core dump if the WAL offset is 2GB. Attached patch fixes

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Petr Jelinek
On 2015-07-30 23:23, Tom Lane wrote: Gavin Flower gavinflo...@archidevsys.co.nz writes: On 31/07/15 02:24, Heikki Linnakangas wrote: There is a big downside to expanding xmin/xmax to 64 bits: it takes space. More space means more memory needed for caching, more memory bandwidth, more I/O, etc.

Re: [HACKERS] [PATCH] Reload SSL certificates on SIGHUP

2015-07-30 Thread Andreas Karlsson
On 07/29/2015 03:24 AM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: I don't have a problem with rebuilding the SSL context on every reload cycle. We already do a lot of extra reloading every time, so a bit more shouldn't hurt. But I'm not so sure whether we should do that in the

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Peter Geoghegan
On Thu, Jul 30, 2015 at 11:32 AM, Robert Haas robertmh...@gmail.com wrote: Very interesting. And great performance numbers. Thanks for taking the time to investigate this - really cool. Thanks. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Peter Geoghegan
On Thu, Jul 30, 2015 at 3:47 AM, Simon Riggs si...@2ndquadrant.com wrote: This is a good optimization for the common case where tuples are mostly already in order. We could increase the usefulness of this by making UPDATE pick blocks that are close to a tuple's original block, rather than

Re: [HACKERS] Updatable view?

2015-07-30 Thread Jim Nasby
On 7/30/15 1:03 AM, Tatsuo Ishii wrote: Is anyone working on implementing or interested in implementing automatic updatable view which uses two or more tables involved (join)? SQL1999 allows it in certain conditions. I think it would be nice to have... but not to the point of working on it

Re: [HACKERS] Proposal: backend niceness / session_priority

2015-07-30 Thread Jim Nasby
On 7/30/15 10:54 AM, Tom Lane wrote: =?ISO-8859-15?Q?Jos=E9_Luis_Tall=F3n?= jltal...@adv-solutions.net writes: Since PostgreSQL lacks the resource management capabilities of the Big Ones ( Resource Groups - Red, WorkLoad Manager - Blue ) or the Resource Governor in MS SQL Server, we can

[HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Peter Geoghegan
On Thu, Jul 30, 2015 at 4:26 AM, Greg Stark st...@mit.edu wrote: I'm a bit confused where the big win comes from though. Is what's going on that the external sort only exceeded memory by a small amount so nearly all the tuples are still in memory? Yes, that's why this can be much faster just

Re: [HACKERS] Improving test coverage of extensions with pg_dump

2015-07-30 Thread Andreas Karlsson
On 07/30/2015 04:48 AM, Michael Paquier wrote: On Thu, Jul 30, 2015 at 5:54 AM, Andreas Karlsson andr...@proxel.se wrote: What I do not like though is how the path src/test/tables_fk/t/ tells us nothing about what features are of PostgreSQL are tested here. For this I personally prefer the

Re: [HACKERS] creating extension including dependencies

2015-07-30 Thread Michael Paquier
On Thu, Jul 30, 2015 at 10:58 PM, Petr Jelinek wrote: On 2015-07-27 15:18, Michael Paquier wrote: Something also has not been discussed yet: what to do with new_version and old_version (the options of CreateExtensionStmt)? As of now if those options are defined they are not passed down to the

Re: [HACKERS] Proposal: backend niceness / session_priority

2015-07-30 Thread Andreas Karlsson
On 07/31/2015 12:18 AM, Jim Nasby wrote: This has been proposed before, and rejected before, and I'm not seeing anything particularly new here. Without a credible mechanism for throttling I/O, nice alone does not seem very promising. Some OSes respect nice when it comes to IO scheduling, so

Re: [HACKERS] Reduce ProcArrayLock contention

2015-07-30 Thread Amit Kapila
On Wed, Jul 29, 2015 at 11:48 PM, Andres Freund and...@anarazel.de wrote: On 2015-07-29 12:54:59 -0400, Robert Haas wrote: I would try to avoid changing lwlock.c. It's pretty easy when so doing to create mechanisms that work now but make further upgrades to the general lwlock mechanism

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-30 Thread Rahila Syed
I think it's related to the problem of figuring out how many dead tuples you expect to find in the overall heap, which you need to do to have any hope of this being a comprehensive estimate. An estimate of number of index scans while vacuuming can be done using estimate of total dead tuples in

Re: [HACKERS] Updatable view?

2015-07-30 Thread Tatsuo Ishii
I think it would be nice to have... but not to the point of working on it myself. Might be worth an email to -general to see how many people have immediate use for it. What I am thinking about is, 1) Implement certain class of updatable views allowed in SQL:1999 (UNION ALL, natural

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-30 Thread Fabrízio de Royes Mello
On Thu, Jul 30, 2015 at 11:28 PM, Andres Freund and...@anarazel.de wrote: @@ -57,7 +57,8 @@ static relopt_bool boolRelOpts[] = If we go through this list, I'd rather make informed decisions about each reloption. Otherwise we're going to get patches for each of them separately over the next

Re: [HACKERS] security labels on databases are bad for dump restore

2015-07-30 Thread Adam Brightwell
1. pg_dumpall -g 2. pg_dump --create per database Gah, OK, I see your point. But we better document this, because if you need a PhD in PostgreSQL-ology to take a backup, we're not in a good place. Agreed. Though, honestly, I find this to be a cumbersome approach. I think it just makes

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-30 Thread Michael Paquier
On Fri, Jul 31, 2015 at 11:28 AM, Andres Freund and...@anarazel.de wrote: @@ -57,7 +57,8 @@ static relopt_bool boolRelOpts[] = If we go through this list, I'd rather make informed decisions about each reloption. Otherwise we're going to get patches for each of them separately over the next

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-30 Thread Fabrízio de Royes Mello
On Thu, Jul 30, 2015 at 10:46 PM, Michael Paquier michael.paqu...@gmail.com wrote: This patch size has increased from 16k to 157k because of the output of the isolation tests you just added. This is definitely too large and actually the test coverage is rather limited. Hence I think that they

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-30 Thread Andres Freund
@@ -57,7 +57,8 @@ static relopt_bool boolRelOpts[] = If we go through this list, I'd rather make informed decisions about each reloption. Otherwise we're going to get patches for each of them separately over the next versions. @@ -73,7 +75,8 @@ static relopt_bool boolRelOpts[] =

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-30 Thread Andres Freund
On 2015-07-30 09:03:01 -0700, Jeff Janes wrote: On Wed, Jul 29, 2015 at 6:10 AM, Andres Freund and...@anarazel.de wrote: What do you think about something roughly like the attached? I've not evaluated the code, but applying it does solve the problem I was seeing. Cool, thanks for

[HACKERS] Updatable view?

2015-07-30 Thread Tatsuo Ishii
Hi, Is anyone working on implementing or interested in implementing automatic updatable view which uses two or more tables involved (join)? SQL1999 allows it in certain conditions. Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php

Re: [HACKERS] security labels on databases are bad for dump restore

2015-07-30 Thread Noah Misch
On Wed, Jul 29, 2015 at 10:50:53AM -0400, Robert Haas wrote: On Wed, Jul 29, 2015 at 12:39 AM, Noah Misch n...@leadboat.com wrote: On Tue, Jul 28, 2015 at 03:36:13PM -0400, Robert Haas wrote: On Tue, Jul 28, 2015 at 3:33 PM, Andres Freund and...@anarazel.de wrote: Hm? Let me try again: If

Re: [HACKERS] Don'st start streaming after creating a slot in pg_receivexlog

2015-07-30 Thread Michael Paquier
On Wed, Jul 29, 2015 at 10:20 PM, Andres Freund and...@anarazel.de wrote: On 2015-07-29 22:17:27 +0900, Michael Paquier wrote: Here is a patch implementing those things. IMO if-not-exists does not make much sense anymore What? It's rather useful to be able to discern between 'slot was already

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Heikki Linnakangas
On 05/25/2015 11:43 PM, Tomas Vondra wrote: There are 6 files attached, but only 0002-0006 are actually part of the multivariate statistics patch itself. All of these patches are huge. In order to review this in a reasonable amount of time, we need to do this in several steps. So let's see

Re: [HACKERS] dblink: add polymorphic functions.

2015-07-30 Thread Tom Lane
Joe Conway m...@joeconway.com writes: What about just TYPE then? SELECT x::TYPE(some_expression) FROM ... SELECT CAST (x AS TYPE(some_expression)) FROM ... Yeah, that would work. Quick-hack proof-of-concept patch attached. Some usage examples in the regression database: regression=#

Re: [HACKERS] CREATE FUNCTION .. LEAKPROOF docs

2015-07-30 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/29/2015 04:46 PM, Joe Conway wrote: On 06/14/2015 03:46 AM, Dean Rasheed wrote: I think the docs for the LEAKPROOF option in create_function.sgml ought to mention RLS as well as security barrier views. Also the current text is no longer

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-30 Thread Jeff Janes
On Wed, Jul 29, 2015 at 6:10 AM, Andres Freund and...@anarazel.de wrote: On 2015-07-29 14:22:23 +0200, Andres Freund wrote: On 2015-07-29 15:14:23 +0300, Heikki Linnakangas wrote: Ah, ok, that should work, as long as you also re-check the variable's value after queueing. Want to write

[HACKERS] TAP tests are badly named

2015-07-30 Thread Andrew Dunstan
We should describe test sets by what they test, not by how they test. TAP is a testing tool/protocol. The current set of tests we have test the programs in src/bin, and we should really name the test set by a name that reflects that, rather than the fact that we are using TAP tools to run

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-30 Thread Andres Freund
On 2015-07-30 17:36:52 +0300, Heikki Linnakangas wrote: In 9.4, LWLockAcquire holds the spinlock when it marks the lock as held, until it has updated the variable. And LWLockWaitForVar() holds the spinlock when it checks that the lock is held and that the variable's value matches. So it cannot

Re: [HACKERS] Volatility of pg_xact_commit_timestamp() and pg_last_committed_xact()

2015-07-30 Thread Alvaro Herrera
Alvaro Herrera wrote: Robert Haas wrote: On Thu, Jul 16, 2015 at 9:49 AM, Fujii Masao masao.fu...@gmail.com wrote: Volatilities of pg_xact_commit_timestamp() and pg_last_committed_xact() are now STABLE. But ISTM that those functions can return different results even within a single

Re: [HACKERS] security labels on databases are bad for dump restore

2015-07-30 Thread Robert Haas
On Thu, Jul 30, 2015 at 2:49 AM, Noah Misch n...@leadboat.com wrote: What exact formula did you have in mind? It must not be merely 1. pg_dumpall -g 2. pg_dump (without --create) per database which _never_ works: it emits no CREATE DATABASE statements. Perhaps this? 1. pg_dumpall -g 2.

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Robert Haas
On Wed, Jul 29, 2015 at 9:05 PM, Peter Geoghegan p...@heroku.com wrote: The behavior of external sorts that do not require any merge step due to only having one run (what EXPLAIN ANALYZE output shows as an external sort, and not a merge sort) seems like an area that can be significantly

Re: [HACKERS] Reduce ProcArrayLock contention

2015-07-30 Thread Robert Haas
On Wed, Jul 29, 2015 at 2:18 PM, Andres Freund and...@anarazel.de wrote: On 2015-07-29 12:54:59 -0400, Robert Haas wrote: I would try to avoid changing lwlock.c. It's pretty easy when so doing to create mechanisms that work now but make further upgrades to the general lwlock mechanism

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 04:05 AM, Peter Geoghegan wrote: Patch -- quicksort with spillover = With the attached patch, I propose to add an additional, better one run special case optimization. This new special case splits the single run into 2 subruns. One of the runs is comprised

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-30 Thread Andres Freund
On 2015-07-29 20:23:24 +0300, Heikki Linnakangas wrote: Backend A has called LWLockWaitForVar(X) on a lock, and is now waiting on it. The lock holder releases the lock, and wakes up A. But before A wakes up and sees that the lock is free, another backend acquires the lock again. It runs

Re: [HACKERS] Reduce ProcArrayLock contention

2015-07-30 Thread Andres Freund
On 2015-07-29 12:54:59 -0400, Robert Haas wrote: I would try to avoid changing lwlock.c. It's pretty easy when so doing to create mechanisms that work now but make further upgrades to the general lwlock mechanism difficult. I'd like to avoid that. I'm massively doubtful that re-implementing

Re: [HACKERS] security labels on databases are bad for dump restore

2015-07-30 Thread Alvaro Herrera
Noah Misch wrote: What exact formula did you have in mind? It must not be merely 1. pg_dumpall -g 2. pg_dump (without --create) per database which _never_ works: it emits no CREATE DATABASE statements. Perhaps this? 1. pg_dumpall -g 2. Issue a handwritten CREATE DATABASE statement

Re: [HACKERS] Proposal: backend niceness / session_priority

2015-07-30 Thread Tom Lane
=?ISO-8859-15?Q?Jos=E9_Luis_Tall=F3n?= jltal...@adv-solutions.net writes: Since PostgreSQL lacks the resource management capabilities of the Big Ones ( Resource Groups - Red, WorkLoad Manager - Blue ) or the Resource Governor in MS SQL Server, we can try and approximate the requested

Re: [HACKERS] pgbench stats per script other stuff

2015-07-30 Thread Fabien COELHO
v6 is just a rebase after a bug fix by Andres Freund. Also a small question: The patch currently displays pgbench scripts starting numbering at 0. Probably a little too geek... should start at 1? v7 is a rebase after another small bug fix in pgbench. -- Fabien.diff --git

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 03:55 PM, Tomas Vondra wrote: On 07/30/2015 10:21 AM, Heikki Linnakangas wrote: I have some doubts about the clause reduction and functional dependencies part of this. It seems to treat functional dependency as a boolean property, but even with the classic zipcode and city case,

[HACKERS] Proposal: backend niceness / session_priority

2015-07-30 Thread José Luis Tallón
Hackers, I have found myself needing to run some maintenance routines (VACUUM, REINDEX, REFRESH MATERIALIZED VIEW mostly) at a lower priority so as not to disturb concurrent *highly transactional* connections. This issue is also noted within the TODO[0] list in the Wiki . * There was

Re: [HACKERS] Division by zero in selfuncs.c:estimate_hash_bucketsize()

2015-07-30 Thread Tom Lane
Piotr Stefaniak postg...@piotr-stefaniak.me writes: the two asserts below will fail with this query (ran against the regression db): I've applied fixes for this and the other thing. Thanks for the report! regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] [COMMITTERS] pgsql: Row-Level Security Policies (RLS)

2015-07-30 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/30/2015 06:52 AM, Dean Rasheed wrote: On 30 July 2015 at 01:35, Joe Conway m...@joeconway.com wrote: On 06/01/2015 02:21 AM, Dean Rasheed wrote: While going through this, I spotted another issue --- in a DML query with additional non-target

Re: [HACKERS] dblink: add polymorphic functions.

2015-07-30 Thread Alvaro Herrera
Tom Lane wrote: Joe Conway m...@joeconway.com writes: What about just TYPE then? SELECT x::TYPE(some_expression) FROM ... SELECT CAST (x AS TYPE(some_expression)) FROM ... Yeah, that would work. Quick-hack proof-of-concept patch attached. I'm amazed that this works without

Re: [HACKERS] Planner debug views

2015-07-30 Thread Jim Nasby
On 7/29/15 2:40 PM, Alvaro Herrera wrote: Qingqing Zhou wrote: Can we simplify above with foreign table methods? There are two major concerns about this method per previous discussions: security and usability. I think the main cause is the sharing foreign table design. I think foreign data

Re: [HACKERS] dblink: add polymorphic functions.

2015-07-30 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: Yeah, that would work. Quick-hack proof-of-concept patch attached. This is a rather ugly, but I guess not untenable. I suppose we don't care about any actual typmod, do we?. We get the type and typmod both from the expression.

Re: [HACKERS] Failing assertions in indxpath.c, placeholder.c and brin_minmax.c

2015-07-30 Thread Alvaro Herrera
Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: Bottom line is that somebody failed to consider the possibility of a null comparison value reaching the BRIN index lookup machinery. The code stanza that's failing supposes that only IS NULL or IS NOT NULL

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-30 Thread Michael Paquier
On Fri, Jul 31, 2015 at 8:30 AM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Fri, Jul 24, 2015 at 4:05 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Fri, Jul 24, 2015 at 7:11 AM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Thu, Jul 2, 2015 at 2:03 PM,

Re: [HACKERS] Remaining 'needs review' patchs in July commitfest

2015-07-30 Thread Robert Haas
On Wed, Jul 29, 2015 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jul 28, 2015 at 3:51 PM, Heikki Linnakangas hlinn...@iki.fi wrote: plpgsql raise statement with context Impasse. Everyone wants this feature in some form, but no consensus on

Re: [HACKERS] pg_dump quietly ignore missing tables - is it bug?

2015-07-30 Thread Heikki Linnakangas
On 07/25/2015 07:08 PM, Pavel Stehule wrote: I am sending a new patch - without checking wildcard chars. The documentation says the option is called --strict-names, while the code has --strict-mode. I like --strict-names more, mode seems redundant, and it's not clear what it's strict about.

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Simon Riggs
On 30 July 2015 at 08:00, Heikki Linnakangas hlinn...@iki.fi wrote: Hmm. You don't really need to merge the in-memory array into the tape, as you know that all the tuples in the in-memory must come after the tuples already on the tape. You can just return all the tuples from the tape first,

Re: [HACKERS] Autonomous Transaction is back

2015-07-30 Thread Rajeev rastogi
On 28 July 2015 03:21, Josh Berkus Wrote: On 07/27/2015 02:47 AM, Rajeev rastogi wrote: Why have any fixed maximum? Since we are planning to have nested autonomous transaction, so it is required to have limit on this so that resources can be controlled. Is there a particular reason why this

Re: [HACKERS] Configurable location for extension .control files

2015-07-30 Thread Heikki Linnakangas
On 07/02/2015 11:37 PM, Oskari Saarenmaa wrote: I'm somewhat interested in both #1 #2 for other projects, but I wrote this patch to address #3, i.e. to simplify the test setup we have in place for pgmemcache (https://github.com/ohmu/pgmemcache/blob/master/localtests.sh) and other extensions.

Re: [HACKERS] Foreign join pushdown vs EvalPlanQual

2015-07-30 Thread Etsuro Fujita
On 2015/07/07 19:15, Etsuro Fujita wrote: On 2015/07/06 9:42, Kouhei Kaigai wrote: However, we need to pay attention on advantages towards the alternatives. Hooks on add_paths_to_joinrel() enables to implement identical stuff, with less complicated logic to reproduce left / right relations from

Re: [HACKERS] We need to support ForeignRecheck for late row locking, don't we?

2015-07-30 Thread Etsuro Fujita
On 2015/07/27 18:16, Kouhei Kaigai wrote: On 2015/07/24 23:51, Kouhei Kaigai wrote: On 2015/07/22 19:10, Etsuro Fujita wrote: While working on the issue Foreign join pushdown vs EvalPlanQual, I happened to notice odd behaviors of late row locking in FDWs. I think the reason for that is

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 01:47 PM, Simon Riggs wrote: On 30 July 2015 at 08:00, Heikki Linnakangas hlinn...@iki.fi wrote: So here's a shorter/different explanation of this optimization: When it's time to perform the sort, instead of draining the in-memory heap one tuple at a time to the last tape, you

[HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-07-30 Thread Greg Stark
On Thu, Jul 30, 2015 at 12:09 PM, Heikki Linnakangas hlinn...@iki.fi wrote: True, you need a heap to hold the next tuple from each tape in the merge step. To avoid exceeding work_mem, you'd need to push some tuples from the in-memory array to the tape to make room for that. In practice,