[HACKERS] Many processes blocked at ProcArrayLock

2014-12-02 Thread Xiaoyulei
Test configuration: Hardware: 4P intel server, 60 core 120 hard thread. Memory:512G SSD:2.4T PG: max_connections = 160 # (change requires restart) shared_buffers = 32GB work_mem = 128MB

Re: [HACKERS] using Core Foundation locale functions

2014-12-02 Thread Peter Geoghegan
On Fri, Nov 28, 2014 at 8:43 AM, Peter Eisentraut pete...@gmx.net wrote: At the moment, this is probably just an experiment that shows where refactoring and better abstractions might be suitable if we want to support multiple locale libraries. If we want to pursue ICU, I think this could be a

Re: [HACKERS] excessive amounts of consumed memory (RSS), triggering OOM killer

2014-12-02 Thread Tomas Vondra
Dne 2014-12-02 02:52, Tom Lane napsal: Tomas Vondra t...@fuzzy.cz writes: On 2.12.2014 01:33, Tom Lane wrote: What I suspect you're looking at here is the detritus of creation of a huge number of memory contexts. mcxt.c keeps its own state about existing contents in TopMemoryContext. So, if we

[HACKERS] Serialization exception : Who else was involved?

2014-12-02 Thread Olivier MATROT
Hello, I'm using PostgreSQL .9.2.8 on Windows from a .NET application using Npgsql. I'm working in the Radiology Information System field. We have thousands of users against a big accounting database. We're using the SERIALIZABLE isolation level to ensure data consistency. Because of

Re: [HACKERS] Role Attribute Bitmask Catalog Representation

2014-12-02 Thread Stephen Frost
Adam, * Adam Brightwell (adam.brightw...@crunchydatasolutions.com) wrote: Ok. Though, this would affect how CATUPDATE is handled. Peter Eisentraut previously raised a question about whether superuser checks should be included with catupdate which led me to create the following post.

Re: [HACKERS] excessive amounts of consumed memory (RSS), triggering OOM killer

2014-12-02 Thread Tomas Vondra
Dne 2 Prosinec 2014, 10:59, Tomas Vondra napsal(a): Dne 2014-12-02 02:52, Tom Lane napsal: Tomas Vondra t...@fuzzy.cz writes: Also, this explains the TopMemoryContext size, but not the RSS size (or am I missing something)? Very possibly you're left with islands that prevent reclaiming very

[HACKERS] pg_stat_statement normalization fails due to temporary tables

2014-12-02 Thread Andres Freund
Hi, pg_stat_statement's query normalization fails when temporary tables are used in a query. That's because JumbleRangeTable() uses the relid in the RTE to build the query fingerprint. I think in this case pgss from 9.1 actually would do better than 9.2+ as the hash lookup previously didn't use

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-02 Thread Alex Shulgin
Michael Paquier michael.paqu...@gmail.com writes: On Tue, Dec 2, 2014 at 1:58 AM, Alex Shulgin a...@commandprompt.com wrote: Here's the patch rebased against current HEAD, that is including the recently committed action_at_recovery_target option. If this patch gets in, it gives a good

Re: [HACKERS] [v9.5] Custom Plan API

2014-12-02 Thread Robert Haas
On Tue, Nov 25, 2014 at 3:44 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Today, I had a talk with Hanada-san to clarify which can be a common portion of them and how to implement it. Then, we concluded both of features can be shared most of the infrastructure. Let me put an introduction of

Re: [HACKERS] pg_stat_statement normalization fails due to temporary tables

2014-12-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: pg_stat_statement's query normalization fails when temporary tables are used in a query. That's because JumbleRangeTable() uses the relid in the RTE to build the query fingerprint. I think in this case pgss from 9.1 actually would do better than

Re: [HACKERS] pg_stat_statement normalization fails due to temporary tables

2014-12-02 Thread Andres Freund
On 2014-12-02 09:59:00 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: pg_stat_statement's query normalization fails when temporary tables are used in a query. That's because JumbleRangeTable() uses the relid in the RTE to build the query fingerprint. I think in this

Re: [HACKERS] test_shm_mq failing on mips*

2014-12-02 Thread Robert Haas
On Tue, Nov 25, 2014 at 10:42 AM, Christoph Berg c...@df7cb.de wrote: Re: To Robert Haas 2014-11-24 20141124200824.ga22...@msg.df7cb.de Does it fail every time when run on a machine where it fails sometimes? So far there's a consistent host - fail-or-not mapping, but most mips/mipsel build

Re: [HACKERS] Nitpicky doc corrections for BRIN functions of pageinspect

2014-12-02 Thread Alvaro Herrera
Peter Geoghegan wrote: On Wed, Nov 19, 2014 at 8:02 PM, Michael Paquier michael.paqu...@gmail.com wrote: Just a small thing I noticed while looking at pageinspect.sgml, the set of SQL examples related to BRIN indexes uses lower-case characters for reserved keywords. This has been

Re: [HACKERS] On partitioning

2014-12-02 Thread Robert Haas
On Tue, Nov 25, 2014 at 8:20 PM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: Before going too much further with this I'd mock up schemas for your proposed catalogs and a list of DDL operations to be supported, with the corresponding syntax, and float that here for comment. More people

Re: [HACKERS] superuser() shortcuts

2014-12-02 Thread Robert Haas
On Wed, Nov 26, 2014 at 10:12 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: In the context at hand, I think most of the messages in question are currently phrased like must be superuser to do X. I'd be fine with changing that to permission denied to do X,

Re: [HACKERS] 9.2 recovery/startup problems

2014-12-02 Thread Robert Haas
On Wed, Nov 26, 2014 at 7:13 PM, Jeff Janes jeff.ja...@gmail.com wrote: If I do a pg_ctl stop -mf, then both files go away. If I do a pg_ctl stop -mi, then neither goes away. It is only with the /sbin/reboot that I get the fatal combination of _init being gone but the other still present.

Re: [HACKERS] memory explosion on planning complex query

2014-12-02 Thread Robert Haas
On Wed, Nov 26, 2014 at 7:24 PM, Andrew Dunstan and...@dunslane.net wrote: On 11/26/2014 05:00 PM, Andrew Dunstan wrote: Attached is some anonymized DDL for a fairly complex schema from a PostgreSQL Experts client. Also attached is an explain query that runs against the schema. The client's

Re: [HACKERS] dblink_get_connections() result for no connections

2014-12-02 Thread Robert Haas
On Tue, Nov 25, 2014 at 3:49 PM, Tom Lane t...@sss.pgh.pa.us wrote: While fooling around with the array_agg(anyarray) patch, I noted that dblink_get_connections() returns NULL if there are no active connections. It seems like an empty array might be a saner definition --- thoughts? I don't

Re: [HACKERS] copy.c handling for RLS is insecure

2014-12-02 Thread Robert Haas
On Thu, Nov 27, 2014 at 2:03 AM, Stephen Frost sfr...@snowman.net wrote: Alright, I've done the change to use the RangeVar from CopyStmt, but also added a check wherein we verify that the relation's OID returned from the planned query is the same as the relation's OID that we did the RLS check

Re: [HACKERS] test_shm_mq failing on mips*

2014-12-02 Thread Christoph Berg
Re: Robert Haas 2014-12-02 CA+Tgmoa9HE=1GObU7MKuGLDBjBNSNRW0bDE4P3JA=P=9mqg...@mail.gmail.com I can't tell from this exactly what's going wrong. Questions: 1. Are there any background worker processes running at the same time? If so, how many? We'd expect to see 3. 2. Can we printout of

Re: [HACKERS] initdb: Improve error recovery.

2014-12-02 Thread Robert Haas
On Thu, Nov 27, 2014 at 7:28 AM, Mats Erik Andersson b...@gisladisker.se wrote: I would like to improve error recovery of initdb when the password file is empty. The present code declares Error 0 as the cause of failure. It suffices to use ferrer() since fgets() returns NULL also at a

Re: [HACKERS] superuser() shortcuts

2014-12-02 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Wed, Nov 26, 2014 at 10:12 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: In the context at hand, I think most of the messages in question are currently phrased like must be superuser to do X. I'd be fine

Re: [HACKERS] copy.c handling for RLS is insecure

2014-12-02 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Thu, Nov 27, 2014 at 2:03 AM, Stephen Frost sfr...@snowman.net wrote: Alright, I've done the change to use the RangeVar from CopyStmt, but also added a check wherein we verify that the relation's OID returned from the planned query is the same

Re: [HACKERS] superuser() shortcuts

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 11:29 AM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Wed, Nov 26, 2014 at 10:12 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: In the context at hand, I think most of the messages in

Re: [HACKERS] Add CREATE support to event triggers

2014-12-02 Thread Robert Haas
Normally you would replicate between an older master and a newer replica, so this shouldn't be an issue. I find it unlikely that we would de-support some syntax that works in an older version: it would break pg_dump, for one thing. The most common way in which we break forward-compatibility

Re: [HACKERS] [PATCH] HINT: pg_hba.conf changed since last config reload

2014-12-02 Thread Robert Haas
On Thu, Nov 27, 2014 at 8:49 AM, Bruce Momjian br...@momjian.us wrote: On Thu, Nov 6, 2014 at 05:46:42PM -0500, Peter Eisentraut wrote: Finally, the fact that a configuration change is in progress is privileged information. Unprivileged users can deduct from the presence of this message that

Re: [HACKERS] 9.2 recovery/startup problems

2014-12-02 Thread Jeff Janes
On Tue, Dec 2, 2014 at 7:41 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 26, 2014 at 7:13 PM, Jeff Janes jeff.ja...@gmail.com wrote: If I do a pg_ctl stop -mf, then both files go away. If I do a pg_ctl stop -mi, then neither goes away. It is only with the /sbin/reboot that I

Re: [HACKERS] why is PG_AUTOCONF_FILENAME is pg_config_manual.h?

2014-12-02 Thread Robert Haas
On Thu, Nov 27, 2014 at 12:56 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Nov 27, 2014 at 11:58 PM, Peter Eisentraut pete...@gmx.net wrote: Surely that's not a value that we expect users to be able to edit. Is pg_config_manual.h just abused as a place that's included everywhere? (I

[HACKERS] Report search_path value back to the client.

2014-12-02 Thread Alexander Kukushkin
Hi, As of now postgres is reporting only really important variables, but among them there are also not so important, like 'application_name'. The only reason to report it, was: help pgbouncer, and perhaps other connection poolers. Change was introduced by commit:

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-02 Thread Robert Haas
On Wed, Nov 26, 2014 at 11:00 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Nov 26, 2014 at 8:27 PM, Syed, Rahila rahila.s...@nttdata.com wrote: Don't we need to initialize doPageCompression similar to doPageWrites in InitXLOGAccess? Yep, you're right. I missed this code path.

Re: [HACKERS] Testing DDL deparsing support

2014-12-02 Thread Robert Haas
On Thu, Nov 27, 2014 at 11:43 PM, Ian Barwick i...@2ndquadrant.com wrote: DDL deparsing is a feature that allows collection of DDL commands as they are executed in a server, in some flexible, complete, and fully-contained format that allows manipulation, storage, and transmission. This

Re: [HACKERS] Using pg_rewind for differential backup

2014-12-02 Thread Robert Haas
On Fri, Nov 28, 2014 at 2:49 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It also would be quite straightforward to write a separate tool to do just that. Would be better than conflating pg_rewind with this. You could use pg_rewind as the basis for it - it's under the same license as

Re: [HACKERS] Fillfactor for GIN indexes

2014-12-02 Thread Robert Haas
On Fri, Nov 28, 2014 at 4:27 AM, Alexander Korotkov aekorot...@gmail.com wrote: On Fri, Nov 21, 2014 at 8:12 AM, Michael Paquier michael.paqu...@gmail.com wrote: Please find attached a simple patch adding fillfactor as storage parameter for GIN indexes. The default value is the same as the one

Re: [HACKERS] patch : Allow toast tables to be moved to a different tablespace

2014-12-02 Thread Robert Haas
On Fri, Nov 28, 2014 at 11:38 AM, Alex Shulgin a...@commandprompt.com wrote: Should I add this patch to the next CommitFest? If you don't want it to get forgotten about, yes. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers

Re: [HACKERS] Add shutdown_at_recovery_target option to recovery.conf

2014-12-02 Thread Robert Haas
On Fri, Nov 28, 2014 at 11:59 AM, Petr Jelinek p...@2ndquadrant.com wrote: I'm a bit late to the party, but wouldn't recovery_target_action = ... have been a better name for this? It'd be in line with the other recovery_target_* parameters, and also a bit shorter than the imho somewhat ugly

Re: [HACKERS] Testing DDL deparsing support

2014-12-02 Thread Alvaro Herrera
Robert Haas wrote: On Thu, Nov 27, 2014 at 11:43 PM, Ian Barwick i...@2ndquadrant.com wrote: A simple schedule to demonstrate this is available; execute from the src/test/regress/ directory like this: ./pg_regress \ --temp-install=./tmp_check \ --top-builddir=../../..

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-02 Thread Josh Berkus
On 12/02/2014 06:25 AM, Alex Shulgin wrote: I am not in favor of this part. It may be better to let the users know that their old configuration is not valid anymore with an error. This patch cuts in the flesh with a huge axe, let's be sure that users do not ignore the side pain effects, or

Re: [HACKERS] Review of GetUserId() Usage

2014-12-02 Thread Robert Haas
On Sat, Nov 29, 2014 at 3:00 PM, Stephen Frost sfr...@snowman.net wrote: * Stephen Frost (sfr...@snowman.net) wrote: * Stephen Frost (sfr...@snowman.net) wrote: Attached is a patch to address the pg_cancel/terminate_backend and the statistics info as discussed previously. It sounds like

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Robert Haas
On Sat, Nov 29, 2014 at 11:46 PM, Jim Nasby jim.na...@bluetreble.com wrote: What do you mean by never succeed? Is it skipping a large number of pages? Might re-trying the locks within the same vacuum help, or are the user locks too persistent? You are confused. He's talking about the

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-02 Thread Alvaro Herrera
Josh Berkus wrote: On 12/02/2014 06:25 AM, Alex Shulgin wrote: Whatever tricks we might employ will likely be defeated by the fact that the oldschool user will fail to *include* recovery.conf in the main conf file. Well, can we merge this patch and then fight out what to do for the

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Alvaro Herrera
Robert Haas wrote: On Sat, Nov 29, 2014 at 11:46 PM, Jim Nasby jim.na...@bluetreble.com wrote: What do you mean by never succeed? Is it skipping a large number of pages? Might re-trying the locks within the same vacuum help, or are the user locks too persistent? You are confused. He's

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-02 Thread Josh Berkus
On 12/02/2014 10:31 AM, Alvaro Herrera wrote: Josh Berkus wrote: On 12/02/2014 06:25 AM, Alex Shulgin wrote: Whatever tricks we might employ will likely be defeated by the fact that the oldschool user will fail to *include* recovery.conf in the main conf file. Well, can we merge this

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Josh Berkus
On 12/02/2014 10:35 AM, Alvaro Herrera wrote: If the table is large, the time window for this to happen is large also; there might never be a time window large enough between two lock acquisitions for one autovacuum run to complete in a table. This starves the table from vacuuming completely,

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Andres Freund
On 2014-12-02 11:02:07 -0800, Josh Berkus wrote: On 12/02/2014 10:35 AM, Alvaro Herrera wrote: If the table is large, the time window for this to happen is large also; there might never be a time window large enough between two lock acquisitions for one autovacuum run to complete in a

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Josh Berkus
On 12/02/2014 11:08 AM, Andres Freund wrote: On 2014-12-02 11:02:07 -0800, Josh Berkus wrote: On 12/02/2014 10:35 AM, Alvaro Herrera wrote: If the table is large, the time window for this to happen is large also; there might never be a time window large enough between two lock acquisitions

Re: [HACKERS] Removing INNER JOINs

2014-12-02 Thread Robert Haas
On Sun, Nov 30, 2014 at 12:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bottom line, given all the restrictions on whether the optimization can happen, I have very little enthusiasm for the whole idea. I do not think the benefit will be big enough to justify the amount of mess this will

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Andres Freund
On 2014-12-02 11:12:40 -0800, Josh Berkus wrote: On 12/02/2014 11:08 AM, Andres Freund wrote: On 2014-12-02 11:02:07 -0800, Josh Berkus wrote: On 12/02/2014 10:35 AM, Alvaro Herrera wrote: If the table is large, the time window for this to happen is large also; there might never be a time

Re: [HACKERS] Sequence Access Method WIP

2014-12-02 Thread Andres Freund
On 2014-11-24 13:16:24 +0200, Heikki Linnakangas wrote: To be clear: I don't think this API is very good for its stated purpose, for implementing global sequences for use in a cluster. For the reasons I've mentioned before. I'd like to see two changes to this proposal: 1. Make the AM

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Jeff Janes
On Tue, Dec 2, 2014 at 11:12 AM, Josh Berkus j...@agliodbs.com wrote: On 12/02/2014 11:08 AM, Andres Freund wrote: On 2014-12-02 11:02:07 -0800, Josh Berkus wrote: On 12/02/2014 10:35 AM, Alvaro Herrera wrote: If the table is large, the time window for this to happen is large also;

Re: [HACKERS] Getting references for OID

2014-12-02 Thread Borodin Vladimir
Hello, How can I get all depend objects for oid of object? For example, I have oid of db db_id and I want to get all oids of namespaces, which contains this db. Thank you! Hello, Dmitry. Actually, this list is for developers (as described here [0]). You should ask this question on

Re: [HACKERS] superuser() shortcuts

2014-12-02 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Tue, Dec 2, 2014 at 11:29 AM, Stephen Frost sfr...@snowman.net wrote: It includes the actual error message, unlike what we have today which is guidence as to what's required to get past the permission denied error. In other words, I disagree

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Andres Freund
On 2014-12-02 11:23:31 -0800, Jeff Janes wrote: I think it would be more promising to work on downgrading lock strengths so that fewer things conflict, and it would be not much more work than what you propose. I think you *massively* underestimate the effort required to to lower lock levels.

Re: [HACKERS] Review of GetUserId() Usage

2014-12-02 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: On Sat, Nov 29, 2014 at 3:00 PM, Stephen Frost sfr...@snowman.net wrote: * Stephen Frost (sfr...@snowman.net) wrote: Updated patch attached which also changes the error messages (which hadn't been updated in the prior versions and really

Re: [HACKERS] Removing INNER JOINs

2014-12-02 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Sun, Nov 30, 2014 at 12:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bottom line, given all the restrictions on whether the optimization can happen, I have very little enthusiasm for the whole idea. I do not think the benefit will be big enough

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Jeff Janes
On Tue, Dec 2, 2014 at 11:41 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-12-02 11:23:31 -0800, Jeff Janes wrote: I think it would be more promising to work on downgrading lock strengths so that fewer things conflict, and it would be not much more work than what you propose.

Re: [HACKERS] Review of GetUserId() Usage

2014-12-02 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: 3. It messes around with pg_signal_backend(). There are currently no cases in which pg_signal_backend() throws an error, which is good, because it lets you write queries against pg_stat_activity() that don't fail halfway through, even if you are

Re: [HACKERS] Review of GetUserId() Usage

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 2:50 PM, Stephen Frost sfr...@snowman.net wrote: 1. It makes more of the crappy error message change that Andres and I already objected to on the other thread. Whether you disagree with those objections or not, don't make an end-run around them by putting more of the

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Andres Freund
On 2014-12-02 12:22:42 -0800, Jeff Janes wrote: Or maybe I overestimate how hard it would be to make vacuum restartable. That's a massive project. Which is why I'm explicitly *not* suggesting that. What I instead suggest is a separate threshhold after which vacuum isn't going to abort

Re: [HACKERS] superuser() shortcuts

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 2:35 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Tue, Dec 2, 2014 at 11:29 AM, Stephen Frost sfr...@snowman.net wrote: It includes the actual error message, unlike what we have today which is guidence as to what's

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-02 Thread Andres Freund
On 2014-12-02 17:25:14 +0300, Alex Shulgin wrote: I'd be in favor of a solution that works the same way as before the patch, without the need for extra trigger files, etc., but that doesn't seem to be nearly possible. Whatever tricks we might employ will likely be defeated by the fact that

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Robert Haas
On Tue, Nov 25, 2014 at 1:38 PM, Peter Geoghegan p...@heroku.com wrote: On Tue, Nov 25, 2014 at 4:01 AM, Robert Haas robertmh...@gmail.com wrote: - This appears to needlessly reindent the comments for PG_CACHE_LINE_SIZE. Actually, the word only is removed (because PG_CACHE_LINE_SIZE has a new

Re: [HACKERS] Selectivity estimation for inet operators

2014-12-02 Thread Emre Hasegeli
I spent a fair chunk of the weekend hacking on this patch to make it more understandable and fix up a lot of what seemed to me pretty clear arithmetic errors in the upper layers of the patch. However, I couldn't quite convince myself to commit it, because the business around estimation for

Re: [HACKERS] Doing better at HINTing an appropriate column within errorMissingColumn()

2014-12-02 Thread Robert Haas
On Tue, Nov 25, 2014 at 7:13 PM, Peter Geoghegan p...@heroku.com wrote: Alright, so let me summarize what I think are the next steps in working towards getting this patch committed. I should produce a new revision which: * Uses default costings. * Applies a generic final quality check that

Re: [HACKERS] superuser() shortcuts

2014-12-02 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Tue, Dec 2, 2014 at 2:35 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Tue, Dec 2, 2014 at 11:29 AM, Stephen Frost sfr...@snowman.net wrote: It includes the actual error message, unlike what we

Re: [HACKERS] Selectivity estimation for inet operators

2014-12-02 Thread Emre Hasegeli
Actually, there's a second large problem with this patch: blindly iterating through all combinations of MCV and histogram entries makes the runtime O(N^2) in the statistics target. I made up some test data (by scanning my mail logs) and observed the following planning times, as reported by

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Peter Geoghegan
On Tue, Dec 2, 2014 at 1:00 PM, Robert Haas robertmh...@gmail.com wrote: I'd prefer not to have a #define in pg_config_manual.h. Only stuff that we expect a reasonably decent number of users to need to change should be in that file, and this is too marginal for that. If anybody other than

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Bruce Momjian
On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state type is just the same as the input type. It would only be aggregates like avg(),

Re: [HACKERS] 9.2 recovery/startup problems

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 11:54 AM, Jeff Janes jeff.ja...@gmail.com wrote: During abort processing after getting a SIGTERM, the back end truncates 59288 to zero size, and unlinks all the other files (including 59288_init). The actual removal of 59288 is left until the checkpoint. So if you

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 4:21 PM, Peter Geoghegan p...@heroku.com wrote: I'm not sure about that. I'd prefer to have tuplesort (and one or two other sites) set the abbreviation is possible in principle flag. Otherwise, sortsupport needs to assume that the leading attribute is going to be the

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-02 Thread Alex Shulgin
Andres Freund and...@2ndquadrant.com writes: On 2014-12-02 17:25:14 +0300, Alex Shulgin wrote: I'd be in favor of a solution that works the same way as before the patch, without the need for extra trigger files, etc., but that doesn't seem to be nearly possible. Whatever tricks we might

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Peter Geoghegan
On Tue, Dec 2, 2014 at 2:07 PM, Robert Haas robertmh...@gmail.com wrote: Well, maybe you should make the updates we've agreed on and I can take another look at it. Agreed. But I didn't think that I was proposing to change anything about the level at which the decision about whether to

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 5:16 PM, Peter Geoghegan p...@heroku.com wrote: On Tue, Dec 2, 2014 at 2:07 PM, Robert Haas robertmh...@gmail.com wrote: Well, maybe you should make the updates we've agreed on and I can take another look at it. Agreed. But I didn't think that I was proposing to

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Peter Geoghegan
On Tue, Dec 2, 2014 at 2:21 PM, Robert Haas robertmh...@gmail.com wrote: Right, and what I'm saying is that maybe the applicability flag shouldn't be stored in the SortSupport object, but passed down as an argument. But then how does that information get to any given sortsupport routine?

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: On Tue, Dec 2, 2014 at 2:21 PM, Robert Haas robertmh...@gmail.com wrote: Right, and what I'm saying is that maybe the applicability flag shouldn't be stored in the SortSupport object, but passed down as an argument. But then how does that information

Re: [HACKERS] Nitpicky doc corrections for BRIN functions of pageinspect

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 12:23 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Peter Geoghegan wrote: On Wed, Nov 19, 2014 at 8:02 PM, Michael Paquier michael.paqu...@gmail.com wrote: Just a small thing I noticed while looking at pageinspect.sgml, the set of SQL examples related to BRIN

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 2:17 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 26, 2014 at 11:00 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Nov 26, 2014 at 8:27 PM, Syed, Rahila rahila.s...@nttdata.com wrote: Don't we need to initialize doPageCompression similar to

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Jim Nasby
On 12/2/14, 2:22 PM, Jeff Janes wrote: Or maybe I overestimate how hard it would be to make vacuum restartable. You would have to save a massive amount of state (upto maintenance_work_mem tid list, the block you left off on both the table and all of the indexes in that table), and you would

Re: [HACKERS] [v9.5] Custom Plan API

2014-12-02 Thread Kouhei Kaigai
-Original Message- From: Simon Riggs [mailto:si...@2ndquadrant.com] Sent: Thursday, November 27, 2014 8:48 PM To: Kaigai Kouhei(海外 浩平) Cc: Robert Haas; Thom Brown; Kohei KaiGai; Tom Lane; Alvaro Herrera; Shigeru Hanada; Stephen Frost; Andres Freund; PgHacker; Jim Mlodgenski; Peter

Re: [HACKERS] Many processes blocked at ProcArrayLock

2014-12-02 Thread Michael Paquier
On Tue, Dec 2, 2014 at 5:07 PM, Xiaoyulei xiaoyu...@huawei.com wrote: Test configuration: Hardware: 4P intel server, 60 core 120 hard thread. Memory:512G SSD:2.4T PG: max_connections = 160 # (change requires restart) shared_buffers = 32GB

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Peter Geoghegan
On Tue, Dec 2, 2014 at 2:16 PM, Peter Geoghegan p...@heroku.com wrote: On Tue, Dec 2, 2014 at 2:07 PM, Robert Haas robertmh...@gmail.com wrote: Well, maybe you should make the updates we've agreed on and I can take another look at it. Agreed. Attached, revised patchset makes these updates. I

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-12-02 Thread Peter Geoghegan
On Tue, Dec 2, 2014 at 5:28 PM, Peter Geoghegan p...@heroku.com wrote: Attached, revised patchset makes these updates. Whoops. Missed some obsolete comments. Here is a third commit that makes a further small modification to one comment. -- Peter Geoghegan From

[HACKERS] About xmllint checking for the validity of postgres.xml in 9.5

2014-12-02 Thread Michael Paquier
Hi all, Since commit 5d93ce2d, the output of xmllint is checked by passing --valid to it. Isn't that a regression with what we were doing for pre-9.4 versions? For example, with 9.4 and older versions it is possible to compile man pages even if the xml spec is not entirely valid when using

Re: [HACKERS] About xmllint checking for the validity of postgres.xml in 9.5

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 12:09 PM, Michael Paquier michael.paqu...@gmail.com wrote: Since commit 5d93ce2d, the output of xmllint is checked by passing --valid to it. Isn't that a regression with what we were doing for pre-9.4 versions? For example, with 9.4 and older versions it is possible to

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-02 Thread Robert Haas
On Tue, Dec 2, 2014 at 7:16 PM, Michael Paquier michael.paqu...@gmail.com wrote: In the latest versions of the patch, control of compression is done within full_page_writes by assigning a new value 'compress'. Something that I am scared of is that if we enforce compression when

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 12:35 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Dec 2, 2014 at 7:16 PM, Michael Paquier michael.paqu...@gmail.com wrote: In the latest versions of the patch, control of compression is done within full_page_writes by assigning a new value 'compress'. Something

Re: [HACKERS] On partitioning

2014-12-02 Thread Amit Langote
Hi Robert, From: Robert Haas [mailto:robertmh...@gmail.com] * Catalog schema: CREATE TABLE pg_catalog.pg_partitioned_rel ( partrelidoidNOT NULL, partkindoidNOT NULL, partissub bool NOT NULL, partkey

Re: [HACKERS] using Core Foundation locale functions

2014-12-02 Thread Noah Misch
On Fri, Nov 28, 2014 at 11:43:28AM -0500, Peter Eisentraut wrote: In light of the recent discussions about using ICU on OS X, I looked into the Core Foundation locale functions (Core Foundation = traditional Mac API in OS X, as opposed to the Unix/POSIX APIs). Attached is a proof of concept

Re: [HACKERS] using Core Foundation locale functions

2014-12-02 Thread Craig Ringer
On 12/02/2014 12:52 AM, David E. Wheeler wrote: Gotta say, I’m thrilled to see movement on this front, and especially pleased to see how consensus seems to be building around an abstracted interface to keep options open. This platform-specific example really highlights the need for it (I

Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2014-12-02 Thread Kouhei Kaigai
On Tue, Nov 25, 2014 at 3:44 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Today, I had a talk with Hanada-san to clarify which can be a common portion of them and how to implement it. Then, we concluded both of features can be shared most of the infrastructure. Let me put an introduction

Re: [HACKERS] using Core Foundation locale functions

2014-12-02 Thread Peter Geoghegan
On Tue, Dec 2, 2014 at 10:07 PM, Craig Ringer cr...@2ndquadrant.com wrote: On 12/02/2014 12:52 AM, David E. Wheeler wrote: Gotta say, I’m thrilled to see movement on this front, and especially pleased to see how consensus seems to be building around an abstracted interface to keep options

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Kouhei Kaigai
On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state type is just the same as the input type. It would only be aggregates like

Re: [HACKERS] printing table in asciidoc with psql

2014-12-02 Thread Michael Paquier
On Mon, Nov 17, 2014 at 7:48 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2014-11-07 22:37 GMT+01:00 Alvaro Herrera alvhe...@2ndquadrant.com: I did \o /tmp/tst, then \dS create table eh | oh (); \dS and then filtered the output file to HTML. The CREATE TABLE tag ended up in the same

Re: [HACKERS] printing table in asciidoc with psql

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 3:52 PM, Michael Paquier michael.paqu...@gmail.com wrote: This patch has been marked as Waiting on Author for a couple of weeks, and the problems mentioned before have not been completely addressed, hence marking this patch as returned with feedback. It would be nice to

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-12-02 Thread Michael Paquier
On Tue, Nov 18, 2014 at 12:33 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Nov 17, 2014 at 10:31 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-17 10:21:04 -0500, Robert Haas wrote: Andres, where are we with this patch? 1. You're going to commit it, but haven't gotten

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 3:23 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state