Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 08:44 -0500, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: On 3 December 2013 02:02, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: ISTM that the real solution to this particular problem is to decouple the extensions that are currently in contrib from

Re: [HACKERS] ruleutils vs. empty targetlists

2013-12-04 Thread Dean Rasheed
On 3 December 2013 23:37, Tom Lane t...@sss.pgh.pa.us wrote: Thinking some more about bug #8648, it occurred to me that ruleutils.c isn't exactly prepared for the case either: regression=# create table nocols(); CREATE TABLE regression=# create view vv1 as select exists (select * from

Re: [HACKERS] pg_stat_statements: calls under-estimation propagation

2013-12-04 Thread Sameer Thakur
I've cleaned this up - revision attached - and marked it ready for committer. Thank you for this. I did the basic hygiene test. The patch applies correctly and compiles with no warnings. Did not find anything broken in basic functionality. In the documentation i have a minor suggestion of

Re: [HACKERS] logical changeset generation v6.7

2013-12-04 Thread Kyotaro HORIGUCHI
Hello, this is cont'd comments. 0008 and after to come later.. I had nothing to comment for patch 0008. = 0009: - In repl_scanner.l, you omitted double-doublequote handling for replication but it should be implemented. Zero-length identifier check might be needed depending on the

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Mon, 2013-12-02 at 15:44 -0500, Stephen Frost wrote: How are we going to handle new keywords being added in new major versions? A pg_dump of the extension template script is then going to be loaded into the new major version but will not actually be able to be run because it'll error

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 10:23 -0500, Robert Haas wrote: In more normal cases, however, the system can (and probably should) figure out what was intended by choosing the *shortest* path to get to the intended version. For example, if someone ships 1.0, 1.0--1.1, 1.1, and 1.1--1.2, the system

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Heikki Linnakangas
On 12/04/2013 01:08 AM, Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus j...@agliodbs.com wrote: Would certainly be nice. Realistically, getting good automated performace tests will require paying someone like Greg S., Mark or me for 6

Re: [HACKERS] logical changeset generation v6.7

2013-12-04 Thread Andres Freund
On 2013-12-04 17:31:50 +0900, Kyotaro HORIGUCHI wrote: = 0009: - In repl_scanner.l, you omitted double-doublequote handling for replication but it should be implemented. Zero-length identifier check might be needed depending on the upper-layer. I am not sure what you mean here.

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-04 Thread Dean Rasheed
On 2 December 2013 04:55, Pavel Stehule pavel.steh...@gmail.com wrote: Hello it looks well, thank you Regards Pavel I've been thinking about this some more, and there's another case that concerns me slightly. We're now making some of the DROP...IF EXISTS commands tolerate non-existent

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
On 2013-12-04 11:13:58 +0900, KONDO Mitsumasa wrote: 4) Start the slave and connect to it using psql and in another session I can see all archive recovery log Hmm... I had thought my mistake in reading your email, but it reproduce again. When I sat small recovery_time_delay(=3), it might

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-12-04 Thread Magnus Hagander
On Mon, Dec 2, 2013 at 10:42 PM, Peter Eisentraut pete...@gmx.net wrote: On 11/19/13, 11:30 PM, Peter Geoghegan wrote: +1 from me. That's +1 for *not* including this? Right. I agree with not including this. If you're looking for more of those, here's another +1 for not including

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Christian Kruse
Hi, On 04/12/13 11:13, KONDO Mitsumasa wrote: 1) Clusters - build master - build slave and attach to the master using SR and config recovery_time_delay to 1min. 2) Stop de Slave 3) Run some transactions on the master using pgbench to generate a lot of archives 4) Start the slave

Re: [HACKERS] Problem with displaying wide tables in psql

2013-12-04 Thread Alexander Korotkov
On Wed, Dec 4, 2013 at 11:20 AM, Sergey Muraviov sergey.k.murav...@gmail.com wrote: Thank you for this trick. It would be nice if this trick was documented. However, with the pager I can't see wide value on one screen, select and copy it entirely. And I have to press many keys to find the

Re: [HACKERS] [bug fix] pg_ctl stop times out when it should respond quickly

2013-12-04 Thread MauMau
From: Tom Lane t...@sss.pgh.pa.us I think the reason why it was coded like that was that we hadn't written postmaster_is_alive() yet, or maybe we had but didn't want to trust it. However, with the coding you have here, we're fully exposed to any failure modes postmaster_is_alive() may have; so

Re: [HACKERS] [bug fix or improvement?] Correctly place DLLs for ECPG apps in bin folder

2013-12-04 Thread MauMau
From: MauMau maumau...@gmail.com In addition, I'll remove libpq.dll from lib folder unless somebody objects. Currently, libpq.dll is placed in both bin and lib. I guess libpq.dll was left in lib because it was considered necessary for ECPG DLLs. The attached patch also removes libpq.dll from

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
I think all of this data cannot fit in shared_buffers, you might want to increase shared_buffers to larger size (not 30GB but close to your data size) to see how it behaves. When I use shared_buffers larger than my data size such as 10 GB, results scale nearly as expected at least for this

[HACKERS] [bug fix] psql's \conninfo reports incorrect destination on Windows

2013-12-04 Thread MauMau
Hello, I've found a bug that psql's \conninfo displays incorrect information on Windows. Please find attached the patch and commit this. [Problem] When I run psql postgres on Windows and execute \conninfo, it outputs the text below. It reports that psql connected to the server via UNIX

Re: [HACKERS] [PERFORM] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
Maybe you could help test this patch: http://www.postgresql.org/message-id/20131115194725.gg5...@awork2.anarazel.de Which repository should I apply these patches. I tried main repository, 9.3 stable and source code of 9.3.1, and in my trials at least of one the patches is failed. What patch

Re: [HACKERS] ruleutils vs. empty targetlists

2013-12-04 Thread Tom Lane
Dean Rasheed dean.a.rash...@gmail.com writes: On 3 December 2013 23:37, Tom Lane t...@sss.pgh.pa.us wrote: Thinking some more about bug #8648, it occurred to me that ruleutils.c isn't exactly prepared for the case either: ... So I'm leaning towards just doing + if (colno ==

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Andres Freund and...@2ndquadrant.com On 2013-12-04 11:13:58 +0900, KONDO Mitsumasa wrote: 4) Start the slave and connect to it using psql and in another session I can see all archive recovery log Hmm... I had thought my mistake in reading your email, but it reproduce again.

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
On 2013-12-04 22:47:47 +0900, Mitsumasa KONDO wrote: 2013/12/4 Andres Freund and...@2ndquadrant.com When it happened, psql cannot connect standby server at all. I think this behavior is not good. It should only delay recovery position and can seen old delay table data. That doesn't sound like

Re: [HACKERS] ruleutils vs. empty targetlists

2013-12-04 Thread Alvaro Herrera
Tom Lane escribió: What I'm thinking about this today is that really the *right* solution is to allow syntactically-empty SELECT lists; once we've bought into the notion of zero-column tables, the notion that you can't have an empty select list is just fundamentally at odds with that. And

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Christian Kruse christ...@2ndquadrant.com You created a master node and a hot standby with 300 delay. Then you stopped the standby, did the pgbench and startet the hot standby again. It did not get in line with the master. Is this correct? No. First, I start master, and execute

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
Hi, On 2013-12-03 19:33:16 +, Simon Riggs wrote: - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT and XLOG_XACT_COMMIT_COMPACT checks Why just those? Why not aborts and restore points also? What would the advantage of waiting on anything but commits be? If it's not a commit, the

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Andres Freund and...@2ndquadrant.com On 2013-12-04 22:47:47 +0900, Mitsumasa KONDO wrote: 2013/12/4 Andres Freund and...@2ndquadrant.com When it happened, psql cannot connect standby server at all. I think this behavior is not good. It should only delay recovery position and

Re: [HACKERS] Changes in Trigger Firing

2013-12-04 Thread Alvaro Herrera
Sameer Kumar wrote: CreateTrigStmt is passed to CreateTrigger function as an arguement. I am struggling to understand how the values for various members of trigger are set and where [which file] calls CreateTrigStmt. Can someone provide some help on this? I think you need better tools

[HACKERS] [bug fix] pg_ctl fails with config-only directory

2013-12-04 Thread MauMau
Hello, I've found a bug and would like to fix it, but I cannot figure out how to do that well. Could you give me any advice? I encountered this on PG 9.2, but it will probably exist in later versions. [Problem] On Windows, a user with Administrator privileges can start the database

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Stephen Frost
* Jeff Davis (pg...@j-davis.com) wrote: On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: When it comes to dump/reload, I'd much rather see a mechanism which uses our deep understanding of the extension's objects (as database objects) to

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-12-04 Thread Ian Lawrence Barwick
2013/11/8 Tom Lane t...@sss.pgh.pa.us: Albe Laurenz laurenz.a...@wien.gv.at writes: What I would like to do is add a custom resjunk column (e.g. a bytea) in AddForeignUpdateTargets that carries a row identifier from the scan state to the modify state. Would that be possible? Can I have

Re: [HACKERS] Problem with displaying wide tables in psql

2013-12-04 Thread Pavel Stehule
Hello postgres=# \pset format wrapped Output format (format) is wrapped. postgres=# select 'afadsafasd fasdf asdfasd fsad fas df sadf sad f sadf sadf sa df sadfsadfasd fsad fsa df sadf asd fa sfd sadfsadf asdf sad f sadf sad fadsf'; ?column?

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Stephen Frost
* Jeff Davis (pg...@j-davis.com) wrote: On Mon, 2013-12-02 at 15:44 -0500, Stephen Frost wrote: How are we going to handle new keywords being added in new major versions? A pg_dump of the extension template script is then going to be loaded into the new major version but will not

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-04 Thread Pavel Stehule
2013/12/4 Dean Rasheed dean.a.rash...@gmail.com On 2 December 2013 04:55, Pavel Stehule pavel.steh...@gmail.com wrote: Hello it looks well, thank you Regards Pavel I've been thinking about this some more, and there's another case that concerns me slightly. We're now making

Re: [HACKERS] [patch] Adding EXTRA_REGRESS_OPTS to all pg_regress invocations

2013-12-04 Thread Bruce Momjian
On Mon, May 6, 2013 at 11:51:47PM -0700, Christoph Berg wrote: make check supports EXTRA_REGRESS_OPTS to pass extra options to pg_regress, but all the other places where pg_regress is used do not allow this. The attached patch adds EXTRA_REGRESS_OPTS to Makefile.global.in (for contrib

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: So, I proposed this patch previously and I still think it's a good idea, but it got voted down on the grounds that it didn't deal with clock drift.  I view that as insufficient reason to reject the feature, but others disagreed.  Unless some of those

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2013-12-04 Thread Kohei KaiGai
Hanada-san, Thanks for your reviewing, 2013/12/4 Shigeru Hanada shigeru.han...@gmail.com: I first reviewed postgres_fdw portion of the patches to learn the outline of Custom Plan. Wiki page is also a good textbook of the feature. I have some random comments about the basic design of Custom

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2013-12-04 Thread Kohei KaiGai
Thanks for fixing many my carelessness. I didn't know seek was an irregular verb... Best regards, 2013/12/4 Shigeru Hanada shigeru.han...@gmail.com: 2013/11/29 Kohei KaiGai kai...@kaigai.gr.jp: I merged all the propositions from Jim. Thanks, it made the documentation quality better. Also, I

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Christian Kruse
Hi, On 04/12/13 07:22, Kevin Grittner wrote: There are many things that a system admin can get wrong.  Failing to supply this feature because the sysadmin might not be running ntpd (or equivalent) correctly seems to me to be like not having the software do fsync because the sysadmin might not

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved problem. Is it sound to run performance tests on kvm? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Tue, 03 Dec 2013 10:44:15 -0800 Josh Berkus j...@agliodbs.com wrote: It seems clear that Kernel.org, since 2.6, has been in the business of pushing major, hackish, changes to the IO stack without testing them or even thinking too hard about what the side-effects might be. This is perhaps

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 04:30 PM, Peter Eisentraut wrote: On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved problem. Is it sound to run performance tests on kvm? as sounds as on any other platform imho, the

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 1:42 AM, Álvaro Hernández Tortosa wrote: IMHO, a data structure like the above would be completely self-contained and allow any autoconfiguring tool or GUI tool to be easily created, if the syntax is programmable. It would certainly make the config file more verbose, but at the

Re: [HACKERS] Minor patch for the uuid-ossp extension

2013-12-04 Thread Peter Eisentraut
On 11/23/13, 7:12 AM, Mario Weilguni wrote: Well, in that case and since this is a rarely used extension (I guess so), maybe it would be the best to simply rename that extension to uuidossp (or whatever) and don't make any special treatment for it? Why? This is a solved problem, and renaming

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-12-04 Thread Albe Laurenz
Ian Lawrence Barwick wrote: 2013/11/8 Tom Lane t...@sss.pgh.pa.us: [ thinks for awhile... ] Hm. In principle you can put any expression you want into the tlist during AddForeignUpdateTargets. However, if it's not a Var then the planner won't understand that it's something that needs to be

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 16:51, Peter Eisentraut wrote: On 12/4/13, 1:42 AM, Álvaro Hernández Tortosa wrote: IMHO, a data structure like the above would be completely self-contained and allow any autoconfiguring tool or GUI tool to be easily created, if the syntax is programmable. It would certainly

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Tue, Dec 3, 2013 at 5:57 PM, Tom Dunstan pg...@tomd.cc wrote: On 4 December 2013 01:24, Robert Haas robertmh...@gmail.com wrote: Yeah, more or less, but the key is ensuring that it wouldn't let you create the constraint in the first place if the partial index specified *didn't* match the

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Claudio Freire
On Wed, Dec 4, 2013 at 9:19 AM, Metin Doslu me...@citusdata.com wrote: Here are the results of vmstat 1 while running 8 parallel TPC-H Simple (#6) queries: Although there is no need for I/O, wa fluctuates between 0 and 1. procs ---memory-- ---swap-- -io --system--

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 14:27:10 -0200, Claudio Freire wrote: On Wed, Dec 4, 2013 at 9:19 AM, Metin Doslu me...@citusdata.com wrote: Here are the results of vmstat 1 while running 8 parallel TPC-H Simple (#6) queries: Although there is no need for I/O, wa fluctuates between 0 and 1. procs

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
Notice the huge %sy What kind of VM are you using? HVM or paravirtual? This instance is paravirtual.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
I'd strongly suggest doing a perf record -g -a wait a bit, ctrl-c; perf report run to check what's eating up the time. Here is one example: + 38.87% swapper [kernel.kallsyms] [k] hypercall_page + 9.32% postgres [kernel.kallsyms] [k] hypercall_page + 6.80% postgres

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 11:25 AM, Robert Haas wrote: On Tue, Dec 3, 2013 at 5:57 PM, Tom Dunstan pg...@tomd.cc wrote: On 4 December 2013 01:24, Robert Haas robertmh...@gmail.com wrote: Yeah, more or less, but the key is ensuring that it wouldn't let you create the constraint in the first place if the

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 18:43:35 +0200, Metin Doslu wrote: I'd strongly suggest doing a perf record -g -a wait a bit, ctrl-c; perf report run to check what's eating up the time. Here is one example: + 38.87% swapper [kernel.kallsyms] [k] hypercall_page + 9.32% postgres [kernel.kallsyms]

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 11:44 AM, Andrew Dunstan and...@dunslane.net wrote: Oh. I misinterpreted what this feature was about, then. I thought it was about restricting the reference to a subset of the *referenced* table, but it seems to be about restricting the constraint to a subset of the

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Peter Eisentraut
src/backend/access/transam/xlog.c:5889: trailing whitespace. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 12:00 PM, Robert Haas wrote: On Wed, Dec 4, 2013 at 11:44 AM, Andrew Dunstan and...@dunslane.net wrote: Oh. I misinterpreted what this feature was about, then. I thought it was about restricting the reference to a subset of the *referenced* table, but it seems to be about

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Magnus Hagander
On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Dec 3, 2013 at 7:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Maybe we should just bite the bullet and change the WAL format for heap_freeze (inventing an all-new record

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Wed, 2013-12-04 at 09:50 -0500, Stephen Frost wrote: I still don't see that Extension Templates are all bad: * They preserve the fact that two instances of the same extension (e.g. in different databases) were created from the same template. This is only true if we change the

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 04:33 PM, Jonathan Corbet wrote: On Tue, 03 Dec 2013 10:44:15 -0800 Josh Berkus j...@agliodbs.com wrote: It seems clear that Kernel.org, since 2.6, has been in the business of pushing major, hackish, changes to the IO stack without testing them or even thinking too hard about

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: When it comes to dump/reload, I'd much rather see a mechanism which uses our deep understanding of the extension's objects (as database objects) to implement the dump/reload than a text blob which is

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 12:18 PM, Andrew Dunstan and...@dunslane.net wrote: Interestingly, the variant for which you can't think of a use case is the one I've missed most. Typical examples in my experience are things like project.project_manager_id references person (id) where

Re: [HACKERS] Changes in Trigger Firing

2013-12-04 Thread Sameer Kumar
CreateTrigStmt is passed to CreateTrigger function as an arguement. I am struggling to understand how the values for various members of trigger are set and where [which file] calls CreateTrigStmt. Can someone provide some help on this? I think you need better tools to guide you

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Claudio Freire
On Wed, Dec 4, 2013 at 1:54 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-04 18:43:35 +0200, Metin Doslu wrote: I'd strongly suggest doing a perf record -g -a wait a bit, ctrl-c; perf report run to check what's eating up the time. Here is one example: + 38.87% swapper

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
You could try HVM. I've noticed it fare better under heavy CPU load, and it's not fully-HVM (it still uses paravirtualized network and I/O). I already tried with HVM (cc2.8xlarge instance on Amazon EC2) and observed same problem.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 16:00:40 -0200, Claudio Freire wrote: On Wed, Dec 4, 2013 at 1:54 PM, Andres Freund and...@2ndquadrant.com wrote: All that time is spent in your virtualization solution. One thing to try is to look on the host system, sometimes profiles there can be more meaningful. You

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
Didn't follow the thread from the start. So, this is EC2? Have you checked, with a recent enough version of top or whatever, how much time is reported as stolen? Yes, this EC2. stolen is randomly reported as 1, mostly as 0.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
Here are some extra information: - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is disappeared for 8 core machines and come back with 16 core machines on Amazon EC2. Would it be related with PostgreSQL locking mechanism? - I tried this test with 4 core machines including my

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 20:19:55 +0200, Metin Doslu wrote: - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is disappeared for 8 core machines and come back with 16 core machines on Amazon EC2. Would it be related with PostgreSQL locking mechanism? You could try my lwlock-scalability

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
You could try my lwlock-scalability improvement patches - for some workloads here, the improvements have been rather noticeable. Which version are you testing? I'm testing with PostgreSQL 9.3.1.

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Joshua D. Drake
On 12/04/2013 07:32 AM, Stefan Kaltenbrunner wrote: On 12/04/2013 04:30 PM, Peter Eisentraut wrote: On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved problem. Is it sound to run performance tests

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 07:30 PM, Joshua D. Drake wrote: On 12/04/2013 07:32 AM, Stefan Kaltenbrunner wrote: On 12/04/2013 04:30 PM, Peter Eisentraut wrote: On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Merlin Moncure
On Mon, Dec 2, 2013 at 10:26 PM, David Fetter da...@fetter.org wrote: On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: On 11/28/2013 03:24 AM, David Fetter wrote: WITH, or SRF, or whatever, the point is that we need to be able to specify what we're sending--probably single

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 11:22 AM, Álvaro Hernández Tortosa wrote: Would it be well-received a new file format that keeps it simple for both hand editing and generation of the configuration, and at the same time offers the features I have mentioned? I don't see how that would work exactly: You want to add

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 19:49, Peter Eisentraut wrote: On 12/4/13, 11:22 AM, Álvaro Hernández Tortosa wrote: Would it be well-received a new file format that keeps it simple for both hand editing and generation of the configuration, and at the same time offers the features I have mentioned? I don't see

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Josh Berkus
On 12/04/2013 07:33 AM, Jonathan Corbet wrote: Wow, Josh, I'm surprised to hear this from you. Well, I figured it was too angry to propose for an LWN article. ;-) The active/inactive list mechanism works great for the vast majority of users. The second-use algorithm prevents a lot of

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Andrew Gierth
Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom Well, okay, but you've not said anything that wouldn't be Tom handled just as well by some logic that adds a fixed Tom integer-constant-zero flag column to the rows going into the Tom tuplesort. Adding such a column unconditionally even for

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread David Fetter
On Wed, Dec 04, 2013 at 12:43:44PM -0600, Merlin Moncure wrote: On Mon, Dec 2, 2013 at 10:26 PM, David Fetter da...@fetter.org wrote: On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: On 11/28/2013 03:24 AM, David Fetter wrote: WITH, or SRF, or whatever, the point is that we

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Well I guess we could say something like: FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE (b-condition) But it's somewhat ugly. OK, those make sense. I wonder whether this should be done via a USING clause on the constraint

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Joshua D. Drake
On 12/04/2013 07:33 AM, Jonathan Corbet wrote: Wow, Josh, I'm surprised to hear this from you. The active/inactive list mechanism works great for the vast majority of users. The second-use algorithm prevents a lot of pathological behavior, like wiping out your entire cache by copying a big

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: I assume what would happen is the slave would PANIC upon seeing a WAL record code it didn't recognize. I wonder if we should for the future have the START_REPLICATION command (or

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: When it comes to dump/reload, I'd much rather see a mechanism which uses our deep understanding of the extension's objects (as database objects) to implement the

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
David Fetter da...@fetter.org writes: The idea here is that such a happy situation will not obtain until much later, if ever, and meanwhile, we need a way to get things accomplished even if it's inelegant, inefficient, etc. The alternative is that those things simply will not get accomplished

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 02:40 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Well I guess we could say something like: FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE (b-condition) But it's somewhat ugly. OK, those make sense. I wonder whether this should be done via

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Robert Haas
On Tue, Dec 3, 2013 at 11:44 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: We should also consider the possibility of a user trying to deliberately install and older release. For example, if the user has 1.0, 1.0--1.1, 1.1, 1.1--1.2, and 1.2--1.0 (a downgrade script) with

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes: Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom Well, okay, but you've not said anything that wouldn't be Tom handled just as well by some logic that adds a fixed Tom integer-constant-zero flag column to the rows going into the Tom

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Merlin Moncure
On Wed, Dec 4, 2013 at 1:39 PM, David Fetter da...@fetter.org wrote: On Wed, Dec 04, 2013 at 12:43:44PM -0600, Merlin Moncure wrote: On Mon, Dec 2, 2013 at 10:26 PM, David Fetter da...@fetter.org wrote: On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: On 11/28/2013 03:24 AM,

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: The downside of SQL-MED, particularly the way postgres implemented the driver API, is that each driver is responsible for for all optimization efforts and I think this is bad. There was never any intention that that would be the final state of things.

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 3:39 AM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-12-03 at 10:23 -0500, Robert Haas wrote: In more normal cases, however, the system can (and probably should) figure out what was intended by choosing the *shortest* path to get to the intended version. For

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Wed, 04 Dec 2013 11:07:04 -0800 Josh Berkus j...@agliodbs.com wrote: On 12/04/2013 07:33 AM, Jonathan Corbet wrote: Wow, Josh, I'm surprised to hear this from you. Well, I figured it was too angry to propose for an LWN article. ;-) So you're going to make us write it for you :) The

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Magnus Hagander
On Wed, Dec 4, 2013 at 9:31 PM, Jonathan Corbet cor...@lwn.net wrote: I also wasn't exaggerating the reception I got when I tried to talk about IO and PostgreSQL at LinuxCon and other events. The majority of Linux hackers I've talked to simply don't want to be bothered with PostgreSQL's

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Magnus Hagander
On Wed, Dec 4, 2013 at 8:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: I assume what would happen is the slave would PANIC upon seeing a WAL record code it didn't recognize. I

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: I think that's an excellent idea. If one of our developers could find the time to attend that, I think that could be very productive. While I'm not on the funds team, I'd definitely vote for funding such participation out of community funds if

[HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
Having nothing better to do over the holiday weekend, I decided to pursue a number of ideas for improving performance that I thought about a long time ago. These include: * Pre-fetching list node pointers. This looks to be moderately promising, but I'm certainly not going to be the one to land

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Andrew Gierth
Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom Well, sure, but I was only suggesting adding it when the Tom aggregate asks for it, probably via a new flag column in Tom pg_aggregate. Sure, I was only pointing out the necessity. Tom The question you're evading is what additional

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Josh Berkus
Jonathan, For those interested in the details... (1) It's not quite 50/50, that's one bound for how the balance is allowed to go. (2) Anybody trying to add tunables to the kernel tends to run into resistance. Exposing thousands of knobs tends to lead to a situation where you *have* to be an

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Merlin Moncure
On Wed, Dec 4, 2013 at 2:31 PM, Jonathan Corbet cor...@lwn.net wrote: For those interested in the details... (1) It's not quite 50/50, that's one bound for how the balance is allowed to go. (2) Anybody trying to add tunables to the kernel tends to run into resistance. Exposing thousands of

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Wed, 04 Dec 2013 13:01:37 -0800 Josh Berkus j...@agliodbs.com wrote: Perhaps even better: the next filesystem, storage, and memory management summit is March 24-25. Link? I can't find anything Googling by that name. I'm pretty sure we can get at least one person there. It looks

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: I guess I could write a proper patch to have code setting up a scankey also set a flag that indicated that it was acceptable to assume that the special built-in comparator would do fine. ... I'd be happy with a scheme with only one built-in comparator,

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 4:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: Peter Geoghegan p...@heroku.com writes: I guess I could write a proper patch to have code setting up a scankey also set a flag that indicated that it was acceptable to assume that the special built-in comparator would do fine.

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes: Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom But anyway, what I'm thinking right now is that these questions Tom would all go away if the aggregate transfunction were receiving Tom the rows and sticking them into the tuplestore. It could

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/2/13, 9:14 AM, Dimitri Fontaine wrote: What I want to build is an “extension distribution” software that knows how to prepare anything from PGXN (and other places) so that it's fully ready for being used in the database. Then the main client would run as a CREATE EXTENSION

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/2/13, 2:33 PM, Greg Stark wrote: Just tossing an idea out there. What if you could install an extension by specifying not a local file name but a URL. Obviously there's a security issue but for example we could allow only https URLs with verified domain names that are in a list of

  1   2   >