Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 08:44 -0500, Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > > On 3 December 2013 02:02, Dimitri Fontaine wrote: > > > ISTM that the real solution to this particular problem is to decouple > > > the extensions that are currently in contrib from a specific po

Re: [HACKERS] ruleutils vs. empty targetlists

2013-12-04 Thread Dean Rasheed
On 3 December 2013 23:37, Tom Lane wrote: > Thinking some more about bug #8648, it occurred to me that ruleutils.c > isn't exactly prepared for the case either: > > regression=# create table nocols(); > CREATE TABLE > regression=# create view vv1 as select exists (select * from nocols); > CREATE V

Re: [HACKERS] pg_stat_statements: calls under-estimation propagation

2013-12-04 Thread Sameer Thakur
> >I've cleaned this up - revision attached - and marked it "ready for > committer". > Thank you for this. > I did the basic hygiene test. The patch applies correctly and compiles with no warnings. Did not find anything broken in basic functionality. In the documentation i have a minor suggesti

Re: [HACKERS] logical changeset generation v6.7

2013-12-04 Thread Kyotaro HORIGUCHI
Hello, this is cont'd comments. > 0008 and after to come later.. I had nothing to comment for patch 0008. = 0009: - In repl_scanner.l, you omitted double-doublequote handling for replication but it should be implemented. Zero-length identifier check might be needed depending on the

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Mon, 2013-12-02 at 15:44 -0500, Stephen Frost wrote: > How are we going to handle new keywords > being added in new major versions? A pg_dump of the extension template > script is then going to be loaded into the new major version but will > not actually be able to be run because it'll error ou

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 10:23 -0500, Robert Haas wrote: > In more normal cases, however, the system can (and probably should) > figure out what was intended by choosing the *shortest* path to get to > the intended version. For example, if someone ships 1.0, 1.0--1.1, > 1.1, and 1.1--1.2, the system

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Heikki Linnakangas
On 12/04/2013 01:08 AM, Tom Lane wrote: Magnus Hagander writes: On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: Would certainly be nice. Realistically, getting good automated performace tests will require paying someone like Greg S., Mark or me for 6 solid months to develop them, since w

Re: [HACKERS] logical changeset generation v6.7

2013-12-04 Thread Andres Freund
On 2013-12-04 17:31:50 +0900, Kyotaro HORIGUCHI wrote: > = 0009: > > - In repl_scanner.l, you omitted double-doublequote handling for >replication but it should be implemented. Zero-length >identifier check might be needed depending on the upper-layer. I am not sure what you mean he

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-04 Thread Dean Rasheed
On 2 December 2013 04:55, Pavel Stehule wrote: > Hello > > it looks well, thank you > > Regards > > Pavel > I've been thinking about this some more, and there's another case that concerns me slightly. We're now making some of the DROP...IF EXISTS commands tolerate non-existent types as well as no

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
On 2013-12-04 11:13:58 +0900, KONDO Mitsumasa wrote: > >4) Start the slave and connect to it using psql and in another session I can > >see > >all archive recovery log > Hmm... I had thought my mistake in reading your email, but it reproduce again. > When I sat small recovery_time_delay(=3), i

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-12-04 Thread Magnus Hagander
On Mon, Dec 2, 2013 at 10:42 PM, Peter Eisentraut wrote: > On 11/19/13, 11:30 PM, Peter Geoghegan wrote: > >>> +1 from me. > >> > > >> > > >> > That's +1 for *not* including this? > > Right. > > I agree with not including this. > > If you're looking for more of those, here's another +1 for not in

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Christian Kruse
Hi, On 04/12/13 11:13, KONDO Mitsumasa wrote: > >1) Clusters > >- build master > >- build slave and attach to the master using SR and config > >recovery_time_delay to > >1min. > > > >2) Stop de Slave > > > >3) Run some transactions on the master using pgbench to generate a lot of > >archives > >

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-04 Thread Alexander Korotkov
On Wed, Dec 4, 2013 at 11:20 AM, Sergey Muraviov < sergey.k.murav...@gmail.com> wrote: > Thank you for this trick. > It would be nice if this trick was documented. > > However, with the pager I can't see wide value on one screen, select and > copy it entirely. > And I have to press many keys to fi

Re: [HACKERS] [bug fix] "pg_ctl stop" times out when it should respond quickly

2013-12-04 Thread MauMau
From: "Tom Lane" I think the reason why it was coded like that was that we hadn't written postmaster_is_alive() yet, or maybe we had but didn't want to trust it. However, with the coding you have here, we're fully exposed to any failure modes postmaster_is_alive() may have; so there's not a lot

Re: [HACKERS] [bug fix or improvement?] Correctly place DLLs for ECPG apps in bin folder

2013-12-04 Thread MauMau
From: "MauMau" In addition, I'll remove libpq.dll from lib folder unless somebody objects. Currently, libpq.dll is placed in both bin and lib. I guess libpq.dll was left in lib because it was considered necessary for ECPG DLLs. The attached patch also removes libpq.dll from lib folder. I do

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> I think all of this data cannot fit in shared_buffers, you might want to increase shared_buffers > to larger size (not 30GB but close to your data size) to see how it behaves. When I use shared_buffers larger than my data size such as 10 GB, results scale nearly as expected at least for this

[HACKERS] [bug fix] psql's \conninfo reports incorrect destination on Windows

2013-12-04 Thread MauMau
Hello, I've found a bug that psql's \conninfo displays incorrect information on Windows. Please find attached the patch and commit this. [Problem] When I run "psql postgres" on Windows and execute \conninfo, it outputs the text below. It reports that psql connected to the server via UNIX do

Re: [HACKERS] [PERFORM] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> Maybe you could help test this patch: > http://www.postgresql.org/message-id/20131115194725.gg5...@awork2.anarazel.de Which repository should I apply these patches. I tried main repository, 9.3 stable and source code of 9.3.1, and in my trials at least of one the patches is failed. What patch co

Re: [HACKERS] ruleutils vs. empty targetlists

2013-12-04 Thread Tom Lane
Dean Rasheed writes: > On 3 December 2013 23:37, Tom Lane wrote: >> Thinking some more about bug #8648, it occurred to me that ruleutils.c >> isn't exactly prepared for the case either: >> ... So I'm leaning towards just doing >> >> + if (colno == 0) >> + app

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Andres Freund > On 2013-12-04 11:13:58 +0900, KONDO Mitsumasa wrote: > > >4) Start the slave and connect to it using psql and in another session > I can see > > >all archive recovery log > > Hmm... I had thought my mistake in reading your email, but it reproduce > again. > > When I sat

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
On 2013-12-04 22:47:47 +0900, Mitsumasa KONDO wrote: > 2013/12/4 Andres Freund > When it happened, psql cannot connect standby server at all. I think this > behavior is not good. > It should only delay recovery position and can seen old delay table data. That doesn't sound like a good plan - even

Re: [HACKERS] ruleutils vs. empty targetlists

2013-12-04 Thread Alvaro Herrera
Tom Lane escribió: > What I'm thinking about this today is that really the *right* solution > is to allow syntactically-empty SELECT lists; once we've bought into the > notion of zero-column tables, the notion that you can't have an empty > select list is just fundamentally at odds with that. And

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Christian Kruse > You created a master node and a hot standby with 300 delay. Then > you stopped the standby, did the pgbench and startet the hot standby > again. It did not get in line with the master. Is this correct? > No. First, I start master, and execute pgbench. Second, I sta

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Andres Freund
Hi, On 2013-12-03 19:33:16 +, Simon Riggs wrote: > > - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT and > > XLOG_XACT_COMMIT_COMPACT checks > > Why just those? Why not aborts and restore points also? What would the advantage of waiting on anything but commits be? If it's not a commit,

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Mitsumasa KONDO
2013/12/4 Andres Freund > On 2013-12-04 22:47:47 +0900, Mitsumasa KONDO wrote: > > 2013/12/4 Andres Freund > > When it happened, psql cannot connect standby server at all. I think this > > behavior is not good. > > It should only delay recovery position and can seen old delay table data. > > Tha

Re: [HACKERS] Changes in Trigger Firing

2013-12-04 Thread Alvaro Herrera
Sameer Kumar wrote: > > CreateTrigStmt is passed to CreateTrigger function as an arguement. I am > struggling to understand how the values for various members of trigger are > set and where [which file] calls CreateTrigStmt. > > > Can someone provide some help on this? I think you need better

[HACKERS] [bug fix] pg_ctl fails with config-only directory

2013-12-04 Thread MauMau
Hello, I've found a bug and would like to fix it, but I cannot figure out how to do that well. Could you give me any advice? I encountered this on PG 9.2, but it will probably exist in later versions. [Problem] On Windows, a user with Administrator privileges can start the database server.

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Stephen Frost
* Jeff Davis (pg...@j-davis.com) wrote: > On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: > > Stephen Frost writes: > > > When it comes to dump/reload, I'd much rather see a mechanism which uses > > > our deep understanding of the extension's objects (as database objects) > > > to implement the

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-12-04 Thread Ian Lawrence Barwick
2013/11/8 Tom Lane : > Albe Laurenz writes: >> What I would like to do is add a custom resjunk column >> (e.g. a bytea) in AddForeignUpdateTargets that carries a row identifier >> from the scan state to the modify state. >> Would that be possible? Can I have anything else than a Var >> in a resjun

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-04 Thread Pavel Stehule
Hello postgres=# \pset format wrapped Output format (format) is wrapped. postgres=# select 'afadsafasd fasdf asdfasd fsad fas df sadf sad f sadf sadf sa df sadfsadfasd fsad fsa df sadf asd fa sfd sadfsadf asdf sad f sadf sad fadsf'; ?column? --

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Stephen Frost
* Jeff Davis (pg...@j-davis.com) wrote: > On Mon, 2013-12-02 at 15:44 -0500, Stephen Frost wrote: > > How are we going to handle new keywords > > being added in new major versions? A pg_dump of the extension template > > script is then going to be loaded into the new major version but will > > not

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-04 Thread Pavel Stehule
2013/12/4 Dean Rasheed > On 2 December 2013 04:55, Pavel Stehule wrote: > > Hello > > > > it looks well, thank you > > > > Regards > > > > Pavel > > > > I've been thinking about this some more, and there's another case that > concerns me slightly. We're now making some of the DROP...IF EXISTS >

Re: [HACKERS] [patch] Adding EXTRA_REGRESS_OPTS to all pg_regress invocations

2013-12-04 Thread Bruce Momjian
On Mon, May 6, 2013 at 11:51:47PM -0700, Christoph Berg wrote: > "make check" supports EXTRA_REGRESS_OPTS to pass extra options to > pg_regress, but all the other places where pg_regress is used do not > allow this. The attached patch adds EXTRA_REGRESS_OPTS to > Makefile.global.in (for contrib mo

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Kevin Grittner
Robert Haas wrote: > So, I proposed this patch previously and I still think it's a > good idea, but it got voted down on the grounds that it didn't > deal with clock drift.  I view that as insufficient reason to > reject the feature, but others disagreed.  Unless some of those > people have chang

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2013-12-04 Thread Kohei KaiGai
Hanada-san, Thanks for your reviewing, 2013/12/4 Shigeru Hanada : > I first reviewed postgres_fdw portion of the patches to learn the > outline of Custom Plan. Wiki page is also a good textbook of the > feature. I have some random comments about the basic design of Custom > Plan: > > (1) IIUC a

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2013-12-04 Thread Kohei KaiGai
Thanks for fixing many my carelessness. I didn't know "seek" was an irregular verb... Best regards, 2013/12/4 Shigeru Hanada : > 2013/11/29 Kohei KaiGai : >> I merged all the propositions from Jim. Thanks, it made the documentation >> quality better. Also, I fixed up cosmetic stuff around whitesp

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Christian Kruse
Hi, On 04/12/13 07:22, Kevin Grittner wrote: > There are many things that a system admin can get wrong.  Failing > to supply this feature because the sysadmin might not be running > ntpd (or equivalent) correctly seems to me to be like not having > the software do fsync because the sysadmin might

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: > running a > few kvm instances that get bootstrapped automatically is something that > is a solved problem. Is it sound to run performance tests on kvm? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to you

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Tue, 03 Dec 2013 10:44:15 -0800 Josh Berkus wrote: > It seems clear that Kernel.org, since 2.6, has been in the business of > pushing major, hackish, changes to the IO stack without testing them or > even thinking too hard about what the side-effects might be. This is > perhaps unsurprising g

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 04:30 PM, Peter Eisentraut wrote: On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved problem. Is it sound to run performance tests on kvm? as sounds as on any other platform imho, the pe

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 1:42 AM, Álvaro Hernández Tortosa wrote: > IMHO, a data structure like the above would be completely > self-contained and allow any autoconfiguring tool or GUI tool to be > easily created, if the syntax is programmable. It would certainly make > the config file more verbose, but at

Re: [HACKERS] Minor patch for the uuid-ossp extension

2013-12-04 Thread Peter Eisentraut
On 11/23/13, 7:12 AM, Mario Weilguni wrote: > Well, in that case and since this is a rarely used extension (I guess > so), maybe it would be the best to simply rename that extension to > uuidossp (or whatever) and don't make any special treatment for it? Why? This is a solved problem, and renamin

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-12-04 Thread Albe Laurenz
Ian Lawrence Barwick wrote: > 2013/11/8 Tom Lane : >> [ thinks for awhile... ] Hm. In principle you can put any expression >> you want into the tlist during AddForeignUpdateTargets. However, if it's >> not a Var then the planner won't understand that it's something that needs >> to be supplied b

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 16:51, Peter Eisentraut wrote: On 12/4/13, 1:42 AM, Álvaro Hernández Tortosa wrote: IMHO, a data structure like the above would be completely self-contained and allow any autoconfiguring tool or GUI tool to be easily created, if the syntax is programmable. It would certainly m

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Tue, Dec 3, 2013 at 5:57 PM, Tom Dunstan wrote: > On 4 December 2013 01:24, Robert Haas wrote: >> Yeah, more or less, but the key is ensuring that it wouldn't let you >> create the constraint in the first place if the partial index >> specified *didn't* match the WHERE clause. For example, su

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Claudio Freire
On Wed, Dec 4, 2013 at 9:19 AM, Metin Doslu wrote: > > Here are the results of "vmstat 1" while running 8 parallel TPC-H Simple > (#6) queries: Although there is no need for I/O, "wa" fluctuates between 0 > and 1. > > procs ---memory-- ---swap-- -io --system-- > -cpu--

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 14:27:10 -0200, Claudio Freire wrote: > On Wed, Dec 4, 2013 at 9:19 AM, Metin Doslu wrote: > > > > Here are the results of "vmstat 1" while running 8 parallel TPC-H Simple > > (#6) queries: Although there is no need for I/O, "wa" fluctuates between 0 > > and 1. > > > > procs ---

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
>Notice the huge %sy >What kind of VM are you using? HVM or paravirtual? This instance is paravirtual.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> I'd strongly suggest doing a "perf record -g -a ; > perf report" run to check what's eating up the time. Here is one example: + 38.87% swapper [kernel.kallsyms] [k] hypercall_page + 9.32% postgres [kernel.kallsyms] [k] hypercall_page + 6.80% postgres [kernel.kallsyms] [k] xen_

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 11:25 AM, Robert Haas wrote: On Tue, Dec 3, 2013 at 5:57 PM, Tom Dunstan wrote: On 4 December 2013 01:24, Robert Haas wrote: Yeah, more or less, but the key is ensuring that it wouldn't let you create the constraint in the first place if the partial index specified *didn't* mat

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 18:43:35 +0200, Metin Doslu wrote: > > I'd strongly suggest doing a "perf record -g -a ; > > perf report" run to check what's eating up the time. > > Here is one example: > > + 38.87% swapper [kernel.kallsyms] [k] hypercall_page > + 9.32% postgres [kernel.kallsyms] [k] h

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 11:44 AM, Andrew Dunstan wrote: >> Oh. I misinterpreted what this feature was about, then. I thought it >> was about restricting the reference to a subset of the *referenced* >> table, but it seems to be about restricting the constraint to a subset >> of the *referencing*

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Peter Eisentraut
src/backend/access/transam/xlog.c:5889: trailing whitespace. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 12:00 PM, Robert Haas wrote: On Wed, Dec 4, 2013 at 11:44 AM, Andrew Dunstan wrote: Oh. I misinterpreted what this feature was about, then. I thought it was about restricting the reference to a subset of the *referenced* table, but it seems to be about restricting the constrain

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Magnus Hagander
On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Dec 3, 2013 at 7:11 PM, Tom Lane wrote: > >> Maybe we should just bite the bullet and change the WAL format for > >> heap_freeze (inventing an all-new record type, not repurposing the old > >> one, and allowin

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Wed, 2013-12-04 at 09:50 -0500, Stephen Frost wrote: > > I still don't see that Extension Templates are all bad: > > * They preserve the fact that two instances of the same extension > > (e.g. in different databases) were created from the same template. > > This is only true if we change the

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 04:33 PM, Jonathan Corbet wrote: > On Tue, 03 Dec 2013 10:44:15 -0800 > Josh Berkus wrote: > >> It seems clear that Kernel.org, since 2.6, has been in the business of >> pushing major, hackish, changes to the IO stack without testing them or >> even thinking too hard about what the

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: > Stephen Frost writes: > > When it comes to dump/reload, I'd much rather see a mechanism which uses > > our deep understanding of the extension's objects (as database objects) > > to implement the dump/reload than a text blob which is carried for

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 12:18 PM, Andrew Dunstan wrote: >> Interestingly, the variant for which you can't think of a use case is >> the one I've missed most. Typical examples in my experience are >> things like project.project_manager_id references person (id) where >> person.is_project_manager, o

Re: [HACKERS] Changes in Trigger Firing

2013-12-04 Thread Sameer Kumar
> > > > > > CreateTrigStmt is passed to CreateTrigger function as an arguement. I am > > struggling to understand how the values for various members of trigger > are > > set and where [which file] calls CreateTrigStmt. > > > > > > Can someone provide some help on this? > > I think you need better t

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Claudio Freire
On Wed, Dec 4, 2013 at 1:54 PM, Andres Freund wrote: > On 2013-12-04 18:43:35 +0200, Metin Doslu wrote: >> > I'd strongly suggest doing a "perf record -g -a ; >> > perf report" run to check what's eating up the time. >> >> Here is one example: >> >> + 38.87% swapper [kernel.kallsyms] [k] hyp

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> You could try HVM. I've noticed it fare better under heavy CPU load, > and it's not fully-HVM (it still uses paravirtualized network and > I/O). I already tried with HVM (cc2.8xlarge instance on Amazon EC2) and observed same problem.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 16:00:40 -0200, Claudio Freire wrote: > On Wed, Dec 4, 2013 at 1:54 PM, Andres Freund wrote: > > All that time is spent in your virtualization solution. One thing to try > > is to look on the host system, sometimes profiles there can be more > > meaningful. > > You cannot profile th

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> Didn't follow the thread from the start. So, this is EC2? Have you > checked, with a recent enough version of top or whatever, how much time > is reported as "stolen"? Yes, this EC2. "stolen" is randomly reported as 1, mostly as 0.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
Here are some extra information: - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is disappeared for 8 core machines and come back with 16 core machines on Amazon EC2. Would it be related with PostgreSQL locking mechanism? - I tried this test with 4 core machines including my perso

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 20:19:55 +0200, Metin Doslu wrote: > - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is > disappeared for 8 core machines and come back with 16 core machines on > Amazon EC2. Would it be related with PostgreSQL locking mechanism? You could try my lwlock-scalability im

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> You could try my lwlock-scalability improvement patches - for some > workloads here, the improvements have been rather noticeable. Which > version are you testing? I'm testing with PostgreSQL 9.3.1.

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Joshua D. Drake
On 12/04/2013 07:32 AM, Stefan Kaltenbrunner wrote: On 12/04/2013 04:30 PM, Peter Eisentraut wrote: On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved problem. Is it sound to run performance tests on

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 07:30 PM, Joshua D. Drake wrote: > > On 12/04/2013 07:32 AM, Stefan Kaltenbrunner wrote: >> >> On 12/04/2013 04:30 PM, Peter Eisentraut wrote: >>> On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something t

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Merlin Moncure
On Mon, Dec 2, 2013 at 10:26 PM, David Fetter wrote: > On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: >> On 11/28/2013 03:24 AM, David Fetter wrote: >> > WITH, or SRF, or whatever, the point is that we need to be able to >> > specify what we're sending--probably single opaque string

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 11:22 AM, Álvaro Hernández Tortosa wrote: > Would it be well-received a new file format that keeps it simple for > both hand editing and generation of the configuration, and at the same > time offers the features I have mentioned? I don't see how that would work exactly: You want to ad

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 19:49, Peter Eisentraut wrote: On 12/4/13, 11:22 AM, Álvaro Hernández Tortosa wrote: Would it be well-received a new file format that keeps it simple for both hand editing and generation of the configuration, and at the same time offers the features I have mentioned? I don't see

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Josh Berkus
On 12/04/2013 07:33 AM, Jonathan Corbet wrote: > Wow, Josh, I'm surprised to hear this from you. Well, I figured it was too angry to propose for an LWN article. ;-) > The active/inactive list mechanism works great for the vast majority of > users. The second-use algorithm prevents a lot of patho

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> Well, okay, but you've not said anything that wouldn't be Tom> handled just as well by some logic that adds a fixed Tom> integer-constant-zero flag column to the rows going into the Tom> tuplesort. Adding such a column unconditionally even for non-hypothe

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread David Fetter
On Wed, Dec 04, 2013 at 12:43:44PM -0600, Merlin Moncure wrote: > On Mon, Dec 2, 2013 at 10:26 PM, David Fetter wrote: > > On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: > >> On 11/28/2013 03:24 AM, David Fetter wrote: > >> > WITH, or SRF, or whatever, the point is that we need to b

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Tom Lane
Andrew Dunstan writes: >>> Well I guess we could say something like: >>> >>> FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE >>> (b-condition) >>> >>> But it's somewhat ugly. > OK, those make sense. I wonder whether this should be done via a USING > clause on the constraint t

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Joshua D. Drake
On 12/04/2013 07:33 AM, Jonathan Corbet wrote: Wow, Josh, I'm surprised to hear this from you. The active/inactive list mechanism works great for the vast majority of users. The second-use algorithm prevents a lot of pathological behavior, like wiping out your entire cache by copying a big f

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Tom Lane
Magnus Hagander writes: > On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: >> I assume what would happen is the slave would PANIC upon seeing a WAL >> record code it didn't recognize. > I wonder if we should for the future have the START_REPLICATION command (or > the IDENTIFY_SYSTEM would probabl

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Tom Lane
Jeff Davis writes: > On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: >> Stephen Frost writes: >>> When it comes to dump/reload, I'd much rather see a mechanism which uses >>> our deep understanding of the extension's objects (as database objects) >>> to implement the dump/reload than a text bl

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
David Fetter writes: > The idea here is that such a happy situation will not obtain until > much later, if ever, and meanwhile, we need a way to get things > accomplished even if it's inelegant, inefficient, etc. The > alternative is that those things simply will not get accomplished at > all. I

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 02:40 PM, Tom Lane wrote: Andrew Dunstan writes: Well I guess we could say something like: FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE (b-condition) But it's somewhat ugly. OK, those make sense. I wonder whether this should be done via a USING clause on t

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Robert Haas
On Tue, Dec 3, 2013 at 11:44 AM, Dimitri Fontaine wrote: >> We should also consider the possibility of a user trying to >> deliberately install and older release. For example, if the user has >> 1.0, 1.0--1.1, 1.1, 1.1--1.2, and 1.2--1.0 (a downgrade script) with >> default_full_version = 1.2, an

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> Well, okay, but you've not said anything that wouldn't be > Tom> handled just as well by some logic that adds a fixed > Tom> integer-constant-zero flag column to the rows going into the > Tom> tuplesort. > Adding such a column uncondit

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Merlin Moncure
On Wed, Dec 4, 2013 at 1:39 PM, David Fetter wrote: > On Wed, Dec 04, 2013 at 12:43:44PM -0600, Merlin Moncure wrote: >> On Mon, Dec 2, 2013 at 10:26 PM, David Fetter wrote: >> > On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: >> >> On 11/28/2013 03:24 AM, David Fetter wrote: >> >>

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
Merlin Moncure writes: > The downside of SQL-MED, particularly the way postgres implemented the > driver API, is that each driver is responsible for for all > optimization efforts and I think this is bad. There was never any intention that that would be the final state of things. All the FDW API

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 3:39 AM, Jeff Davis wrote: > On Tue, 2013-12-03 at 10:23 -0500, Robert Haas wrote: >> In more normal cases, however, the system can (and probably should) >> figure out what was intended by choosing the *shortest* path to get to >> the intended version. For example, if someo

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Wed, 04 Dec 2013 11:07:04 -0800 Josh Berkus wrote: > On 12/04/2013 07:33 AM, Jonathan Corbet wrote: > > Wow, Josh, I'm surprised to hear this from you. > > Well, I figured it was too angry to propose for an LWN article. ;-) So you're going to make us write it for you :) > > The active/inact

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Magnus Hagander
On Wed, Dec 4, 2013 at 9:31 PM, Jonathan Corbet wrote: > > I also wasn't exaggerating the reception I got when I tried to talk > > about IO and PostgreSQL at LinuxCon and other events. The majority of > > Linux hackers I've talked to simply don't want to be bothered with > > PostgreSQL's perform

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Magnus Hagander
On Wed, Dec 4, 2013 at 8:43 PM, Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: > >> I assume what would happen is the slave would PANIC upon seeing a WAL > >> record code it didn't recognize. > > > I wonder if we should for the future have the STAR

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: > I think that's an excellent idea. If one of our developers could find the > time to attend that, I think that could be very productive. While I'm not > on the funds team, I'd definitely vote for funding such participation out > of community funds if

[HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
Having nothing better to do over the holiday weekend, I decided to pursue a number of ideas for improving performance that I thought about a long time ago. These include: * Pre-fetching list node pointers. This looks to be moderately promising, but I'm certainly not going to be the one to land it,

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> Well, sure, but I was only suggesting adding it when the Tom> aggregate asks for it, probably via a new flag column in Tom> pg_aggregate. Sure, I was only pointing out the necessity. Tom> The question you're evading is what additional functionality Tom>

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Josh Berkus
Jonathan, > For those interested in the details... (1) It's not quite 50/50, that's one > bound for how the balance is allowed to go. (2) Anybody trying to add > tunables to the kernel tends to run into resistance. Exposing thousands of > knobs tends to lead to a situation where you *have* to be

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Merlin Moncure
On Wed, Dec 4, 2013 at 2:31 PM, Jonathan Corbet wrote: > For those interested in the details... (1) It's not quite 50/50, that's one > bound for how the balance is allowed to go. (2) Anybody trying to add > tunables to the kernel tends to run into resistance. Exposing thousands of > knobs tends

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Wed, 04 Dec 2013 13:01:37 -0800 Josh Berkus wrote: > > Perhaps even better: the next filesystem, storage, and memory management > > summit is March 24-25. > > Link? I can't find anything Googling by that name. I'm pretty sure we > can get at least one person there. It looks like the page

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Peter Geoghegan writes: > I guess I could write a proper patch to have code setting up a scankey > also set a flag that indicated that it was acceptable to assume that > the special built-in comparator would do fine. ... > I'd be happy with a scheme with only one built-in comparator, and > allowed

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 4:28 PM, Tom Lane wrote: > Peter Geoghegan writes: >> I guess I could write a proper patch to have code setting up a scankey >> also set a flag that indicated that it was acceptable to assume that >> the special built-in comparator would do fine. ... >> I'd be happy with a

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> But anyway, what I'm thinking right now is that these questions > Tom> would all go away if the aggregate transfunction were receiving > Tom> the rows and sticking them into the tuplestore. It could add > Tom> whatever columns it felt

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/2/13, 9:14 AM, Dimitri Fontaine wrote: > What I want to build is an “extension distribution” software that knows > how to prepare anything from PGXN (and other places) so that it's fully > ready for being used in the database. Then the main client would run as > a CREATE EXTENSION "ddl_comman

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/2/13, 2:33 PM, Greg Stark wrote: > Just tossing an idea out there. What if you could install an extension > by specifying not a local file name but a URL. Obviously there's a > security issue but for example we could allow only https URLs with > verified domain names that are in a list of app

  1   2   >