Re: [HACKERS] Proposal: variant of regclass

2013-12-04 Thread Pavel Golub
Hello, Tom. You wrote: TL> Tatsuo Ishii writes: >> I would like to add a variant of regclass, which is exactly same as >> current regclass except it does not raise an error when the target >> table is not found. Instead it returns InvalidOid (0). TL> I've sometimes thought we should just make a

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-04 Thread Dean Rasheed
On 5 December 2013 01:33, Peter Eisentraut wrote: > Can someone in this thread clarify the commit fest situation? I see two > entries that appear to be the same: > > https://commitfest.postgresql.org/action/patch_view?id=1174 > https://commitfest.postgresql.org/action/patch_view?id=1175 > > I thi

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-04 Thread Sergey Muraviov
And my patch affects the row view only. postgres=# \x 1 postgres=# create table wide_table (value text); postgres=# insert into wide_table values ('afadsafasd fasdf asdfasd fsad fas df sadf sad f sadf sadf sa df sadfsadfasd fsad fsa df sadf asd fa sfd sadfsadf asdf sad f sadf sad fadsf'); postgre

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Wed, 2013-12-04 at 15:28 -0500, Robert Haas wrote: > My experience with software upgrades is that > cases like this, and even weirder things, happen pretty routinely, so > I think more control is good. There would still be control: just use full SQL scripts appropriately. I'm sure there's stil

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Peter Eisentraut writes: > On Wed, 2013-12-04 at 20:27 -0500, Tom Lane wrote: >> Lazy people? I'm not in a hurry to drop it; it's not costing us much to >> just sit there, other than in this connection which we see how to fix. > Actually, I think it probably costs a fair portion of extension aut

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-04 Thread Peter Eisentraut
On Thu, 2013-12-05 at 09:02 +0530, Amit Kapila wrote: > This is certainly not a stupid idea, rather something on similar lines > has been discussed previously in this thread. > Tom has suggested something similar, but I am not sure if there was a > conclusion on that point. Please see the > relavan

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Eisentraut
On Wed, 2013-12-04 at 20:27 -0500, Tom Lane wrote: > Lazy people? I'm not in a hurry to drop it; it's not costing us much to > just sit there, other than in this connection which we see how to fix. Actually, I think it probably costs a fair portion of extension authors when their initial code cra

Re: [HACKERS] [bug fix] pg_ctl fails with config-only directory

2013-12-04 Thread Amit Kapila
On Wed, Dec 4, 2013 at 7:57 PM, MauMau wrote: > Hello, > > I've found a bug and would like to fix it, but I cannot figure out how to do > that well. Could you give me any advice? I encountered this on PG 9.2, but > it will probably exist in later versions. > > [Problem] > On Windows, a user with

[HACKERS] same-address mappings vs. relative pointers

2013-12-04 Thread Robert Haas
During development of the dynamic shared memory facility, Noah and I spent a lot of time arguing about whether it was practical to ensure that a dynamic shared memory segment got mapped at the same address in every backend that used it. The argument went something like this: Me: We'll never be ab

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Amit Kapila
On Wed, Dec 4, 2013 at 11:49 PM, Metin Doslu wrote: > Here are some extra information: > > - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is > disappeared for 8 core machines and come back with 16 core machines on > Amazon EC2. Would it be related with PostgreSQL locking mechanism

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-12-04 Thread Ian Lawrence Barwick
2013/12/5 Albe Laurenz : > Ian Lawrence Barwick wrote: >> 2013/11/8 Tom Lane : >>> [ thinks for awhile... ] Hm. In principle you can put any expression >>> you want into the tlist during AddForeignUpdateTargets. However, if it's >>> not a Var then the planner won't understand that it's something

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Amit Kapila
On Wed, Dec 4, 2013 at 10:40 AM, Claudio Freire wrote: > On Wed, Dec 4, 2013 at 12:57 AM, Amit Kapila wrote: >>> As a quick side, we also repeated the same experiment on an EC2 instance >>> with 16 CPU cores, and found that the scale out behavior became worse there. >>> (We also tried increasing

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-04 Thread Amit Kapila
On Thu, Dec 5, 2013 at 7:25 AM, Peter Eisentraut wrote: > On Thu, 2013-11-14 at 12:11 +0530, Amit Kapila wrote: >>If an application wants to allow these connection parameters to be >> used, it would need to do PQenableStartServer() first. If it doesn't, >> those connection parameters will be r

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Stephen Frost
* Peter Eisentraut (pete...@gmx.net) wrote: > On 12/3/13, 9:20 AM, Stephen Frost wrote: > > Another option, which I generally like better, is to have a new package > > format for PGXN that contains the results of "make install", > > more-or-less, synonymous to Debian source vs. .deb packages. > >

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 5:28 PM, Peter Geoghegan wrote: > I'm also curious about the impact on insertion into primary key > indexes. Presently, we hold an exclusive buffer lock for the duration > of a couple of operations when checkUnique != UNIQUE_CHECK_NO. > _bt_binsrch() is one such operation. T

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-04 Thread Peter Eisentraut
On Thu, 2013-11-14 at 12:11 +0530, Amit Kapila wrote: >If an application wants to allow these connection parameters to be > used, it would need to do PQenableStartServer() first. If it doesn't, > those connection parameters will be rejected. Stupid idea: Would it work that we require an enviro

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-04 Thread Peter Eisentraut
Can someone in this thread clarify the commit fest situation? I see two entries that appear to be the same: https://commitfest.postgresql.org/action/patch_view?id=1174 https://commitfest.postgresql.org/action/patch_view?id=1175 I think the first one is a duplicate or obsolete. -- Sent via p

Re: [HACKERS] Proposal: variant of regclass

2013-12-04 Thread Tatsuo Ishii
> Tatsuo Ishii writes: >> I would like to add a variant of regclass, which is exactly same as >> current regclass except it does not raise an error when the target >> table is not found. Instead it returns InvalidOid (0). > > I've sometimes thought we should just make all the reg* input converter

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 12:58 PM, Peter Geoghegan wrote: > I'm kind of > curious as to what this benchmark would like like on a server with > many more cores. I'm also curious about the impact on insertion into primary key indexes. Presently, we hold an exclusive buffer lock for the duration of a

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Peter Eisentraut writes: > On Wed, 2013-12-04 at 19:45 -0500, Robert Haas wrote: >> On Wed, Dec 4, 2013 at 6:56 PM, Tom Lane wrote: >>> Yeah, that's another thing we could simplify if we fixed this problem >>> at the source. I think these decisions date from a time when we still >>> cared about

Re: [HACKERS] Proposal: variant of regclass

2013-12-04 Thread Tom Lane
Tatsuo Ishii writes: > I would like to add a variant of regclass, which is exactly same as > current regclass except it does not raise an error when the target > table is not found. Instead it returns InvalidOid (0). I've sometimes thought we should just make all the reg* input converters act tha

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 2:02 PM, Álvaro Hernández Tortosa wrote: > So optional fields are either purely optional (i.e., only for tools > that want to use them; everyone else may ignore, but preserve, them) and > some other are just NULLABLEs, depending on the parameter). But my point stands: If it's optio

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Eisentraut
On Wed, 2013-12-04 at 19:45 -0500, Robert Haas wrote: > On Wed, Dec 4, 2013 at 6:56 PM, Tom Lane wrote: > > Yeah, that's another thing we could simplify if we fixed this problem > > at the source. I think these decisions date from a time when we still > > cared about the speed of fmgr_oldstyle. >

Re: [HACKERS] Changes in Trigger Firing

2013-12-04 Thread Michael Paquier
On Wed, Dec 4, 2013 at 11:16 PM, Alvaro Herrera wrote: > Sameer Kumar wrote: > I think you need better tools to guide you in exploring the source code. > For example, you can use cscope to tell you where is CreateTrigStmt > used, and you would find gram.y; and use it to tell you where > CreateTrig

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
David Fetter writes: > On Wed, Dec 04, 2013 at 03:04:31PM -0500, Tom Lane wrote: >> If that's the argument, why not just use dblink or dbilink, and be >> happy? This discussion sounds a whole lot like it's trending to a >> conclusion of wanting one of those in core, which is not where I'd >> like

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Andres Freund
Hi, On 2013-12-03 10:44:15 -0800, Josh Berkus wrote: > I don't know where we'll get the resources to implement our own storage, > but it's looking like we don't have a choice. As long as our storage layer is a s suboptimal as it is today, I think it's a purely detractory to primarily blame the ke

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 6:56 PM, Tom Lane wrote: > Robert Haas writes: >> Hmm. And yet, there's this: > >> * When a type narrower than Datum is stored in a Datum, we place it in the >> * low-order bits and are careful that the DatumGetXXX macro for it discards >> * the unused high-order bits (

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Simon Riggs
On 3 December 2013 18:46, Robert Haas wrote: > On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello > wrote: >> On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse >> wrote: >>> >>> Hi Fabrizio, >>> >>> looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It >>> applies and compiles w/

[HACKERS] Proposal: variant of regclass

2013-12-04 Thread Tatsuo Ishii
I would like to propose to add a variant of regclass. Background: Pgpool-II (http://www.pgpool.net) needs to get information of tables by querying PostgreSQL's system catalog. For efficiency and correctness of the info (search path consideration), pgpool-II issues such queries piggy packing the us

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread David Fetter
On Wed, Dec 04, 2013 at 03:04:31PM -0500, Tom Lane wrote: > David Fetter writes: > > The idea here is that such a happy situation will not obtain until > > much later, if ever, and meanwhile, we need a way to get things > > accomplished even if it's inelegant, inefficient, etc. The > > alternativ

[HACKERS] Changeset Extraction Interfaces

2013-12-04 Thread Andres Freund
Hi, Short recap: >From the perspective of the user interface the changeset extraction feature consists out of two abstract interfaces that the "user" has to do with: 1) The "slot" or "changestream" management interface which manages individual streams of changes. The user can create and destroy

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
I wrote: > Yeah, that's another thing we could simplify if we fixed this problem > at the source. I think these decisions date from a time when we still > cared about the speed of fmgr_oldstyle. BTW, the text you're quoting is from 2007, but it's just documenting behavior that's mostly a lot olde

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Robert Haas writes: > Hmm. And yet, there's this: > * When a type narrower than Datum is stored in a Datum, we place it in the > * low-order bits and are careful that the DatumGetXXX macro for it discards > * the unused high-order bits (as opposed to, say, assuming they are zero). > * This i

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 20:44, Peter Eisentraut wrote: On 12/4/13, 2:02 PM, Álvaro Hernández Tortosa wrote: So optional fields are either purely optional (i.e., only for tools that want to use them; everyone else may ignore, but preserve, them) and some other are just NULLABLEs, depending on the para

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 6:33 PM, Peter Geoghegan wrote: > On Wed, Dec 4, 2013 at 1:59 PM, Robert Haas wrote: >> Yeah, I think if we can make something like this work, it would be >> neat-o. Getting this working for int4 would be a good win, as Peter >> says, but getting it working for both int4 a

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Peter Geoghegan writes: > On Wed, Dec 4, 2013 at 1:59 PM, Robert Haas wrote: >> Yeah, I think if we can make something like this work, it would be >> neat-o. Getting this working for int4 would be a good win, as Peter >> says, but getting it working for both int4 and int8 with the same code >> w

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 1:59 PM, Robert Haas wrote: > Yeah, I think if we can make something like this work, it would be > neat-o. Getting this working for int4 would be a good win, as Peter > says, but getting it working for both int4 and int8 with the same code > would be a significantly better

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Tom Dunstan
On 5 Dec 2013, at 03:48, Andrew Dunstan wrote: >>> Well I guess we could say something like: >>> >>>FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE >>>(b-condition) >>> > > OK, those make sense. I wonder whether this should be done via a USING clause > on the constra

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Tom Dunstan
On 5 Dec 2013, at 06:10, Tom Lane wrote: > Andrew Dunstan writes: Well I guess we could say something like: FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE (b-condition) > > I like what you have above. Yeah. Given both the apparent ambiguity of the

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/3/13, 9:20 AM, Stephen Frost wrote: > Another option, which I generally like better, is to have a new package > format for PGXN that contains the results of "make install", > more-or-less, synonymous to Debian source vs. .deb packages. > > Perhaps we could even have psql understand that form

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/2/13, 2:33 PM, Greg Stark wrote: > Just tossing an idea out there. What if you could install an extension > by specifying not a local file name but a URL. Obviously there's a > security issue but for example we could allow only https URLs with > verified domain names that are in a list of app

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Peter Eisentraut
On 12/2/13, 9:14 AM, Dimitri Fontaine wrote: > What I want to build is an “extension distribution” software that knows > how to prepare anything from PGXN (and other places) so that it's fully > ready for being used in the database. Then the main client would run as > a CREATE EXTENSION "ddl_comman

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> But anyway, what I'm thinking right now is that these questions > Tom> would all go away if the aggregate transfunction were receiving > Tom> the rows and sticking them into the tuplestore. It could add > Tom> whatever columns it felt

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 4:28 PM, Tom Lane wrote: > Peter Geoghegan writes: >> I guess I could write a proper patch to have code setting up a scankey >> also set a flag that indicated that it was acceptable to assume that >> the special built-in comparator would do fine. ... >> I'd be happy with a

Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Tom Lane
Peter Geoghegan writes: > I guess I could write a proper patch to have code setting up a scankey > also set a flag that indicated that it was acceptable to assume that > the special built-in comparator would do fine. ... > I'd be happy with a scheme with only one built-in comparator, and > allowed

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Wed, 04 Dec 2013 13:01:37 -0800 Josh Berkus wrote: > > Perhaps even better: the next filesystem, storage, and memory management > > summit is March 24-25. > > Link? I can't find anything Googling by that name. I'm pretty sure we > can get at least one person there. It looks like the page

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Merlin Moncure
On Wed, Dec 4, 2013 at 2:31 PM, Jonathan Corbet wrote: > For those interested in the details... (1) It's not quite 50/50, that's one > bound for how the balance is allowed to go. (2) Anybody trying to add > tunables to the kernel tends to run into resistance. Exposing thousands of > knobs tends

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Josh Berkus
Jonathan, > For those interested in the details... (1) It's not quite 50/50, that's one > bound for how the balance is allowed to go. (2) Anybody trying to add > tunables to the kernel tends to run into resistance. Exposing thousands of > knobs tends to lead to a situation where you *have* to be

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> Well, sure, but I was only suggesting adding it when the Tom> aggregate asks for it, probably via a new flag column in Tom> pg_aggregate. Sure, I was only pointing out the necessity. Tom> The question you're evading is what additional functionality Tom>

[HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
Having nothing better to do over the holiday weekend, I decided to pursue a number of ideas for improving performance that I thought about a long time ago. These include: * Pre-fetching list node pointers. This looks to be moderately promising, but I'm certainly not going to be the one to land it,

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: > I think that's an excellent idea. If one of our developers could find the > time to attend that, I think that could be very productive. While I'm not > on the funds team, I'd definitely vote for funding such participation out > of community funds if

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Magnus Hagander
On Wed, Dec 4, 2013 at 8:43 PM, Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: > >> I assume what would happen is the slave would PANIC upon seeing a WAL > >> record code it didn't recognize. > > > I wonder if we should for the future have the STAR

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Magnus Hagander
On Wed, Dec 4, 2013 at 9:31 PM, Jonathan Corbet wrote: > > I also wasn't exaggerating the reception I got when I tried to talk > > about IO and PostgreSQL at LinuxCon and other events. The majority of > > Linux hackers I've talked to simply don't want to be bothered with > > PostgreSQL's perform

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Jonathan Corbet
On Wed, 04 Dec 2013 11:07:04 -0800 Josh Berkus wrote: > On 12/04/2013 07:33 AM, Jonathan Corbet wrote: > > Wow, Josh, I'm surprised to hear this from you. > > Well, I figured it was too angry to propose for an LWN article. ;-) So you're going to make us write it for you :) > > The active/inact

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 3:39 AM, Jeff Davis wrote: > On Tue, 2013-12-03 at 10:23 -0500, Robert Haas wrote: >> In more normal cases, however, the system can (and probably should) >> figure out what was intended by choosing the *shortest* path to get to >> the intended version. For example, if someo

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
Merlin Moncure writes: > The downside of SQL-MED, particularly the way postgres implemented the > driver API, is that each driver is responsible for for all > optimization efforts and I think this is bad. There was never any intention that that would be the final state of things. All the FDW API

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Merlin Moncure
On Wed, Dec 4, 2013 at 1:39 PM, David Fetter wrote: > On Wed, Dec 04, 2013 at 12:43:44PM -0600, Merlin Moncure wrote: >> On Mon, Dec 2, 2013 at 10:26 PM, David Fetter wrote: >> > On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: >> >> On 11/28/2013 03:24 AM, David Fetter wrote: >> >>

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> Well, okay, but you've not said anything that wouldn't be > Tom> handled just as well by some logic that adds a fixed > Tom> integer-constant-zero flag column to the rows going into the > Tom> tuplesort. > Adding such a column uncondit

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Robert Haas
On Tue, Dec 3, 2013 at 11:44 AM, Dimitri Fontaine wrote: >> We should also consider the possibility of a user trying to >> deliberately install and older release. For example, if the user has >> 1.0, 1.0--1.1, 1.1, 1.1--1.2, and 1.2--1.0 (a downgrade script) with >> default_full_version = 1.2, an

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 02:40 PM, Tom Lane wrote: Andrew Dunstan writes: Well I guess we could say something like: FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE (b-condition) But it's somewhat ugly. OK, those make sense. I wonder whether this should be done via a USING clause on t

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Tom Lane
David Fetter writes: > The idea here is that such a happy situation will not obtain until > much later, if ever, and meanwhile, we need a way to get things > accomplished even if it's inelegant, inefficient, etc. The > alternative is that those things simply will not get accomplished at > all. I

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Tom Lane
Jeff Davis writes: > On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: >> Stephen Frost writes: >>> When it comes to dump/reload, I'd much rather see a mechanism which uses >>> our deep understanding of the extension's objects (as database objects) >>> to implement the dump/reload than a text bl

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Tom Lane
Magnus Hagander writes: > On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: >> I assume what would happen is the slave would PANIC upon seeing a WAL >> record code it didn't recognize. > I wonder if we should for the future have the START_REPLICATION command (or > the IDENTIFY_SYSTEM would probabl

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Joshua D. Drake
On 12/04/2013 07:33 AM, Jonathan Corbet wrote: Wow, Josh, I'm surprised to hear this from you. The active/inactive list mechanism works great for the vast majority of users. The second-use algorithm prevents a lot of pathological behavior, like wiping out your entire cache by copying a big f

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Tom Lane
Andrew Dunstan writes: >>> Well I guess we could say something like: >>> >>> FOREIGN KEY (a-col) WHERE (a-condition) REFERENCES b(b-col) WHERE >>> (b-condition) >>> >>> But it's somewhat ugly. > OK, those make sense. I wonder whether this should be done via a USING > clause on the constraint t

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread David Fetter
On Wed, Dec 04, 2013 at 12:43:44PM -0600, Merlin Moncure wrote: > On Mon, Dec 2, 2013 at 10:26 PM, David Fetter wrote: > > On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: > >> On 11/28/2013 03:24 AM, David Fetter wrote: > >> > WITH, or SRF, or whatever, the point is that we need to b

Re: [HACKERS] WITHIN GROUP patch

2013-12-04 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> Well, okay, but you've not said anything that wouldn't be Tom> handled just as well by some logic that adds a fixed Tom> integer-constant-zero flag column to the rows going into the Tom> tuplesort. Adding such a column unconditionally even for non-hypothe

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Josh Berkus
On 12/04/2013 07:33 AM, Jonathan Corbet wrote: > Wow, Josh, I'm surprised to hear this from you. Well, I figured it was too angry to propose for an LWN article. ;-) > The active/inactive list mechanism works great for the vast majority of > users. The second-use algorithm prevents a lot of patho

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 19:49, Peter Eisentraut wrote: On 12/4/13, 11:22 AM, Álvaro Hernández Tortosa wrote: Would it be well-received a new file format that keeps it simple for both hand editing and generation of the configuration, and at the same time offers the features I have mentioned? I don't see

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 11:22 AM, Álvaro Hernández Tortosa wrote: > Would it be well-received a new file format that keeps it simple for > both hand editing and generation of the configuration, and at the same > time offers the features I have mentioned? I don't see how that would work exactly: You want to ad

Re: [HACKERS] Status of FDW pushdowns

2013-12-04 Thread Merlin Moncure
On Mon, Dec 2, 2013 at 10:26 PM, David Fetter wrote: > On Tue, Dec 03, 2013 at 11:15:36AM +0800, Craig Ringer wrote: >> On 11/28/2013 03:24 AM, David Fetter wrote: >> > WITH, or SRF, or whatever, the point is that we need to be able to >> > specify what we're sending--probably single opaque string

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 07:30 PM, Joshua D. Drake wrote: > > On 12/04/2013 07:32 AM, Stefan Kaltenbrunner wrote: >> >> On 12/04/2013 04:30 PM, Peter Eisentraut wrote: >>> On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something t

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Joshua D. Drake
On 12/04/2013 07:32 AM, Stefan Kaltenbrunner wrote: On 12/04/2013 04:30 PM, Peter Eisentraut wrote: On 12/4/13, 2:14 AM, Stefan Kaltenbrunner wrote: running a few kvm instances that get bootstrapped automatically is something that is a solved problem. Is it sound to run performance tests on

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> You could try my lwlock-scalability improvement patches - for some > workloads here, the improvements have been rather noticeable. Which > version are you testing? I'm testing with PostgreSQL 9.3.1.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 20:19:55 +0200, Metin Doslu wrote: > - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is > disappeared for 8 core machines and come back with 16 core machines on > Amazon EC2. Would it be related with PostgreSQL locking mechanism? You could try my lwlock-scalability im

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
Here are some extra information: - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is disappeared for 8 core machines and come back with 16 core machines on Amazon EC2. Would it be related with PostgreSQL locking mechanism? - I tried this test with 4 core machines including my perso

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> Didn't follow the thread from the start. So, this is EC2? Have you > checked, with a recent enough version of top or whatever, how much time > is reported as "stolen"? Yes, this EC2. "stolen" is randomly reported as 1, mostly as 0.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 16:00:40 -0200, Claudio Freire wrote: > On Wed, Dec 4, 2013 at 1:54 PM, Andres Freund wrote: > > All that time is spent in your virtualization solution. One thing to try > > is to look on the host system, sometimes profiles there can be more > > meaningful. > > You cannot profile th

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> You could try HVM. I've noticed it fare better under heavy CPU load, > and it's not fully-HVM (it still uses paravirtualized network and > I/O). I already tried with HVM (cc2.8xlarge instance on Amazon EC2) and observed same problem.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Claudio Freire
On Wed, Dec 4, 2013 at 1:54 PM, Andres Freund wrote: > On 2013-12-04 18:43:35 +0200, Metin Doslu wrote: >> > I'd strongly suggest doing a "perf record -g -a ; >> > perf report" run to check what's eating up the time. >> >> Here is one example: >> >> + 38.87% swapper [kernel.kallsyms] [k] hyp

Re: [HACKERS] Changes in Trigger Firing

2013-12-04 Thread Sameer Kumar
> > > > > > CreateTrigStmt is passed to CreateTrigger function as an arguement. I am > > struggling to understand how the values for various members of trigger > are > > set and where [which file] calls CreateTrigStmt. > > > > > > Can someone provide some help on this? > > I think you need better t

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 12:18 PM, Andrew Dunstan wrote: >> Interestingly, the variant for which you can't think of a use case is >> the one I've missed most. Typical examples in my experience are >> things like project.project_manager_id references person (id) where >> person.is_project_manager, o

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: > Stephen Frost writes: > > When it comes to dump/reload, I'd much rather see a mechanism which uses > > our deep understanding of the extension's objects (as database objects) > > to implement the dump/reload than a text blob which is carried for

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-04 Thread Stefan Kaltenbrunner
On 12/04/2013 04:33 PM, Jonathan Corbet wrote: > On Tue, 03 Dec 2013 10:44:15 -0800 > Josh Berkus wrote: > >> It seems clear that Kernel.org, since 2.6, has been in the business of >> pushing major, hackish, changes to the IO stack without testing them or >> even thinking too hard about what the

Re: [HACKERS] Extension Templates S03E11

2013-12-04 Thread Jeff Davis
On Wed, 2013-12-04 at 09:50 -0500, Stephen Frost wrote: > > I still don't see that Extension Templates are all bad: > > * They preserve the fact that two instances of the same extension > > (e.g. in different databases) were created from the same template. > > This is only true if we change the

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-04 Thread Magnus Hagander
On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Dec 3, 2013 at 7:11 PM, Tom Lane wrote: > >> Maybe we should just bite the bullet and change the WAL format for > >> heap_freeze (inventing an all-new record type, not repurposing the old > >> one, and allowin

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 12:00 PM, Robert Haas wrote: On Wed, Dec 4, 2013 at 11:44 AM, Andrew Dunstan wrote: Oh. I misinterpreted what this feature was about, then. I thought it was about restricting the reference to a subset of the *referenced* table, but it seems to be about restricting the constrain

Re: [HACKERS] Time-Delayed Standbys

2013-12-04 Thread Peter Eisentraut
src/backend/access/transam/xlog.c:5889: trailing whitespace. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Wed, Dec 4, 2013 at 11:44 AM, Andrew Dunstan wrote: >> Oh. I misinterpreted what this feature was about, then. I thought it >> was about restricting the reference to a subset of the *referenced* >> table, but it seems to be about restricting the constraint to a subset >> of the *referencing*

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 18:43:35 +0200, Metin Doslu wrote: > > I'd strongly suggest doing a "perf record -g -a ; > > perf report" run to check what's eating up the time. > > Here is one example: > > + 38.87% swapper [kernel.kallsyms] [k] hypercall_page > + 9.32% postgres [kernel.kallsyms] [k] h

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Andrew Dunstan
On 12/04/2013 11:25 AM, Robert Haas wrote: On Tue, Dec 3, 2013 at 5:57 PM, Tom Dunstan wrote: On 4 December 2013 01:24, Robert Haas wrote: Yeah, more or less, but the key is ensuring that it wouldn't let you create the constraint in the first place if the partial index specified *didn't* mat

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
> I'd strongly suggest doing a "perf record -g -a ; > perf report" run to check what's eating up the time. Here is one example: + 38.87% swapper [kernel.kallsyms] [k] hypercall_page + 9.32% postgres [kernel.kallsyms] [k] hypercall_page + 6.80% postgres [kernel.kallsyms] [k] xen_

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Metin Doslu
>Notice the huge %sy >What kind of VM are you using? HVM or paravirtual? This instance is paravirtual.

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Andres Freund
On 2013-12-04 14:27:10 -0200, Claudio Freire wrote: > On Wed, Dec 4, 2013 at 9:19 AM, Metin Doslu wrote: > > > > Here are the results of "vmstat 1" while running 8 parallel TPC-H Simple > > (#6) queries: Although there is no need for I/O, "wa" fluctuates between 0 > > and 1. > > > > procs ---

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-04 Thread Claudio Freire
On Wed, Dec 4, 2013 at 9:19 AM, Metin Doslu wrote: > > Here are the results of "vmstat 1" while running 8 parallel TPC-H Simple > (#6) queries: Although there is no need for I/O, "wa" fluctuates between 0 > and 1. > > procs ---memory-- ---swap-- -io --system-- > -cpu--

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-04 Thread Robert Haas
On Tue, Dec 3, 2013 at 5:57 PM, Tom Dunstan wrote: > On 4 December 2013 01:24, Robert Haas wrote: >> Yeah, more or less, but the key is ensuring that it wouldn't let you >> create the constraint in the first place if the partial index >> specified *didn't* match the WHERE clause. For example, su

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Álvaro Hernández Tortosa
On 04/12/13 16:51, Peter Eisentraut wrote: On 12/4/13, 1:42 AM, Álvaro Hernández Tortosa wrote: IMHO, a data structure like the above would be completely self-contained and allow any autoconfiguring tool or GUI tool to be easily created, if the syntax is programmable. It would certainly m

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-12-04 Thread Albe Laurenz
Ian Lawrence Barwick wrote: > 2013/11/8 Tom Lane : >> [ thinks for awhile... ] Hm. In principle you can put any expression >> you want into the tlist during AddForeignUpdateTargets. However, if it's >> not a Var then the planner won't understand that it's something that needs >> to be supplied b

Re: [HACKERS] Minor patch for the uuid-ossp extension

2013-12-04 Thread Peter Eisentraut
On 11/23/13, 7:12 AM, Mario Weilguni wrote: > Well, in that case and since this is a rarely used extension (I guess > so), maybe it would be the best to simply rename that extension to > uuidossp (or whatever) and don't make any special treatment for it? Why? This is a solved problem, and renamin

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-04 Thread Peter Eisentraut
On 12/4/13, 1:42 AM, Álvaro Hernández Tortosa wrote: > IMHO, a data structure like the above would be completely > self-contained and allow any autoconfiguring tool or GUI tool to be > easily created, if the syntax is programmable. It would certainly make > the config file more verbose, but at

  1   2   >