Re: [HACKERS] Get more from indices.

2013-12-05 Thread Kyotaro HORIGUCHI
Hello, > > I've modified the patch to work in such a way. Also, as ISTM the patch > > is more complicated than what the patch really does, I've simplified the > > patch. > > I've revised the patch a bit. Please find attached the patch. Thank you, but it seems to me too simplified. You made tw

[HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Michael Paquier
Hi all, It happens that the following regression tests are failing if they are run on a database not named "regression": - updatable_views - foreign_data - sequence Those tests are failing because some relations of information_schemas contain information that are database-dependent. Please see the

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Michael Paquier
On Thu, Dec 5, 2013 at 5:12 PM, Michael Paquier wrote: > Those tests are failing because some relations of information_schemas > contain information that are database-dependent. I forgot to mention that it is our QE team at VMware that reported me a portion of this failure, and I just had to dig a

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread KONDO Mitsumasa
(2013/12/04 16:39), Claudio Freire wrote: On Wed, Dec 4, 2013 at 4:28 AM, Tatsuo Ishii wrote: Can we avoid the Linux kernel problem by simply increasing our shared buffer size, say up to 80% of memory? It will be swap more easier. Is that the case? If the system has not enough memory, the ke

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 11:07 AM, Josh Berkus wrote: > I also wasn't exaggerating the reception I got when I tried to talk > about IO and PostgreSQL at LinuxCon and other events. The majority of > Linux hackers I've talked to simply don't want to be bothered with > PostgreSQL's performance needs,

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Magnus Hagander
On Thu, Dec 5, 2013 at 1:45 AM, Simon Riggs wrote: > On 3 December 2013 18:46, Robert Haas wrote: > > On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello > > wrote: > >> On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse < > christ...@2ndquadrant.com> > >> wrote: > >>> > >>> Hi Fabrizio, > >>

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is > disappeared for 8 core machines and come back with 16 core machines on > Amazon EC2. Would it be related with PostgreSQL locking mechanism? If we build with -DLWLOCK_STATS to print locking stats from PostgreSQL, we see tons of

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Andres Freund
On 2013-12-05 11:15:20 +0200, Metin Doslu wrote: > > - When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is > > disappeared for 8 core machines and come back with 16 core machines on > > Amazon EC2. Would it be related with PostgreSQL locking mechanism? > > If we build with -DLWLOCK_ST

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> Is your workload bigger than RAM? RAM is bigger than workload (more than a couple of times). > I think a good bit of the contention > you're seeing in that listing is populating shared_buffers - and might > actually vanish once you're halfway cached. > From what I've seen so far the bigger prob

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Andres Freund
On 2013-12-05 11:33:29 +0200, Metin Doslu wrote: > > Is your workload bigger than RAM? > > RAM is bigger than workload (more than a couple of times). > > I think a good bit of the contention > > you're seeing in that listing is populating shared_buffers - and might > > actually vanish once you're

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
Hi Robert, On 2013-12-04 23:32:27 -0500, Robert Haas wrote: > But I'm also learning painfully that this kind of thing only goes so > far. For example, I spent some time looking at what it would take to > provide a dynamic shared memory equivalent of palloc/pfree, a facility > that I feel fairly s

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Simon Riggs
On 5 December 2013 08:51, Magnus Hagander wrote: > Not recalling the older thread, but it seems the "breaks on clock drift", I > think we can fairly easily make that situation "good enough". Just have > IDENTIFY_SYSTEM return the current timestamp on the master, and refuse to > start if the time

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-05 Thread Pavel Stehule
Hello 2013/12/5 Peter Eisentraut > Can someone in this thread clarify the commit fest situation? I see two > entries that appear to be the same: > > https://commitfest.postgresql.org/action/patch_view?id=1174 > https://commitfest.postgresql.org/action/patch_view?id=1175 > > I think the first o

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-05 Thread Pavel Stehule
2013/12/5 Dean Rasheed > On 5 December 2013 01:33, Peter Eisentraut wrote: > > Can someone in this thread clarify the commit fest situation? I see two > > entries that appear to be the same: > > > > https://commitfest.postgresql.org/action/patch_view?id=1174 > > https://commitfest.postgresql.or

Re: [HACKERS] Changes in Trigger Firing

2013-12-05 Thread Sameer Kumar
> > > I read somewhere that "the best editor is the > one you master" (1) :) > 1: http://www.postgresql.org/message-id/m2wrs6giyp@hi-media.com > > Thanks, I am using eclipse now. Any comments about the utility of this feature? Or is it just me who thinks this can be useful? I think users/dev

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 5:28 PM, Peter Geoghegan wrote: > I'm also curious about the impact on insertion into primary key > indexes. Presently, we hold an exclusive buffer lock for the duration > of a couple of operations when checkUnique != UNIQUE_CHECK_NO. > _bt_binsrch() is one such operation. T

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Andres Freund
On 2013-12-04 20:25:53 -0500, Tom Lane wrote: > Tatsuo Ishii writes: > > I would like to add a variant of regclass, which is exactly same as > > current regclass except it does not raise an error when the target > > table is not found. Instead it returns InvalidOid (0). > > I've sometimes thought

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Pavel Stehule
2013/12/5 Andres Freund > On 2013-12-04 20:25:53 -0500, Tom Lane wrote: > > Tatsuo Ishii writes: > > > I would like to add a variant of regclass, which is exactly same as > > > current regclass except it does not raise an error when the target > > > table is not found. Instead it returns Invalid

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Andres Freund
On 2013-12-05 11:54:20 +0100, Pavel Stehule wrote: > 2013/12/5 Andres Freund > We can introduce some assert polymorphic function > > CREATE OR REPLACE FUNCTION notnull(any, message text) RETURNS any, that can > be used for check inside SQL Uh. How is that going to help applications that upgraded

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Simon Riggs
On 5 December 2013 01:55, Peter Eisentraut wrote: > On Thu, 2013-11-14 at 12:11 +0530, Amit Kapila wrote: >>If an application wants to allow these connection parameters to be >> used, it would need to do PQenableStartServer() first. If it doesn't, >> those connection parameters will be rejecte

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Andres Freund
On 2013-12-04 20:55:08 -0500, Peter Eisentraut wrote: > On Thu, 2013-11-14 at 12:11 +0530, Amit Kapila wrote: > >If an application wants to allow these connection parameters to be > > used, it would need to do PQenableStartServer() first. If it doesn't, > > those connection parameters will be r

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Pavel Stehule
2013/12/5 Andres Freund > On 2013-12-05 11:54:20 +0100, Pavel Stehule wrote: > > 2013/12/5 Andres Freund > > We can introduce some assert polymorphic function > > > > CREATE OR REPLACE FUNCTION notnull(any, message text) RETURNS any, that > can > > be used for check inside SQL > > Uh. How is tha

Re: [HACKERS] better atomics - v0.2

2013-12-05 Thread Andres Freund
On 2013-11-19 10:37:35 -0500, Tom Lane wrote: > Andres Freund writes: > > The only animal we have that doesn't support quiet inlines today is > > HP-UX/ac++, and I think - as in patch 1 in the series - we might be able > > to simply suppress the warning there. > > Or just not worry about it, if i

Re: [HACKERS] Get more from indices.

2013-12-05 Thread Etsuro Fujita
Kyotaro HORIGUCHI wrote: > Thank you, but it seems to me too simplified. You made two major functional > changes. Thank you for the comments! > One is, you put the added code for getrelation_info() out of the block for > the condition (info->relam == BTREE_AM_OID) (though amcanorder would be > pr

Re: [HACKERS] Get more from indices.

2013-12-05 Thread Etsuro Fujita
I wrote: > Kyotaro HORIGUCHI wrote: > > Another is, you changed pathkeys expantion to be all-or-nothing decision. > > While this change should simplify the code slightly, it also dismisses > > the oppotunity for partially-extended pathkeys. Could you let me know > > the > reason > > why you did so.

Re: [HACKERS] [bug fix] "pg_ctl stop" times out when it should respond quickly

2013-12-05 Thread MauMau
From: "Tom Lane" If you're going to do a postmaster_is_alive check, why bother with repeated get_pgpid()? As I said yesterday, I removed get_pgpid() calls. I'll add this patch to 2014-1 commitfest this weekend if it is not committed until then. Regards MauMau pg_stop_fail_v2.patch Descri

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Andres Freund
On 2013-12-04 18:48:44 -0500, Robert Haas wrote: > * When a type narrower than Datum is stored in a Datum, we place it in the > * low-order bits and are careful that the DatumGetXXX macro for it discards > * the unused high-order bits (as opposed to, say, assuming they are zero). > * This is ne

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Pavel Golub
Hello, Andres. You wrote: AF> On 2013-12-04 20:25:53 -0500, Tom Lane wrote: >> Tatsuo Ishii writes: >> > I would like to add a variant of regclass, which is exactly same as >> > current regclass except it does not raise an error when the target >> > table is not found. Instead it returns Invalid

Re: [HACKERS] tracking commit timestamps

2013-12-05 Thread Andres Freund
On 2013-12-02 02:39:55 -0500, Jaime Casanova wrote: > === performance === > > i expected a regression on performance with the module turned on > because of the new XLOG records and wrote of files in pg_committs but > the performance drop is excessive. > Master 437.835674 tps > Patch, g

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 4:56 AM, Andres Freund wrote: > Hi Robert, > > On 2013-12-04 23:32:27 -0500, Robert Haas wrote: >> But I'm also learning painfully that this kind of thing only goes so >> far. For example, I spent some time looking at what it would take to >> provide a dynamic shared memory

Re: [HACKERS] [bug fix] pg_ctl fails with config-only directory

2013-12-05 Thread MauMau
From: "Amit Kapila" On Wed, Dec 4, 2013 at 7:57 PM, MauMau wrote: * Approach 1 When postgres starts, it removes Administrator privileges from its own process. But is this possible at all? Windows security API is complex and provides many functions. It seems difficult to understand them. I

Re: [HACKERS] logical changeset generation v6.7

2013-12-05 Thread Kyotaro HORIGUCHI
Hello, > Will send the rebased version as soon as I've addressed your comments. Thank you. > > = 0001: > > > > - You assined HeapTupleGetOid(tuple) into relid to read in > >several points but no modification. Nevertheless, you left > >HeapTupleGetOid not replaced there. I think 're

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Heikki Linnakangas
On 12/05/2013 07:30 AM, Tom Lane wrote: Peter Eisentraut writes: On Wed, 2013-12-04 at 20:27 -0500, Tom Lane wrote: Lazy people? I'm not in a hurry to drop it; it's not costing us much to just sit there, other than in this connection which we see how to fix. Actually, I think it probably c

Re: [HACKERS] shared memory message queues

2013-12-05 Thread Andres Freund
Hi, Planned to look at this for a while... Not a detailed review, just some thoughts. I'll let what I read sink in and possibly comment later. On 2013-10-31 12:21:31 -0400, Robert Haas wrote: > The attached patches attempt to rectify some of these problems. Well, I wouldn't call it problems. Jus

Re: [HACKERS] Changes in Trigger Firing

2013-12-05 Thread Sameer Kumar
I have been finally able to get the right set of files. I going with below approach: 1) Add a new column in pg_trigger called tgiscascaded 2) Change pg_trigger.h for this 3) Made changes in trigger.c to insert this values 4) Corresponding changes made in - reltrigger.h

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-05 Thread Alvaro Herrera
Andres Freund wrote: > On 2013-12-03 19:55:40 -0300, Alvaro Herrera wrote: > > I added a new isolation spec to test this specific case, and noticed > > something that seems curious to me when that test is run in REPEATABLE > > READ mode: when the UPDATE is aborted, the concurrent FOR UPDATE gets a

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Stephen Frost
* Peter Geoghegan (p...@heroku.com) wrote: > On Wed, Dec 4, 2013 at 11:07 AM, Josh Berkus wrote: > > But you know what? 2.6, overall, still performs better than any kernel > > in the 3.X series, at least for Postgres. > > What about the fseek() scalability issue? Not to mention that the 2.6 whi

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-05 Thread Andres Freund
Hi, On 2013-12-05 10:42:35 -0300, Alvaro Herrera wrote: > I intend to apply these two to 9.3 and master, and > then apply your freeze fix on top (which I'm cleaning up a bit -- will > resend later.) I sure hope it get's cleaned up - it's an evening's hack, with a good glass of wine ontop. Do you

Re: [HACKERS] Changes in Trigger Firing

2013-12-05 Thread Sameer Kumar
One scenario where I can forsee an issue is when someone uses the tablename with-in the trigger function on which the trigger was fired. I am not sure how and what issue might crop up but this will be one of my test cases. -- View this message in context: http://postgresql.1045698.n5.nabble.com

[HACKERS] Feature request: Logging SSL connections

2013-12-05 Thread Dr. Andreas Kunert
Hello, we were really missing the information in our log files if (and which of) our users are using SSL during their connections. The attached patch is a very simple solution to this problem - it just tests if the ssl pointer in Port is null. If no, it adds "SSL" to the logfile, otherwise i

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Heikki Linnakangas
On 12/05/2013 06:32 AM, Robert Haas wrote: During development of the dynamic shared memory facility, Noah and I spent a lot of time arguing about whether it was practical to ensure that a dynamic shared memory segment got mapped at the same address in every backend that used it. My vote goes fo

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > On 2013-12-04 18:48:44 -0500, Robert Haas wrote: >> And record_image_eq does a rather elaborate dance around here, calling >> the appropriate GET_x_BYTES macro depending on the type-width. If we >> can really count on the high-order bits to be zero, that's all >> completel

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 15:57:22 +0200, Heikki Linnakangas wrote: > As a side-note, I've been thinking that we don't really need same-address > mapping for shared_buffers either. Getting rid of it wouldn't buy us > anything right now, but if we wanted e.g to make shared_buffers changeable > without a restart

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Andres Freund
On 2013-12-05 08:58:55 -0500, Tom Lane wrote: > Andres Freund writes: > > I don't think we can get rid of that dance in record_image_eq - it very > > well could used on records originally generated when those bits haven't > > been guaranteed to be zeroed. > > No, you're failing to think about the

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Tom Lane
Michael Paquier writes: > It happens that the following regression tests are failing if they are > run on a database not named "regression": This does not seem like a bug to me, although maybe we'd better update the documentation to specify that you need to use a DB named regression.

[HACKERS] [bug fix] pg_ctl always uses the same event source

2013-12-05 Thread MauMau
Hello, I've removed a limitation regarding event log on Windows with the attached patch. I hesitate to admit this is a bug fix and want to regard this an improvement, but maybe it's a bug fix from users' perspective. Actually, I received problem reports from some users. [Problem] pg_ctl a

Re: [HACKERS] Backup throttling

2013-12-05 Thread Antonin Houska
On 12/02/2013 02:23 PM, Boszormenyi Zoltan wrote: > Hi, > > I am reviewing your patch. Thanks. New version attached. > > * Does it follow the project coding guidelines? > > Yes. A nitpicking: this else branch below might need brackets > because there is also a comment in that branch: > > +

Re: [HACKERS] More legacy code: pg_ctl

2013-12-05 Thread Peter Eisentraut
On 11/18/13, 8:09 PM, Josh Berkus wrote: > a) by default, it returns to the caller without waiting for postgres to > actually start/stop/restart. In this mode, it also always returns > success regardless of result. The reason for this is that until sometime recently (PQping) we didn't have a reli

Re: [HACKERS] More legacy code: pg_ctl

2013-12-05 Thread Peter Eisentraut
On 11/18/13, 8:20 PM, Josh Berkus wrote: > c) that "stop" defaults to "smart" mode, instead of "fast" mode. This has been discussed many times already, so you'd need to check the archives. (I'm not in favor of smart mode either.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 07:44:27 -0500, Robert Haas wrote: > On Thu, Dec 5, 2013 at 4:56 AM, Andres Freund wrote: > > Hi Robert, > > > > On 2013-12-04 23:32:27 -0500, Robert Haas wrote: > >> But I'm also learning painfully that this kind of thing only goes so > >> far. For example, I spent some time lookin

Re: [HACKERS] Feature request: Logging SSL connections

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 8:53 AM, Dr. Andreas Kunert wrote: > we were really missing the information in our log files if (and which > of) our users are using SSL during their connections. > > The attached patch is a very simple solution to this problem - it just > tests if the ssl pointer in Port is null. If n

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Greg Stark
On Thu, Dec 5, 2013 at 8:35 AM, KONDO Mitsumasa wrote: > Yes. And using something efficiently DirectIO is more difficult than > BufferedIO. > If we change write() flag with direct IO in PostgreSQL, it will execute > hardest ugly randomIO. Using DirectIO presumes you're using libaio or threads to

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Pavel Golub writes: > I personally see two approaches: > 1. Implement GUC variable controling this behaviour per session > 2. Introduce new safe reg* variables, e.g. "sregclass", "sregtype" etc. I don't think new types are a good idea. If we are afraid to change the behavior of the input convert

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 15:44:34 +0100, Andres Freund wrote: > On 2013-12-05 07:44:27 -0500, Robert Haas wrote: > > And then I thought, boy, it sucks > > not to be able to declare what kind of a thing we're pointing *at* > > here, but apart from using C++ I see no solution to that problem. I > > guess we co

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Claudio Freire
On Thu, Dec 5, 2013 at 11:42 AM, Greg Stark wrote: > (b) is the way more interesting research project though. I don't think > anyone's tried it and the kernel interface to provide the kinds of > information Postgres needs requires a lot of thought. If it's done > right then Postgres wouldn't need

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 9:41 AM, Tom Lane wrote: > Pavel Golub writes: >> I personally see two approaches: >> 1. Implement GUC variable controling this behaviour per session >> 2. Introduce new safe reg* variables, e.g. "sregclass", "sregtype" etc. > > I don't think new types are a good idea. If we are afra

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 8:57 AM, Heikki Linnakangas wrote: > On 12/05/2013 06:32 AM, Robert Haas wrote: >> During development of the dynamic shared memory facility, Noah and I >> spent a lot of time arguing about whether it was practical to ensure >> that a dynamic shared memory segment got mapped

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > On 2013-12-05 08:58:55 -0500, Tom Lane wrote: >> I'm a bit worried that somebody, particularly third-party code, >> might've sloppily written "return foo" in a V1 function when "return >> Int32GetDatum(foo)" would be correct. In that case, the resultant Datum >> might have

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Peter Eisentraut writes: > We could invent some sneaky syntax variants, like 'pg_klass'::regclass > errors, but '?pg_klass'::regclass does not. Hmm ... cute idea, but shoehorning it into regoperator might be problematic. You'd have to pick a flag character that wasn't a valid operator character,

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Andres Freund
On 2013-12-05 10:02:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2013-12-05 08:58:55 -0500, Tom Lane wrote: > >> I'm a bit worried that somebody, particularly third-party code, > >> might've sloppily written "return foo" in a V1 function when "return > >> Int32GetDatum(foo)" would be c

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 9:44 AM, Andres Freund wrote: >> Why? Lots of people have written lots of programs that do just that. > > Well, but we're a database, not a generic programming library ;) I think we're arguably both. > But what's your alternative if you have a shared_palloc() like thingy?

[HACKERS] [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread MauMau
Hello, My customers and colleagues sometimes (or often?) ask about the following message: FATAL: the database system is starting up This message is often output dozens of times during a failover or PITR. The users seem to be worried because the message level is FATAL and they don't know w

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 9:41 AM, Tom Lane wrote: > Pavel Golub writes: >> I personally see two approaches: >> 1. Implement GUC variable controling this behaviour per session >> 2. Introduce new safe reg* variables, e.g. "sregclass", "sregtype" etc. > > I don't think new types are a good idea. If

Re: [HACKERS] shared memory message queues

2013-12-05 Thread Kohei KaiGai
Sorry for my late. I checked the part-3 (shm-mq-v1.patc) portion as below. Your comments towards part-1 and part-2 are reasonable for me, so I have no argue on this portion. Even though shm_mq_create() expects the "address" is aligned, however, no mechanism to ensure. How about to put Assert() he

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > I was actually thinking about making Datum (and some other types we > have) structs or unions. Currently it's far, far to easy to mix them. We throw > away pretty much all of the little typesafety C has by typedef'ing them > to integral types with lots of autocasting behavi

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Robert Haas writes: > On Thu, Dec 5, 2013 at 9:41 AM, Tom Lane wrote: >> I don't think new types are a good idea. If we are afraid to change >> the behavior of the input converters, what we should do is introduce >> new functions, eg "toregclass(text) returns regclass". > That seems like a pret

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Greg Stark
On Thu, Dec 5, 2013 at 2:54 PM, Claudio Freire wrote: > That's a bad idea in the current state of affairs. MM files haven't > been designed for that usage, and getting stable performance out of > that will be way too difficult. I'm talking about long-term goals here. Either of these two routes wo

Re: [HACKERS] [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 10:25 AM, MauMau wrote: > Report these as FATAL to the client because the client wants to know the > reason. But don't output them to server log because they are not > necessary for DBAs Yeah, this is part of a more general problem, which you have characterized correctly: What is fata

Re: [HACKERS] [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Tom Lane
"MauMau" writes: > Shouldn't we lower the severity or avoiding those messages to server log? No. They are FATAL so far as the individual session is concerned. Possibly some documentation effort is needed here, but I don't think any change in the code behavior would be an improvement. > 1. FATA

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> You could try my lwlock-scalability improvement patches - for some > workloads here, the improvements have been rather noticeable. Which > version are you testing? I tried your patches on next link. As you suspect I didn't see any improvements. I tested it on PostgreSQL 9.2 Stable. http://git.p

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 10:17:18 -0500, Robert Haas wrote: > On Thu, Dec 5, 2013 at 9:44 AM, Andres Freund wrote: > >> Why? Lots of people have written lots of programs that do just that. > > > > Well, but we're a database, not a generic programming library ;) > > I think we're arguably both. Fair enough

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Andres Freund
On 2013-12-05 17:46:44 +0200, Metin Doslu wrote: > I tried your patches on next link. As you suspect I didn't see any > improvements. I tested it on PostgreSQL 9.2 Stable. You tested the correct branch, right? Which commit does "git rev-parse HEAD" show? But generally, as long as your profile hid

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 6:07 AM, Simon Riggs wrote: > On 5 December 2013 01:55, Peter Eisentraut wrote: >> On Thu, 2013-11-14 at 12:11 +0530, Amit Kapila wrote: >>>If an application wants to allow these connection parameters to be >>> used, it would need to do PQenableStartServer() first. If it doesn't, >

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Robert Haas writes: >> Another advantage of this approach is that, IIUC, type input functions >> can't return a NULL value. So 'pg_klass'::regclass could return 0, >> but not NULL. On the other hand, toregclass('pg_klass') *could* >> return NULL, which seems conceptually cleaner. BTW, another a

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> You tested the correct branch, right? Which commit does "git rev-parse > HEAD" show? I applied last two patches manually on PostgreSQL 9.2 Stable.

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Tom Lane
Greg Stark writes: > I think the way to use mmap would be to mmap very large chunks, > possibly whole tables. We would need some way to control page flushes > that doesn't involve splitting mappings and can be efficiently > controlled without having the kernel storing arbitrarily large tags on > p

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> From what I've seen so far the bigger problem than contention in the > lwlocks itself, is the spinlock protecting the lwlocks... Postgres 9.3.1 also reports spindelay, it seems that there is no contention on spinlocks. PID 21121 lwlock 0: shacq 0 exacq 33 blk 1 spindelay 0 PID 21121 lwlock 33:

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Andres Freund
On 2013-12-05 10:34:16 -0500, Tom Lane wrote: > Andres Freund writes: > > I was actually thinking about making Datum (and some other types we > > have) structs or unions. Currently it's far, far to easy to mix them. We > > throw > > away pretty much all of the little typesafety C has by typedef'i

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > On 2013-12-05 10:34:16 -0500, Tom Lane wrote: >> In any case, the number of bugs I can remember that such a thing >> would've prevented is negligible. > Cases talked about upthread, where a plain datatype is returned as a > Datum instead of using FooGetDatum() and the reve

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas wrote: How many allocations? What size will they have have typically, minimum and maximum? The facility is intended to be general, so the answer could vary widely by application. The testing that I

Re: [HACKERS] Feature request: Logging SSL connections

2013-12-05 Thread Marko Kreen
On Thu, Dec 05, 2013 at 09:43:31AM -0500, Peter Eisentraut wrote: > On 12/5/13, 8:53 AM, Dr. Andreas Kunert wrote: > > we were really missing the information in our log files if (and which > > of) our users are using SSL during their connections. > > > > The attached patch is a very simple solutio

[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread David Johnston
Tom Lane-2 wrote > "MauMau" < > maumau307@ > > writes: >> Shouldn't we lower the severity or avoiding those messages to server log? > > No. They are FATAL so far as the individual session is concerned. > Possibly some documentation effort is needed here, but I don't think > any change in the c

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Peter Eisentraut
I think this proposal is a bit deadlocked now. - There are technical concerns about launching a server executable from within a client. - There are conceptual concerns about promoting an embedded database mode. On the other hand: - Everyone would like to have a way to use psql (and other basic

Re: [HACKERS] logical changeset generation v6.7

2013-12-05 Thread Andres Freund
Hi, On 2013-12-05 22:03:51 +0900, Kyotaro HORIGUCHI wrote: > > > - You assined HeapTupleGetOid(tuple) into relid to read in > > >several points but no modification. Nevertheless, you left > > >HeapTupleGetOid not replaced there. I think 'relid' just for > > >repeated reading has far s

Re: [HACKERS] Extension Templates S03E11

2013-12-05 Thread Peter Eisentraut
On 12/1/13, 10:47 PM, Stephen Frost wrote: > Having a management system for sets of objects is a *great* idea- and > one which we already have through schemas. What we don't have is any > kind of versioning system built-in or other metadata about it, nor do we > have good tooling which leverages s

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Andres Freund
On 2013-12-05 11:39:29 -0500, Peter Eisentraut wrote: > I think this proposal is a bit deadlocked now. > > - There are technical concerns about launching a server executable from > within a client. > > - There are conceptual concerns about promoting an embedded database mode. > > On the other ha

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 10:08 AM, Tom Lane wrote: > Peter Eisentraut writes: >> We could invent some sneaky syntax variants, like 'pg_klass'::regclass >> errors, but '?pg_klass'::regclass does not. > > Hmm ... cute idea, but shoehorning it into regoperator might be > problematic. You'd have to pick a flag c

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-05 Thread J Smith
On Thu, Nov 28, 2013 at 5:15 AM, Andres Freund wrote: > Hi, > > Do you still have the core file around? If so could you 'p > *ShmemVariableCache' and 'p *ControlFile'? > So sorry, I didn't see this message until just today. Seems it was accidentally archived before hitting my eyeballs. I see tha

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Fabrízio de Royes Mello
On Thu, Dec 5, 2013 at 7:57 AM, Simon Riggs wrote: > On 5 December 2013 08:51, Magnus Hagander wrote: > > > Not recalling the older thread, but it seems the "breaks on clock > drift", I > > think we can fairly easily make that situation "good enough". Just have > > IDENTIFY_SYSTEM return the cur

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread David E. Wheeler
On Dec 5, 2013, at 7:52 AM, Tom Lane wrote: > BTW, another arguable advantage of fixing this via new functions is > that users could write equivalent (though no doubt slower) functions > for use in pre-9.4 releases, and thus not need to maintain multiple > versions of app code that relies on this

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Josh Berkus
On 12/05/2013 07:40 AM, Greg Stark wrote: > On Thu, Dec 5, 2013 at 2:54 PM, Claudio Freire wrote: >> That's a bad idea in the current state of affairs. MM files haven't >> been designed for that usage, and getting stable performance out of >> that will be way too difficult. > > I'm talking about

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Josh Berkus
On 12/05/2013 05:48 AM, Stephen Frost wrote: > * Peter Geoghegan (p...@heroku.com) wrote: >> On Wed, Dec 4, 2013 at 11:07 AM, Josh Berkus wrote: >>> But you know what? 2.6, overall, still performs better than any kernel >>> in the 3.X series, at least for Postgres. >> >> What about the fseek() sc

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Claudio Freire
On Thu, Dec 5, 2013 at 1:03 PM, Metin Doslu wrote: >> From what I've seen so far the bigger problem than contention in the >> lwlocks itself, is the spinlock protecting the lwlocks... > > Postgres 9.3.1 also reports spindelay, it seems that there is no contention > on spinlocks. Did you check hu

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Stephen Frost
* David Johnston (pol...@yahoo.com) wrote: > ISTM that instituting some level of categorization for messages would be > helpful. Then logging and reporting frameworks would be able to identify > and segregate the logs in whatever way they and the configuration deems > appropriate. I've wanted to

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Josh Berkus
On 12/05/2013 10:21 AM, Stephen Frost wrote: > * David Johnston (pol...@yahoo.com) wrote: >> ISTM that instituting some level of categorization for messages would be >> helpful. Then logging and reporting frameworks would be able to identify >> and segregate the logs in whatever way they and the c

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 1:09 AM, Sergey Muraviov wrote: > And my patch affects the row view only. To help us avoid forgetting about this patch, please add it here: https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterpri

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 9:24 AM, Tom Lane wrote: > Michael Paquier writes: >> It happens that the following regression tests are failing if they are >> run on a database not named "regression": > > This does not seem like a bug to me, although maybe we'd better update the > documentation to specif

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Tom Lane
Josh Berkus writes: > On 12/05/2013 10:21 AM, Stephen Frost wrote: > But ... if we set a firm policy on this, then we could gradually clean > up the error messages piecemeal over the next couple of major versions. > We could also make sure that any new features complied with the > categorization

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Tom Lane
Robert Haas writes: > On Thu, Dec 5, 2013 at 9:24 AM, Tom Lane wrote: >> Michael Paquier writes: >>> It happens that the following regression tests are failing if they are >>> run on a database not named "regression": >> This does not seem like a bug to me, although maybe we'd better update the

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Josh Berkus
On 12/05/2013 10:46 AM, Tom Lane wrote: > Before we could get very far we'd need a better understanding than we have > of what cases a DBA might be interested in. To take the specific example > that started this thread, there wouldn't be a lot of value IMO in a > classification like "connection fa

  1   2   >