Re: [HACKERS] new --maintenance-db options

2012-06-26 Thread Amit Kapila
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Tom Lane Robert Haas robertmh...@gmail.com writes: From pg_upgrade's perspective, it would be nice to have a flag that starts the server in some mode where nobody but pg_upgrade can connect to it

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Magnus Hagander
On Tue, Jun 26, 2012 at 7:06 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I worked on simple patch, that enable access from server side to client side data. It add two new hooks to libpq - one for returning of local context, second for setting of local context. A motivation is

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Pavel Stehule
2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 7:06 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I worked on simple patch, that enable access from server side to client side data. It add two new hooks to libpq - one for returning of local context, second

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Boszormenyi Zoltan
2012-06-26 06:59 keltezéssel, Alvaro Herrera írta: I cleaned up the framework patch a bit. My version's attached. Mainly, returning false for failure in some code paths that are only going to have the caller elog(FATAL) is rather pointless -- it seems much better to just have the code itself

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Magnus Hagander
On Tue, Jun 26, 2012 at 9:50 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 7:06 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I worked on simple patch, that enable access from server side to client side

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Pavel Stehule
2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 9:50 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 7:06 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I worked on simple patch, that

Re: [HACKERS] REVIEW: Optimize referential integrity checks (todo item)

2012-06-26 Thread Vik Reykja
On Wed, Jun 20, 2012 at 2:19 AM, Tom Lane t...@sss.pgh.pa.us wrote: I've marked this patch committed, although in the end there was nothing left of it ;-) Thank you, Dean and Tom! I'm sorry for not participating in this thread, I've been away for the past five weeks and have much catching up

Re: [HACKERS] WIP Patch: Selective binary conversion of CSV file foreign tables

2012-06-26 Thread Etsuro Fujita
Hi Kaigai-san, -Original Message- From: Kohei KaiGai [mailto:kai...@kaigai.gr.jp] Sent: Monday, June 25, 2012 9:49 PM To: Etsuro Fujita Cc: Robert Haas; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] WIP Patch: Selective binary conversion of CSV file foreign tables

Re: [HACKERS] Backport of fsync queue compaction

2012-06-26 Thread Robert Haas
On Tue, Jun 19, 2012 at 5:39 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 19, 2012 at 5:33 PM, Greg Smith g...@2ndquadrant.com wrote: In January of 2011 Robert committed 7f242d880b5b5d9642675517466d31373961cf98 to try and compact the fsync queue when clients find it full.  There's

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 3:59 AM, Boszormenyi Zoltan z...@cybertec.at wrote: Well, I can make the registration interface similar to how LWLocks are treated, but that doesn't avoid modification of the base_timeouts array in case a new internal use case arises. Say: #define USER_TIMEOUTS    4

Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal

2012-06-26 Thread Andres Freund
On Monday, June 25, 2012 08:50:54 PM Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: We most particularly *don't* want DDL to replicate automatically, because the schema changes are deployed along with related software changes, and we like to pilot any changes for at least

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Boszormenyi Zoltan
2012-06-26 13:50 keltezéssel, Robert Haas írta: On Tue, Jun 26, 2012 at 3:59 AM, Boszormenyi Zoltan z...@cybertec.at wrote: Well, I can make the registration interface similar to how LWLocks are treated, but that doesn't avoid modification of the base_timeouts array in case a new internal use

Re: [HACKERS] pg_stat_lwlocks view - lwlocks statistics

2012-06-26 Thread Satoshi Nagayasu
Hi all, I've modified the pg_stat_lwlocks patch to be able to work with the latest PostgreSQL Git code. This patch provides: pg_stat_lwlocks New system view to show lwlock statistics. pg_stat_get_lwlocks() New function to retrieve lwlock statistics. pg_stat_reset_lwlocks()

Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal

2012-06-26 Thread Simon Riggs
On 25 June 2012 17:42, Kevin Grittner kevin.gritt...@wicourts.gov wrote: This is clearly going to depend on the topology.  You would definitely want to try to replicate the DDL for the case on which Simon is focused (which seems to me to be essentially physical replication of catalogs with

Re: [HACKERS] patch: avoid heavyweight locking on hash metapage

2012-06-26 Thread Robert Haas
On Mon, Jun 25, 2012 at 11:05 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Mon, Jun 18, 2012 at 5:42 PM, Robert Haas robertmh...@gmail.com wrote: Hmm.  That was actually a gloss I added on existing code to try to convince myself that it was safe; I don't think that the changes I made make

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 8:03 AM, Boszormenyi Zoltan z...@cybertec.at wrote: I know, but it doesn't feel right to register static functionality. We do it elsewhere. The overhead is pretty minimal compared to other things we already do during startup, and avoiding the need for the array to have a

Re: [HACKERS] new --maintenance-db options

2012-06-26 Thread Tom Lane
Amit Kapila amit.kap...@huawei.com writes: [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Tom Lane The implementation I've wanted to see for some time is that you can start a standalone backend, but it speaks FE/BE protocol to its caller (preferably over pipes, so that there is no

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jun 26, 2012 at 8:03 AM, Boszormenyi Zoltan z...@cybertec.at wrote: I know, but it doesn't feel right to register static functionality. We do it elsewhere. The overhead is pretty minimal compared to other things we already do during startup,

Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal

2012-06-26 Thread Robert Haas
On Mon, Jun 25, 2012 at 3:17 PM, Andres Freund and...@2ndquadrant.com wrote: I suppose the main reason we haven't done it already is that it increases the period of time during which we're using 2X the disk space. I find that an acceptable price if its optional. Making it such doesn't seem to

Re: [HACKERS] pgsql_fdw in contrib

2012-06-26 Thread Kohei KaiGai
Harada-san, I checked your patch, and had an impression that includes many improvements from the previous revision that I looked at the last commit fest. However, I noticed several points to be revised, or investigated. * It seems to me expected results of the regression test is not attached,

Re: [HACKERS] [PATCH 01/16] Overhaul walsender wakeup handling

2012-06-26 Thread Robert Haas
On Fri, Jun 22, 2012 at 12:35 PM, Andres Freund and...@2ndquadrant.com wrote: Can you elaborate on that a bit?  What scenarios did you play around with, and what does win mean in this context? I had two machines connected locally and setup HS and my prototype between them (not at once

Re: [HACKERS] WIP Patch: Selective binary conversion of CSV file foreign tables

2012-06-26 Thread Kohei KaiGai
2012/6/26 Etsuro Fujita fujita.ets...@lab.ntt.co.jp: Hi Kaigai-san, -Original Message- From: Kohei KaiGai [mailto:kai...@kaigai.gr.jp] Sent: Monday, June 25, 2012 9:49 PM To: Etsuro Fujita Cc: Robert Haas; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] WIP Patch: Selective

Re: [HACKERS] [PATCH 01/16] Overhaul walsender wakeup handling

2012-06-26 Thread Andres Freund
On Tuesday, June 26, 2012 04:01:26 PM Robert Haas wrote: On Fri, Jun 22, 2012 at 12:35 PM, Andres Freund and...@2ndquadrant.com wrote: Can you elaborate on that a bit? What scenarios did you play around with, and what does win mean in this context? I had two machines connected locally

Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result

2012-06-26 Thread Fujii Masao
On Wed, Mar 28, 2012 at 10:08 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: Anyway, should I add this patch into the next CF? Or is anyone

Re: [HACKERS] libpq compression

2012-06-26 Thread Robert Haas
On Mon, Jun 25, 2012 at 4:25 PM, k...@rice.edu k...@rice.edu wrote: On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're

Re: [HACKERS] empty backup_label

2012-06-26 Thread Robert Haas
On Sun, Jun 24, 2012 at 5:33 PM, David Kerr d...@mr-paradox.net wrote: Howdy, We're using NetApp's flexclone's whenever we need to move our DB between machines. One specific case where we do that is when we're creating a new streaming replication target. The basic steps we're using are:

Re: [HACKERS] GiST subsplit question

2012-06-26 Thread Robert Haas
On Wed, May 30, 2012 at 4:35 PM, Alexander Korotkov aekorot...@gmail.com wrote: So, do we demote that message to a DEBUG1? Or do we make it more clear what the authors of a specific picksplit are supposed to do to avoid that problem? Or am I misunderstanding something? +1 for demote message

Re: [HACKERS] How to avoid base backup in automated failover

2012-06-26 Thread Robert Haas
On Mon, Jun 4, 2012 at 9:14 AM, chinnaobi chinna...@gmail.com wrote: Recently I was writing an application to implement automated failover with env: Two 2008 R2 servers, Network area storage, asynchronous replication, WAL archive on primary enabled. Is there any way to avoid starting standby

Re: [HACKERS] libpq compression

2012-06-26 Thread Euler Taveira
On 26-06-2012 12:23, Robert Haas wrote: At the risk of making everyone laugh at me, has anyone tested pglz? I observe that if the compression ration and performance are good, we might consider using it for this purpose, too, which would avoid having to add dependencies. Conversely, if they

Re: [HACKERS] empty backup_label

2012-06-26 Thread Magnus Hagander
On Tue, Jun 26, 2012 at 5:24 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 24, 2012 at 5:33 PM, David Kerr d...@mr-paradox.net wrote: Howdy, We're using NetApp's flexclone's whenever we need to move our DB between machines. One specific case where we do that is when we're

Re: [HACKERS] libpq compression

2012-06-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 25, 2012 at 4:25 PM, k...@rice.edu k...@rice.edu wrote: Here is the benchmark list from the Google lz4 page: NameRatio C.speed D.speed LZ4 (r59) 2.084 330 915 LZO 2.05 1x_1 2.038 311 480 QuickLZ 1.5

Re: [HACKERS] Schema version management

2012-06-26 Thread Robert Haas
On Tue, May 22, 2012 at 11:31 PM, Joel Jacobson j...@trustly.com wrote: This is true, which means some users won't be able to use the feature, because they are using an ancient OS or have function names with slashes, hm, is it even possible to have function names with slashes? Sure. If you

Re: [HACKERS] empty backup_label

2012-06-26 Thread David Kerr
On Tue, Jun 26, 2012 at 05:33:42PM +0200, Magnus Hagander wrote: - On Tue, Jun 26, 2012 at 5:24 PM, Robert Haas robertmh...@gmail.com wrote: - On Sun, Jun 24, 2012 at 5:33 PM, David Kerr d...@mr-paradox.net wrote: - Howdy, - - We're using NetApp's flexclone's whenever we need to move our DB

Re: [HACKERS] empty backup_label

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 11:33 AM, Magnus Hagander mag...@hagander.net wrote: On Tue, Jun 26, 2012 at 5:24 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 24, 2012 at 5:33 PM, David Kerr d...@mr-paradox.net wrote: Howdy, We're using NetApp's flexclone's whenever we need to move our DB

Re: [HACKERS] hot standby PSQL 9.1 Windows 2008 Servers

2012-06-26 Thread Robert Haas
On Tue, May 22, 2012 at 12:15 PM, chinnaobi chinna...@gmail.com wrote: You mean when the primary which is going to switch its role to standby might not have sent all the WAL records to the standby and If it is switched to standby it has more WAL records than the standby which is now serves as

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Alvaro Herrera
Excerpts from Boszormenyi Zoltan's message of mar jun 26 03:59:06 -0400 2012: 2012-06-26 06:59 keltezéssel, Alvaro Herrera írta: I cleaned up the framework patch a bit. My version's attached. Mainly, returning false for failure in some code paths that are only going to have the caller

Re: [HACKERS] [v9.3] Row-Level Security

2012-06-26 Thread Robert Haas
On Thu, Jun 14, 2012 at 11:43 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: In the previous discussion, we planned to add a syntax option to clarify the command type to fire the RLS policy, such as FOR UPDATE. But current my opinion is, it is not necessary. For example, we can reference the

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Boszormenyi Zoltan
2012-06-26 18:12 keltezéssel, Alvaro Herrera írta: Excerpts from Boszormenyi Zoltan's message of mar jun 26 03:59:06 -0400 2012: 2012-06-26 06:59 keltezéssel, Alvaro Herrera írta: I cleaned up the framework patch a bit. My version's attached. Mainly, returning false for failure in some code

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-26 Thread Alvaro Herrera
Excerpts from Boszormenyi Zoltan's message of mar jun 26 12:43:34 -0400 2012: So, should I keep the enum TimeoutName? Are global variables for keeping dynamically assigned values preferred over the enum? Currently we have 5 timeout sources in total, 3 of them are used by regular backends,

[HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Nils Goroll
Hi, I am currently trying to understand what looks like really bad scalability of 9.1.3 on a 64core 512GB RAM system: the system runs OK when at 30% usr, but only marginal amounts of additional load seem to push it to 70% and the application becomes highly unresponsive. My current understanding

Re: [HACKERS] PATCH: Improve DROP FUNCTION hint

2012-06-26 Thread Robert Haas
On Mon, Jun 11, 2012 at 11:12 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jun 9, 2012 at 11:42 AM, Dean Rasheed dean.a.rash...@gmail.com wrote: Hi, Attached is a small patch to improve the HINT message produced by CREATE OR REPLACE FUNCTION when the new function definition

Re: [HACKERS] pg_stat_lwlocks view - lwlocks statistics

2012-06-26 Thread Josh Berkus
To implement it, a new array can be added in the local process memory to hold lwlock statistics, and update counters both in the shared memory and the local process memory at once. Then, the session can retrieve 'per-session' statistics from the local process memory via some dedicated

Re: [HACKERS] GiST subsplit question

2012-06-26 Thread Jeff Davis
On Tue, 2012-06-26 at 11:28 -0400, Robert Haas wrote: On Wed, May 30, 2012 at 4:35 PM, Alexander Korotkov aekorot...@gmail.com wrote: So, do we demote that message to a DEBUG1? Or do we make it more clear what the authors of a specific picksplit are supposed to do to avoid that problem? Or

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread David Fetter
On Tue, Jun 26, 2012 at 10:12:52AM +0200, Pavel Stehule wrote: 2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 9:50 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 7:06 AM, Pavel Stehule

Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal

2012-06-26 Thread David Fetter
On Mon, Jun 25, 2012 at 01:50:54PM -0500, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: I am not sure were going to get all that into 9.3. Sure, that was more related to why I was questioning how much these use cases even *could* integrate -- whether it even paid to

Re: [HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Merlin Moncure
On Tue, Jun 26, 2012 at 12:02 PM, Nils Goroll sl...@schokola.de wrote: Hi, I am currently trying to understand what looks like really bad scalability of 9.1.3 on a 64core 512GB RAM system: the system runs OK when at 30% usr, but only marginal amounts of additional load seem to push it to

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Pavel Stehule
2012/6/26 David Fetter da...@fetter.org: On Tue, Jun 26, 2012 at 10:12:52AM +0200, Pavel Stehule wrote: 2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun 26, 2012 at 9:50 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2012/6/26 Magnus Hagander mag...@hagander.net: On Tue, Jun

Re: [HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Tom Lane
Nils Goroll sl...@schokola.de writes: Now that the scene is set, here's the simple question: Why all this? Why not simply use posix mutexes which, on modern platforms, will map to efficient implementations like adaptive mutexes or futexes? (1) They do not exist everywhere. (2) There is

Re: [HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Nils Goroll
Hi Merlin, _POSIX_THREAD_PROCESS_SHARED sure. Also, it's forbidden to do things like invoke i/o in the backend while holding only a spinlock. As to your larger point, it's an interesting assertion -- some data to back it up would help. Let's see if I can get any. ATM I've only got

Re: [HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Nils Goroll
But if you start with let's not support any platforms that don't have this feature This will never be my intention. Nils -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Merlin Moncure
On Tue, Jun 26, 2012 at 3:05 AM, Magnus Hagander mag...@hagander.net wrote: But with a small change to psql they could, without the need for a whole new type of variable. For example, psql could set all those variable as psql.commandlinevarname, which could then be accessed from the DO PL code

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Pavel Stehule
2012/6/26 Merlin Moncure mmonc...@gmail.com: On Tue, Jun 26, 2012 at 3:05 AM, Magnus Hagander mag...@hagander.net wrote: But with a small change to psql they could, without the need for a whole new type of variable. For example, psql could set all those variable as psql.commandlinevarname,

[HACKERS] Posix Shared Mem patch

2012-06-26 Thread Josh Berkus
Robert, all: Last I checked, we had a reasonably acceptable patch to use mostly Posix Shared mem with a very small sysv ram partition. Is there anything keeping this from going into 9.3? It would eliminate a major configuration headache for our users. -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] measuring spinning

2012-06-26 Thread Robert Haas
On Mon, Jun 18, 2012 at 9:36 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jun 16, 2012 at 6:25 PM, Jeff Janes jeff.ja...@gmail.com wrote: Well, this fell through the cracks, because I forgot to add it to the January CommitFest.  Here it is again, rebased. This applies and builds

Re: [HACKERS] [PATCH] Lazy hashaggregate when no aggregation is needed

2012-06-26 Thread Robert Haas
On Fri, Jun 22, 2012 at 10:12 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 19, 2012 at 5:41 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I'm confused by this remark, because surely the query planner does it this way only if there's no LIMIT.  When there is a LIMIT, we

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Alvaro Herrera
Excerpts from Josh Berkus's message of mar jun 26 15:49:59 -0400 2012: Robert, all: Last I checked, we had a reasonably acceptable patch to use mostly Posix Shared mem with a very small sysv ram partition. Is there anything keeping this from going into 9.3? It would eliminate a major

Re: [HACKERS] [v9.3] Row-Level Security

2012-06-26 Thread Kohei KaiGai
2012/6/26 Robert Haas robertmh...@gmail.com: Of course, here is some limitations, to keep the patch size reasonable level to review. - The permission to bypass RLS policy was under discussion.  If and when we should provide a special permission to bypass RLS  policy, the OR

Re: [HACKERS] pg_terminate_backend for same-role

2012-06-26 Thread Robert Haas
On Thu, Mar 29, 2012 at 3:04 AM, Daniel Farina dan...@heroku.com wrote: On Mon, Mar 26, 2012 at 12:14 AM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2012-03-20 at 01:38 -0700, Daniel Farina wrote: On Thu, Mar 15, 2012 at 9:39 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 16, 2012

Re: [HACKERS] [v9.3] Row-Level Security

2012-06-26 Thread Tom Lane
Kohei KaiGai kai...@kaigai.gr.jp writes: 2012/6/26 Robert Haas robertmh...@gmail.com: I think you're missing the point.  Everyone who has commented on this issue is in favor of having some check that causes the RLS predicate *not to get added in the first place*. Here is a simple idea to

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-06-26 Thread Robert Haas
On Wed, May 2, 2012 at 9:01 PM, Josh Berkus j...@agliodbs.com wrote: On 5/2/12 10:20 AM, Jameison Martin wrote: Attached are the following as per various requests:       * test_results.txt: the performance benchmarking results,       * TestTrailingNull.java: the performance benchmarking code,

Re: [HACKERS] tuplesort memory usage: grow_memtuples

2012-06-26 Thread Robert Haas
On Sat, Mar 3, 2012 at 3:22 PM, Jeff Janes jeff.ja...@gmail.com wrote: When sorting small tuples, the memtuples array can use a substantial fraction of the total per-tuple memory used. (In the case of pass by value, it is all of it) The way it grows leads to sub-optimal memory usage. Greg, I

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 4:29 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Josh Berkus's message of mar jun 26 15:49:59 -0400 2012: Robert, all: Last I checked, we had a reasonably acceptable patch to use mostly Posix Shared mem with a very small sysv ram partition.  Is

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes: it is not security issue - just I dislike sending complete stack, when just only one variable should be used. That's a pretty darn weak argument. If I read the patch correctly, what you're proposing involves a dynamic fetch from the client at

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Josh Berkus
On 6/26/12 2:13 PM, Robert Haas wrote: On Tue, Jun 26, 2012 at 4:29 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Josh Berkus's message of mar jun 26 15:49:59 -0400 2012: Robert, all: Last I checked, we had a reasonably acceptable patch to use mostly Posix Shared mem

Re: [HACKERS] proof concept - access to session variables on client side

2012-06-26 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: On Tue, Jun 26, 2012 at 3:05 AM, Magnus Hagander mag...@hagander.net wrote: But with a small change to psql they could, without the need for a whole new type of variable. For example, psql could set all those variable as psql.commandlinevarname, which

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Daniel Farina
On Tue, Jun 26, 2012 at 2:18 PM, Josh Berkus j...@agliodbs.com wrote: On 6/26/12 2:13 PM, Robert Haas wrote: On Tue, Jun 26, 2012 at 4:29 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Josh Berkus's message of mar jun 26 15:49:59 -0400 2012: Robert, all: Last I checked,

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 5:18 PM, Josh Berkus j...@agliodbs.com wrote: On 6/26/12 2:13 PM, Robert Haas wrote: On Tue, Jun 26, 2012 at 4:29 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Josh Berkus's message of mar jun 26 15:49:59 -0400 2012: Robert, all: Last I checked,

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Josh Berkus
On that, I used to be of the opinion that this is a good compromise (a small amount of interlock space, plus mostly posix shmem), but I've heard since then (I think via AgentM indirectly, but I'm not sure) that there are cases where even the small SysV segment can cause problems -- notably

Re: [HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Martijn van Oosterhout
On Tue, Jun 26, 2012 at 01:46:06PM -0500, Merlin Moncure wrote: Well, that would introduce a backend dependency on pthreads, which is unpleasant. Also you'd need to feature test via _POSIX_THREAD_PROCESS_SHARED to make sure you can mutex between processes (and configure your mutexes as such

Re: [HACKERS] pg_terminate_backend for same-role

2012-06-26 Thread Daniel Farina
On Tue, Jun 26, 2012 at 1:58 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Mar 29, 2012 at 3:04 AM, Daniel Farina dan...@heroku.com wrote: On Mon, Mar 26, 2012 at 12:14 AM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2012-03-20 at 01:38 -0700, Daniel Farina wrote: On Thu, Mar 15, 2012

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 5:44 PM, Josh Berkus j...@agliodbs.com wrote: On that, I used to be of the opinion that this is a good compromise (a small amount of interlock space, plus mostly posix shmem), but I've heard since then (I think via AgentM indirectly, but I'm not sure) that there are

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Alvaro Herrera
Excerpts from Daniel Farina's message of mar jun 26 17:40:16 -0400 2012: On that, I used to be of the opinion that this is a good compromise (a small amount of interlock space, plus mostly posix shmem), but I've heard since then (I think via AgentM indirectly, but I'm not sure) that there

Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal

2012-06-26 Thread Kevin Grittner
David Fetter da...@fetter.org wrote: On Mon, Jun 25, 2012 at 01:50:54PM -0500, Kevin Grittner wrote: One fine point regarding before and after images -- if a value doesn't change in an UPDATE, there's no reason to include it in both the BEFORE and AFTER tuple images, as long as we have the

Re: [HACKERS] why roll-your-own s_lock? / improving scalability

2012-06-26 Thread Tom Lane
Martijn van Oosterhout klep...@svana.org writes: And then you have fabulous things like: https://git.reviewboard.kde.org/r/102145/ (OSX defines _POSIX_THREAD_PROCESS_SHARED but does not actually support it.) Seems not very well tested in any case. It might be worthwhile testing futexes on

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Daniel Farina
On Tue, Jun 26, 2012 at 2:53 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Daniel Farina's message of mar jun 26 17:40:16 -0400 2012: On that, I used to be of the opinion that this is a good compromise (a small amount of interlock space, plus mostly posix shmem), but I've

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread A.M.
On Jun 26, 2012, at 5:44 PM, Josh Berkus wrote: On that, I used to be of the opinion that this is a good compromise (a small amount of interlock space, plus mostly posix shmem), but I've heard since then (I think via AgentM indirectly, but I'm not sure) that there are cases where even the

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Josh Berkus
This can be trivially reproduced if one runs an old (SysV shared memory-based) postgresql alongside a potentially newer postgresql with a smaller SysV segment. This can occur with applications that bundle postgresql as part of the app. I'm not saying it doesn't happen at all. I'm saying

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So, what about keeping a FIFO in the data directory? Hm, does that work if the data directory is on NFS? Or some other weird not-really-Unix file system? When the postmaster starts up, it tries to open the file with O_NONBLOCK | O_WRONLY (or

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread A.M.
On Jun 26, 2012, at 6:12 PM, Daniel Farina wrote: (Emphasis mine). I don't think that -hackers at the time gave the zero-shmem rationale much weight (I also was not that happy about the safety mechanism of that patch), but upon more reflection (and taking into account *other* software

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: So let's fix the 80% case with something we feel confident in, and then revisit the no-sysv interlock as a separate patch. That way if we can't fix the interlock issues, we still have a reduced-shmem version of Postgres. Yes. Insisting that we have the

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: In the meantime, insisting that we solve this problem before we do anything is a good recipe for ensuring that nothing happens, just like it hasn't happened for the last half dozen years. (I see Alvaro just made the same point.) And now so has Josh. +1

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
A.M. age...@themactionfaction.com writes: This can be trivially reproduced if one runs an old (SysV shared memory-based) postgresql alongside a potentially newer postgresql with a smaller SysV segment. This can occur with applications that bundle postgresql as part of the app. I don't

[HACKERS] experimental: replace s_lock spinlock code with pthread_mutex on linux

2012-06-26 Thread Nils Goroll
It's still unproven whether it'd be an improvement, but you could expect to prove it one way or the other with a well-defined amount of testing. I've hacked the code to use adaptive pthread mutexes instead of spinlocks. see attached patch. The patch is for the git head, but it can easily be

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Alvaro Herrera
Excerpts from Tom Lane's message of mar jun 26 18:58:45 -0400 2012: Even if you actively try to configure the shmem settings to exactly fill shmmax (which I concede some installation scripts might do), it's going to be hard to do because of the 8K granularity of the main knob,

[HACKERS] embedded list v2

2012-06-26 Thread Andres Freund
Hi, To recapitulate why I think this sort of embedded list is worthwile: * minimal memory overhead (16 bytes for double linked list heads/nodes on 64bit systems) * no additional memory allocation overhead * no additional dereference to access the contents of a list element * most modifications

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
A.M. age...@themactionfaction.com writes: On Jun 26, 2012, at 6:12 PM, Daniel Farina wrote: I'm simply suggesting that for additional benefits it may be worth thinking about getting around nattach and thus SysV shmem, especially with regard to safety, in an open-ended way. I solved this via

Re: [HACKERS] [PATCH 08/16] Introduce the ApplyCache module which can reassemble transactions from a stream of interspersed changes

2012-06-26 Thread Andres Freund
Hi Steve, On Tuesday, June 26, 2012 02:14:22 AM Steve Singer wrote: I planned to have some cutoff 'max_changes_in_memory_per_txn' value. If it has been reached for one transaction all existing changes are spilled to disk. New changes again can be kept in memory till its reached again. Do

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread A.M.
On 06/26/2012 07:30 PM, Tom Lane wrote: A.M. age...@themactionfaction.com writes: On Jun 26, 2012, at 6:12 PM, Daniel Farina wrote: I'm simply suggesting that for additional benefits it may be worth thinking about getting around nattach and thus SysV shmem, especially with regard to safety, in

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread A.M.
On 06/26/2012 07:15 PM, Alvaro Herrera wrote: Excerpts from Tom Lane's message of mar jun 26 18:58:45 -0400 2012: Even if you actively try to configure the shmem settings to exactly fill shmmax (which I concede some installation scripts might do), it's going to be hard to do because of the 8K

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 6:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: So, what about keeping a FIFO in the data directory? Hm, does that work if the data directory is on NFS?  Or some other weird not-really-Unix file system? I would expect NFS to work

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
A.M. age...@themactionfaction.com writes: On 06/26/2012 07:30 PM, Tom Lane wrote: I solved this via fcntl locking. No, you didn't, because fcntl locks aren't inherited by child processes. Too bad, because they'd be a great solution otherwise. You claimed this last time and I replied:

Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal

2012-06-26 Thread David Fetter
On Tue, Jun 26, 2012 at 05:05:27PM -0500, Kevin Grittner wrote: David Fetter da...@fetter.org wrote: On Mon, Jun 25, 2012 at 01:50:54PM -0500, Kevin Grittner wrote: One fine point regarding before and after images -- if a value doesn't change in an UPDATE, there's no reason to include it

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
I wrote: Reflecting on this further, it seems to me that the main remaining failure modes are (1) file locking doesn't work, or (2) idiot DBA manually removes the lock file. Oh, wait, I just remembered the really fatal problem here: to quote from the SUS fcntl spec, All locks

Re: [HACKERS] WIP Patch: Selective binary conversion of CSV file foreign tables

2012-06-26 Thread Etsuro Fujita
Hi Kaigai-san, -Original Message- From: Kohei KaiGai [mailto:kai...@kaigai.gr.jp] Sent: Tuesday, June 26, 2012 11:05 PM To: Etsuro Fujita Cc: Robert Haas; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] WIP Patch: Selective binary conversion of CSV file foreign tables

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Robert Haas
On Tue, Jun 26, 2012 at 6:25 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: So let's fix the 80% case with something we feel confident in, and then revisit the no-sysv interlock as a separate patch.  That way if we can't fix the interlock issues, we still have a

[HACKERS] Optimizer Path Candidates difference in 9.1.3 and 9.2 beta1

2012-06-26 Thread Qi Huang
Hi, hackersI modified the code in add_path() a bit so that all the query path candidates inside pathlist will not be removed and all new path will be added into the pathlist, thus all path candidates are kept in pathlist. I then tested a four-relation query. In 9.1.3, I can see thousands

Re: [HACKERS] [PATCH] Lazy hashaggregate when no aggregation is needed

2012-06-26 Thread Etsuro Fujita
Hi, -Original Message- From: Robert Haas [mailto:robertmh...@gmail.com] Sent: Wednesday, June 27, 2012 5:09 AM To: Etsuro Fujita Cc: Ants Aasma; Jay Levitt; Tom Lane; PostgreSQL-development; Francois Deliege Subject: Re: [HACKERS] [PATCH] Lazy hashaggregate when no aggregation is

Re: [HACKERS] Posix Shared Mem patch

2012-06-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So, here's a patch. Instead of using POSIX shmem, I just took the expedient of using mmap() to map a block of MAP_SHARED|MAP_ANONYMOUS memory. The sysv shm is still allocated, but it's just a copy of PGShmemHeader; the real shared memory is the

Re: [HACKERS] Optimizer Path Candidates difference in 9.1.3 and 9.2 beta1

2012-06-26 Thread Tom Lane
Qi Huang huangq...@hotmail.com writes: Hi, hackersI modified the code in add_path() a bit so that all the query path candidates inside pathlist will not be removed and all new path will be added into the pathlist, thus all path candidates are kept in pathlist. I then tested a

Re: [HACKERS] Optimizer Path Candidates difference in 9.1.3 and 9.2 beta1

2012-06-26 Thread Qi Huang
[ shrug... ] When you're not showing us exactly what you did, it's hard to answer that for sure. But there is some prefiltering logic in joinpath.c that you might have to lobotomize too if you want to keep known-inferior join paths. regards, tom lane Thanks, Tom. Below is what I did for

  1   2   >