Re: [HACKERS] Odd out of memory problem.

2012-03-30 Thread Hitoshi Harada
On Thu, Mar 29, 2012 at 7:38 PM, Peter Eisentraut wrote: > On tis, 2012-03-27 at 00:53 +0100, Greg Stark wrote: >> Hm. So my original plan was dependent on adding the state-merge >> function we've talked about in the past. Not all aggregate functions >> necessarily can support such a function but

[HACKERS] measuring lwlock-related latency spikes

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 1:55 PM, Robert Haas wrote: > Actually, what is really bugging me is that I cannot find any way of > getting a profile that reflects the *time* spent waiting rather than > merely the *number* of waits.  That seems like an obvious thing to > want, and I cannot find a way to

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 10:07 PM, Jeff Janes wrote: > Which I've now done, and it actually goes the other way.  with -F50, > not only do almost all pgbench_accounts updates still lead to a clean, > but a good chunk of the updates to pgbench_tellers lead to a clean as > well.  If cleans are going t

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Jeff Janes
On Fri, Mar 30, 2012 at 9:27 AM, Robert Haas wrote: > > The more interesting waits, in my view anyway, are the ones that come > from LWLockAcquire, which account for nearly all of the semaphore > sleeps.  As you can see, XLogInsert accounts for over half of those, > and ProcArrayEndTransaction and

Re: [HACKERS] [PATCH] Lazy hashaggregate when no aggregation is needed

2012-03-30 Thread Jay Levitt
Tom Lane wrote: Ants Aasma writes: A user complained on pgsql-performance that SELECT col FROM table GROUP BY col LIMIT 2; performs a full table scan. ISTM that it's safe to return tuples from hash-aggregate as they are found when no aggregate functions are in use. Attached is a first shot at th

Re: [HACKERS] Standbys, txid_current_snapshot, wraparound

2012-03-30 Thread Andrey Velikoredchanin
2012/3/29 Marko Kreen > I can't find a place where WAL replay updates values under XLogCtl. > If that really does not happen, that would explain why standbys can > see wrong epoch. > > No clue yet how master can get broken. > > Details about environment: Debian Linux db 2.6.32-5-amd64 #1 SMP Fr

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-30 Thread Albe Laurenz
I wrote: >> Changes from previous version >> = >> >> 1) Don't use remote EXPLAIN for cost/rows estimation, so now planner >> estimates result rows and costs on the basis of local statistics such >> as pg_class and pg_statistic. To update local statistics, I added >> pgs

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-30 Thread Albe Laurenz
I wrote: >> How about getting # of rows estimate by executing EXPLAIN for >> fully-fledged remote query (IOW, contains pushed-down WHERE clause), and >> estimate selectivity of local filter on the basis of the statistics >> which are generated by FDW via do_analyze_rel() and FDW-specific >> samplin

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen writes: > On Sat, Mar 31, 2012 at 1:13 AM, Tom Lane wrote: >> Not if the message is a constant string, which seems like the typical >> situation (think "out of memory").  If the row processor does need a >> buffer for a constructed string, it could make use of some space in its >> "vo

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Sat, Mar 31, 2012 at 1:13 AM, Tom Lane wrote: > Marko Kreen writes: >> On Fri, Mar 30, 2012 at 05:18:42PM -0400, Tom Lane wrote: >>> I'm pretty dissatisfied with the error reporting situation for row >>> processors.  You can't just decide not to solve it, which seems to be >>> the current stat

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Andres Freund
On Friday, March 30, 2012 06:27:36 PM Robert Haas wrote: > Probability=No, score=-3.2 required=5.0 tests=AWL,BAYES_00, > RCVD_IN_DNSWL_LOW autolearn=ham version=3.2.5 I've been playing around > with "perf record" on the IBM POWER7 machine, and it's pretty cool. One > of the things I don't like i

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen writes: > On Fri, Mar 30, 2012 at 05:18:42PM -0400, Tom Lane wrote: >> I'm pretty dissatisfied with the error reporting situation for row >> processors. You can't just decide not to solve it, which seems to be >> the current state of affairs. What I'm inclined to do is to add a >> "c

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Fri, Mar 30, 2012 at 05:18:42PM -0400, Tom Lane wrote: > Marko Kreen writes: > > On Wed, Mar 07, 2012 at 03:14:57PM +0900, Kyotaro HORIGUCHI wrote: > >>> My suggestion - check in getAnotherTuple whether resultStatus is > >>> already error and do nothing then. This allows internal pqAddRow > >>

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen writes: > On Wed, Mar 07, 2012 at 03:14:57PM +0900, Kyotaro HORIGUCHI wrote: >>> My suggestion - check in getAnotherTuple whether resultStatus is >>> already error and do nothing then. This allows internal pqAddRow >>> to set regular "out of memory" error. Otherwise give generic >>>

Re: [HACKERS] pg_upgrade incorrectly equates pg_default and database tablespace

2012-03-30 Thread Jeff Davis
On Thu, 2012-03-22 at 14:55 +0200, Ants Aasma wrote: > Hi, > > while working on a support case I stumbled upon a bug in pg_upgrade. > Upgrade fails with "No such file or directory" when a database is > moved to a non-default tablespace and contains a table that is moved > to pg_default. The cause

[HACKERS] psql: tab-completion fails SET var=

2012-03-30 Thread Erik Rijkers
(in hopes that the current changes to tab-completion will help to get this fixed) tab-completion goes wrong on SET setting=... example: If you want to input "set search_path=myschema;" without spaces around '=', and you try tab-completion halfway the schemaname: set search_path=mysch

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 1:30 PM, Greg Stark wrote: > On Fri, Mar 30, 2012 at 5:27 PM, Robert Haas wrote: >> If you expand that branch of the call tree, you find that all of them >> are coming eventually from secure_read; the server is waiting for a >> new query from the client.  This is, obviousl

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Tom Lane
Greg Stark writes: > On Fri, Mar 30, 2012 at 5:27 PM, Robert Haas wrote: >> If you expand that branch of the call tree, you find that all of them >> are coming eventually from secure_read; the server is waiting for a >> new query from the client.  This is, obviously, overhead we can't >> eliminat

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Daniel Farina
On Fri, Mar 30, 2012 at 10:21 AM, Daniel Farina wrote: > Any enhancement here that can't be used with libpq via, say, drop-in > .so seems unworkable to me, and that's why any solution that is > basically proxying to the database is basically a non-starter outside > the very earliest prototyping st

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Greg Stark
On Fri, Mar 30, 2012 at 5:27 PM, Robert Haas wrote: > If you expand that branch of the call tree, you find that all of them > are coming eventually from secure_read; the server is waiting for a > new query from the client.  This is, obviously, overhead we can't > eliminate from this test; waiting

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Daniel Farina
On Fri, Mar 30, 2012 at 9:11 AM, Andrew Dunstan wrote: > > > On 03/30/2012 11:41 AM, Robert Haas wrote: >> >> On Fri, Mar 30, 2012 at 10:55 AM, Dobes Vandermeer >>  wrote: >>> >>> Well, in our case HTTP is a clear win (but not replacement) and SPDY a >>> potential one (even as a replacement).  Eve

Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-03-30 Thread Peter Eisentraut
On fre, 2012-03-23 at 07:52 -0700, David Fetter wrote: > On Thu, Mar 22, 2012 at 06:05:30PM -0400, Andrew Dunstan wrote: > > On 03/22/2012 05:49 PM, Bruce Momjian wrote: > > >Robert Haas and I are disappointed by this change. I liked the > > >fact that I could post nice-looking SQL queries without

[HACKERS] checkpointer code behaving strangely on postmaster -T

2012-03-30 Thread Alvaro Herrera
I noticed while doing some tests that the checkpointer process does not recover very nicely after a backend crashes under postmaster -T (after all processes have been kill -CONTd, of course, and postmaster told to shutdown via Ctrl-C on its console). For some reason it seems to get stuck on a loo

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Fri, Mar 30, 2012 at 7:04 PM, Marko Kreen wrote: > Have you looked at the examples?  PQgetResult() is pretty good hint > that one resultset finished... Ok, the demos are around this long thread and hard to find, so here is a summary of links: Original design mail: http://archives.postgresq

[HACKERS] tracking context switches with perf record

2012-03-30 Thread Robert Haas
I've been playing around with "perf record" on the IBM POWER7 machine, and it's pretty cool. One of the things I don't like is that some of the tools don't produce very nice text reports that you can cut and paste into an email - it's kind of a text-based GUI where you can zoom around and look at

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Andrew Dunstan
On 03/30/2012 11:41 AM, Robert Haas wrote: On Fri, Mar 30, 2012 at 10:55 AM, Dobes Vandermeer wrote: Well, in our case HTTP is a clear win (but not replacement) and SPDY a potential one (even as a replacement). Even if SPDY is not widely adopted it could still replace FEBE if there's a clear

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Fri, Mar 30, 2012 at 11:59:12AM -0400, Tom Lane wrote: > Marko Kreen writes: > > On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: > >> Marko Kreen writes: > >>> Second conclusion is that current dblink row-processor usage is broken > >>> when user uses multiple SELECTs in SQL as dblin

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen writes: > On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: >> Marko Kreen writes: >>> Second conclusion is that current dblink row-processor usage is broken >>> when user uses multiple SELECTs in SQL as dblink uses plain PQexec(). >> Yeah. Perhaps we should tweak the row-pr

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: > Marko Kreen writes: > > My conclusion is that row-processor API is low-level expert API and > > quite easy to misuse. It would be preferable to have something more > > robust as end-user API, the PQgetRow() is my suggestion for that. > >

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 10:55 AM, Dobes Vandermeer wrote: > Well, in our case HTTP is a clear win (but not replacement) and SPDY a > potential one (even as a replacement).  Even if SPDY is not widely adopted > it could still replace FEBE if there's a clear advantage to using it, I > don't know eno

Re: [HACKERS] Command Triggers patch v18

2012-03-30 Thread Dimitri Fontaine
Robert Haas writes: > Oh, right: I remember that now. I still think it's a bad way to do > it, because the trigger potentially has a lot of work to do to > reconstruct a working command string, and it still ends up getting > executed by the wrong user. For CREATE EXTENSION it's not that bad, Th

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Dobes Vandermeer
On Fri, Mar 30, 2012 at 9:57 PM, Merlin Moncure wrote: > On Thu, Mar 29, 2012 at 10:04 AM, Andrew Dunstan > wrote: > > 1. I've been in discussion with some people about adding simple JSON > extract > > functions. We already have some (i.e. xpath()) for XML. > > Your json work is a great start in

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Dobes Vandermeer
On Fri, Mar 30, 2012 at 4:30 PM, Daniel Farina wrote: > On Thu, Mar 29, 2012 at 10:55 PM, Dobes Vandermeer > wrote: > >> Virtual hosts. Same port.>> I think SPDY or like-protocols [...] give a > crisp treatment to > >> interactive, stateful workloads involving > >> > >> back-and-forth between cl

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Andrew Dunstan
On 03/30/2012 09:57 AM, Merlin Moncure wrote: What I'm saying is that jsonpath probably isn't the whole story: another way of bulk moving json into native backend structures without parsing would also be very helpful. For example, being able to cast a json document into a record or a record a

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Tom Lane
Arun Chaitanya writes: > The link to the paper is > http://www.iith.ac.in/~ravig/courses/cs5050/papers/decorrelation-cesar.pdf Given the authorship of that paper, I'd have to wonder whether Microsoft has filed for any patents regarding these ideas. regards, tom lane --

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Arun Chaitanya
Thanks Robert, Yes. I think I am being over ambitious as I never had any Open Source development experience. Anyways, please go through the idea. I have posted the link to the paper in on of the replies. Please, suggest me other options which I can take up as a GSOC 2012 project. On Fri, Mar 30,

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Arun Chaitanya
Thanks a lot Heikki. I have already posted an example in the mail. The link to the paper is http://www.iith.ac.in/~ravig/courses/cs5050/papers/decorrelation-cesar.pdf Hope this helps, Arun On Fri, Mar 30, 2012 at 5:32 PM, Heikki Linnakangas wrote: > (off-list) > > You'll want to post a link to t

Re: [HACKERS] Command Triggers patch v18

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 4:32 AM, Dimitri Fontaine wrote: > I did that another way in previous incarnations of the patch, which was > to allow for INSTEAD OF event trigger backed by a SECURITY DEFINER > function. When the extension is whitelisted, prevent against recursion > then CREATE EXTENSION i

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Merlin Moncure
On Thu, Mar 29, 2012 at 10:04 AM, Andrew Dunstan wrote: > 1. I've been in discussion with some people about adding simple JSON extract > functions. We already have some (i.e. xpath()) for XML. I've built a couple of applications that push data in and out of xml via manual composition going out an

[HACKERS] Re: [COMMITTERS] pgsql: pg_test_timing utility, to measure clock monotonicity and timing

2012-03-30 Thread Robert Haas
On Wed, Mar 28, 2012 at 10:43 AM, Fujii Masao wrote: > On Wed, Mar 28, 2012 at 9:19 PM, Robert Haas wrote: >> On Tue, Mar 27, 2012 at 10:10 PM, Fujii Masao wrote: >>> On Wed, Mar 28, 2012 at 5:17 AM, Robert Haas wrote: pg_test_timing utility, to measure clock monotonicity and timing cost.

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 7:33 AM, Arun Chaitanya wrote: > I wanted to take up this as a GSOC 2012 project. This would be a great query planner optimization but the chances of getting it done in one summer as a GSoC project seem to me to be nil. You've never had a patch accepted before; picking a m

[HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Arun Chaitanya
Hi, I wanted to take up this as a GSOC 2012 project. SQL supports nested queries. When the inner query contains a correlation variable the present optimizer takes an iterative execution plan. If the inner query scans over a relation, the iterative plan chosen can be sub-optimal. The goal of this

Re: [HACKERS] Finer Extension dependencies

2012-03-30 Thread Dimitri Fontaine
Tom Lane writes: >> That's how I did it first, but Alvaro opposed to that because it allows >> for more than one extension to provide for the same feature name. >> http://archives.postgresql.org/pgsql-hackers/2012-03/msg01425.php > > Right, but the question that has to be considered is how often

Re: [HACKERS] poll: CHECK TRIGGER?

2012-03-30 Thread Pavel Stehule
Hello 2012/3/28 Heikki Linnakangas : > Ok, seems that the API issue is settled, so I'm now looking at the code > actually doing the checking. My first impression is that this is a lot of > code. Can we simplify it? > I played with this and It is not be reduced without darkening current code of pl

Re: [HACKERS] Command Triggers patch v18

2012-03-30 Thread Dimitri Fontaine
Robert Haas writes: > I'm thinking of things like extension whitelisting. When some > unprivileged user says "CREATE EXTENSION harmless", and harmless is > marked as superuser-only, we might like to have a hook that gets > called *at permissions-checking time* and gets to say, oh, well, that > ex

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Daniel Farina
On Thu, Mar 29, 2012 at 10:55 PM, Dobes Vandermeer wrote: >> Virtual hosts. Same port. > > In that case, the frontend would not be tied to a specific PostgreSQL > server, then?  I think initially this might complicate things a bit, and you > could solve it by putting an HTTP proxy in front to do t