Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Daniel Farina
On Thu, Mar 29, 2012 at 10:55 PM, Dobes Vandermeer dob...@gmail.com wrote: Virtual hosts. Same port. In that case, the frontend would not be tied to a specific PostgreSQL server, then?  I think initially this might complicate things a bit, and you could solve it by putting an HTTP proxy in

Re: [HACKERS] Command Triggers patch v18

2012-03-30 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: I'm thinking of things like extension whitelisting. When some unprivileged user says CREATE EXTENSION harmless, and harmless is marked as superuser-only, we might like to have a hook that gets called *at permissions-checking time* and gets to say, oh,

Re: [HACKERS] poll: CHECK TRIGGER?

2012-03-30 Thread Pavel Stehule
Hello 2012/3/28 Heikki Linnakangas heikki.linnakan...@enterprisedb.com: Ok, seems that the API issue is settled, so I'm now looking at the code actually doing the checking. My first impression is that this is a lot of code. Can we simplify it? I played with this and It is not be reduced

Re: [HACKERS] Finer Extension dependencies

2012-03-30 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: That's how I did it first, but Alvaro opposed to that because it allows for more than one extension to provide for the same feature name. http://archives.postgresql.org/pgsql-hackers/2012-03/msg01425.php Right, but the question that has to be considered

[HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Arun Chaitanya
Hi, I wanted to take up this as a GSOC 2012 project. SQL supports nested queries. When the inner query contains a correlation variable the present optimizer takes an iterative execution plan. If the inner query scans over a relation, the iterative plan chosen can be sub-optimal. The goal of

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 7:33 AM, Arun Chaitanya chaitan64a...@gmail.com wrote: I wanted to take up this as a GSOC 2012 project. This would be a great query planner optimization but the chances of getting it done in one summer as a GSoC project seem to me to be nil. You've never had a patch

[HACKERS] Re: [COMMITTERS] pgsql: pg_test_timing utility, to measure clock monotonicity and timing

2012-03-30 Thread Robert Haas
On Wed, Mar 28, 2012 at 10:43 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Mar 28, 2012 at 9:19 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 27, 2012 at 10:10 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Mar 28, 2012 at 5:17 AM, Robert Haas rh...@postgresql.org

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Merlin Moncure
On Thu, Mar 29, 2012 at 10:04 AM, Andrew Dunstan and...@dunslane.net wrote: 1. I've been in discussion with some people about adding simple JSON extract functions. We already have some (i.e. xpath()) for XML. I've built a couple of applications that push data in and out of xml via manual

Re: [HACKERS] Command Triggers patch v18

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 4:32 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: I did that another way in previous incarnations of the patch, which was to allow for INSTEAD OF event trigger backed by a SECURITY DEFINER function. When the extension is whitelisted, prevent against recursion then

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Arun Chaitanya
Thanks a lot Heikki. I have already posted an example in the mail. The link to the paper is http://www.iith.ac.in/~ravig/courses/cs5050/papers/decorrelation-cesar.pdf Hope this helps, Arun On Fri, Mar 30, 2012 at 5:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: (off-list)

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Arun Chaitanya
Thanks Robert, Yes. I think I am being over ambitious as I never had any Open Source development experience. Anyways, please go through the idea. I have posted the link to the paper in on of the replies. Please, suggest me other options which I can take up as a GSOC 2012 project. On Fri, Mar

Re: [HACKERS] Optimizing Nested Correlated Queries by decorrelation: GSOC 2012 Project

2012-03-30 Thread Tom Lane
Arun Chaitanya chaitan64a...@gmail.com writes: The link to the paper is http://www.iith.ac.in/~ravig/courses/cs5050/papers/decorrelation-cesar.pdf Given the authorship of that paper, I'd have to wonder whether Microsoft has filed for any patents regarding these ideas.

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Andrew Dunstan
On 03/30/2012 09:57 AM, Merlin Moncure wrote: What I'm saying is that jsonpath probably isn't the whole story: another way of bulk moving json into native backend structures without parsing would also be very helpful. For example, being able to cast a json document into a record or a record

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Dobes Vandermeer
On Fri, Mar 30, 2012 at 4:30 PM, Daniel Farina dan...@heroku.com wrote: On Thu, Mar 29, 2012 at 10:55 PM, Dobes Vandermeer dob...@gmail.com wrote: Virtual hosts. Same port. I think SPDY or like-protocols [...] give a crisp treatment to interactive, stateful workloads involving

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Dobes Vandermeer
On Fri, Mar 30, 2012 at 9:57 PM, Merlin Moncure mmonc...@gmail.com wrote: On Thu, Mar 29, 2012 at 10:04 AM, Andrew Dunstan and...@dunslane.net wrote: 1. I've been in discussion with some people about adding simple JSON extract functions. We already have some (i.e. xpath()) for XML. Your

Re: [HACKERS] Command Triggers patch v18

2012-03-30 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: Oh, right: I remember that now. I still think it's a bad way to do it, because the trigger potentially has a lot of work to do to reconstruct a working command string, and it still ends up getting executed by the wrong user. For CREATE EXTENSION it's

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 10:55 AM, Dobes Vandermeer dob...@gmail.com wrote: Well, in our case HTTP is a clear win (but not replacement) and SPDY a potential one (even as a replacement).  Even if SPDY is not widely adopted it could still replace FEBE if there's a clear advantage to using it, I

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: Marko Kreen mark...@gmail.com writes: My conclusion is that row-processor API is low-level expert API and quite easy to misuse. It would be preferable to have something more robust as end-user API, the PQgetRow() is my suggestion

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: Marko Kreen mark...@gmail.com writes: Second conclusion is that current dblink row-processor usage is broken when user uses multiple SELECTs in SQL as dblink uses plain PQexec(). Yeah. Perhaps we

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Fri, Mar 30, 2012 at 11:59:12AM -0400, Tom Lane wrote: Marko Kreen mark...@gmail.com writes: On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: Marko Kreen mark...@gmail.com writes: Second conclusion is that current dblink row-processor usage is broken when user uses multiple

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Andrew Dunstan
On 03/30/2012 11:41 AM, Robert Haas wrote: On Fri, Mar 30, 2012 at 10:55 AM, Dobes Vandermeerdob...@gmail.com wrote: Well, in our case HTTP is a clear win (but not replacement) and SPDY a potential one (even as a replacement). Even if SPDY is not widely adopted it could still replace FEBE

[HACKERS] tracking context switches with perf record

2012-03-30 Thread Robert Haas
I've been playing around with perf record on the IBM POWER7 machine, and it's pretty cool. One of the things I don't like is that some of the tools don't produce very nice text reports that you can cut and paste into an email - it's kind of a text-based GUI where you can zoom around and look at

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Fri, Mar 30, 2012 at 7:04 PM, Marko Kreen mark...@gmail.com wrote: Have you looked at the examples?  PQgetResult() is pretty good hint that one resultset finished... Ok, the demos are around this long thread and hard to find, so here is a summary of links: Original design mail:

[HACKERS] checkpointer code behaving strangely on postmaster -T

2012-03-30 Thread Alvaro Herrera
I noticed while doing some tests that the checkpointer process does not recover very nicely after a backend crashes under postmaster -T (after all processes have been kill -CONTd, of course, and postmaster told to shutdown via Ctrl-C on its console). For some reason it seems to get stuck on a

Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-03-30 Thread Peter Eisentraut
On fre, 2012-03-23 at 07:52 -0700, David Fetter wrote: On Thu, Mar 22, 2012 at 06:05:30PM -0400, Andrew Dunstan wrote: On 03/22/2012 05:49 PM, Bruce Momjian wrote: Robert Haas and I are disappointed by this change. I liked the fact that I could post nice-looking SQL queries without having

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Daniel Farina
On Fri, Mar 30, 2012 at 9:11 AM, Andrew Dunstan and...@dunslane.net wrote: On 03/30/2012 11:41 AM, Robert Haas wrote: On Fri, Mar 30, 2012 at 10:55 AM, Dobes Vandermeerdob...@gmail.com  wrote: Well, in our case HTTP is a clear win (but not replacement) and SPDY a potential one (even as a

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Greg Stark
On Fri, Mar 30, 2012 at 5:27 PM, Robert Haas robertmh...@gmail.com wrote: If you expand that branch of the call tree, you find that all of them are coming eventually from secure_read; the server is waiting for a new query from the client.  This is, obviously, overhead we can't eliminate from

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-30 Thread Daniel Farina
On Fri, Mar 30, 2012 at 10:21 AM, Daniel Farina dan...@heroku.com wrote: Any enhancement here that can't be used with libpq via, say, drop-in .so seems unworkable to me, and that's why any solution that is basically proxying to the database is basically a non-starter outside the very earliest

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Tom Lane
Greg Stark st...@mit.edu writes: On Fri, Mar 30, 2012 at 5:27 PM, Robert Haas robertmh...@gmail.com wrote: If you expand that branch of the call tree, you find that all of them are coming eventually from secure_read; the server is waiting for a new query from the client.  This is, obviously,

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 1:30 PM, Greg Stark st...@mit.edu wrote: On Fri, Mar 30, 2012 at 5:27 PM, Robert Haas robertmh...@gmail.com wrote: If you expand that branch of the call tree, you find that all of them are coming eventually from secure_read; the server is waiting for a new query from

[HACKERS] psql: tab-completion fails SET var=

2012-03-30 Thread Erik Rijkers
(in hopes that the current changes to tab-completion will help to get this fixed) tab-completion goes wrong on SET setting=... example: If you want to input set search_path=myschema; without spaces around '=', and you try tab-completion halfway the schemaname: set search_path=mysch

Re: [HACKERS] pg_upgrade incorrectly equates pg_default and database tablespace

2012-03-30 Thread Jeff Davis
On Thu, 2012-03-22 at 14:55 +0200, Ants Aasma wrote: Hi, while working on a support case I stumbled upon a bug in pg_upgrade. Upgrade fails with No such file or directory when a database is moved to a non-default tablespace and contains a table that is moved to pg_default. The cause seems

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Wed, Mar 07, 2012 at 03:14:57PM +0900, Kyotaro HORIGUCHI wrote: My suggestion - check in getAnotherTuple whether resultStatus is already error and do nothing then. This allows internal pqAddRow to set regular out of memory error. Otherwise give

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Fri, Mar 30, 2012 at 05:18:42PM -0400, Tom Lane wrote: Marko Kreen mark...@gmail.com writes: On Wed, Mar 07, 2012 at 03:14:57PM +0900, Kyotaro HORIGUCHI wrote: My suggestion - check in getAnotherTuple whether resultStatus is already error and do nothing then. This allows internal

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Fri, Mar 30, 2012 at 05:18:42PM -0400, Tom Lane wrote: I'm pretty dissatisfied with the error reporting situation for row processors. You can't just decide not to solve it, which seems to be the current state of affairs. What I'm inclined to do is to

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Andres Freund
On Friday, March 30, 2012 06:27:36 PM Robert Haas wrote: Probability=No, score=-3.2 required=5.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW autolearn=ham version=3.2.5 I've been playing around with perf record on the IBM POWER7 machine, and it's pretty cool. One of the things I don't like is

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Marko Kreen
On Sat, Mar 31, 2012 at 1:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: On Fri, Mar 30, 2012 at 05:18:42PM -0400, Tom Lane wrote: I'm pretty dissatisfied with the error reporting situation for row processors.  You can't just decide not to solve it, which seems

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-30 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Sat, Mar 31, 2012 at 1:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: Not if the message is a constant string, which seems like the typical situation (think out of memory).  If the row processor does need a buffer for a constructed string, it could make use

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-30 Thread Albe Laurenz
I wrote: How about getting # of rows estimate by executing EXPLAIN for fully-fledged remote query (IOW, contains pushed-down WHERE clause), and estimate selectivity of local filter on the basis of the statistics which are generated by FDW via do_analyze_rel() and FDW-specific sampling

Re: [HACKERS] Standbys, txid_current_snapshot, wraparound

2012-03-30 Thread Andrey Velikoredchanin
2012/3/29 Marko Kreen mark...@gmail.com I can't find a place where WAL replay updates values under XLogCtl. If that really does not happen, that would explain why standbys can see wrong epoch. No clue yet how master can get broken. Details about environment: Debian Linux db

Re: [HACKERS] [PATCH] Lazy hashaggregate when no aggregation is needed

2012-03-30 Thread Jay Levitt
Tom Lane wrote: Ants Aasmaa...@cybertec.at writes: A user complained on pgsql-performance that SELECT col FROM table GROUP BY col LIMIT 2; performs a full table scan. ISTM that it's safe to return tuples from hash-aggregate as they are found when no aggregate functions are in use. Attached is a

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Jeff Janes
On Fri, Mar 30, 2012 at 9:27 AM, Robert Haas robertmh...@gmail.com wrote: The more interesting waits, in my view anyway, are the ones that come from LWLockAcquire, which account for nearly all of the semaphore sleeps.  As you can see, XLogInsert accounts for over half of those, and

Re: [HACKERS] tracking context switches with perf record

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 10:07 PM, Jeff Janes jeff.ja...@gmail.com wrote: Which I've now done, and it actually goes the other way.  with -F50, not only do almost all pgbench_accounts updates still lead to a clean, but a good chunk of the updates to pgbench_tellers lead to a clean as well.  If

[HACKERS] measuring lwlock-related latency spikes

2012-03-30 Thread Robert Haas
On Fri, Mar 30, 2012 at 1:55 PM, Robert Haas robertmh...@gmail.com wrote: Actually, what is really bugging me is that I cannot find any way of getting a profile that reflects the *time* spent waiting rather than merely the *number* of waits.  That seems like an obvious thing to want, and I