Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-09-19 Thread Etsuro Fujita
From: Tom Lane [mailto:t...@sss.pgh.pa.us] I wrote: Etsuro Fujita fujita.ets...@lab.ntt.co.jp writes: I have a question. I think it would be also better to extend the syntax for the SQL COPY command in the same way, ie, COPY foo from '/home/pgsql/decompress.sh /home/pgsql/foo.csv.gz |'

Re: [HACKERS] [ADMIN] pg_upgrade from 9.1.3 to 9.2 failed

2012-09-19 Thread Bruce Momjian
On Tue, Sep 18, 2012 at 07:22:39PM -0400, Bruce Momjian wrote: Based on the fact that sql_features exists in the information_schema schema, I don't think 'sql_features' table is actually being processed by pg_upgrade, but I think its TOAST table, because it has a high oid, is being

Re: [HACKERS] proposal - assign result of query to psql variable

2012-09-19 Thread Shigeru HANADA
On Fri, Aug 10, 2012 at 3:21 AM, Pavel Stehule pavel.steh...@gmail.com wrote: there is new version of this patch * cleaned var list parser * new regress tests * support FETCH_COUNT 0 Here are my review comments. Submission == The patch is formatted in context diff style, and it

Re: [HACKERS] Reduce palloc's in numeric operations.

2012-09-19 Thread Heikki Linnakangas
On 14.09.2012 11:25, Kyotaro HORIGUCHI wrote: Hello, I will propose reduce palloc's in numeric operations. The numeric operations are slow by nature, but usually it is not a problem for on-disk operations. Altough the slowdown is enhanced on on-memory operations. I inspcted them and found some

Re: [HACKERS] ToDo: allow to get a number of processed rows by COPY statement

2012-09-19 Thread Heikki Linnakangas
On 16.08.2012 14:43, Pavel Stehule wrote: Hello here is updated patch The patch seems to be truncated, it ends with: *** a/src/test/regress/input/copy.source --- b/src/test/regress/input/copy.source *** *** 106,108 this is just a line full of junk that would error out if

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Tom Lane
florian.schoppm...@emc.com (Florian Schoppmann) writes: In PostgreSQL 9.1 and 9.2 (possibly also in earlier versions), the query --8-- WITH source AS ( SELECT i FROM generate_series(1,10) AS i ) SELECT i FROM source, ( SELECT count(*) AS _n FROM

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Merlin Moncure
On Wed, Sep 19, 2012 at 9:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: florian.schoppm...@emc.com (Florian Schoppmann) writes: In PostgreSQL 9.1 and 9.2 (possibly also in earlier versions), the query --8-- WITH source AS ( SELECT i FROM generate_series(1,10) AS i ) SELECT i FROM

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Robert Haas
On Wed, Sep 19, 2012 at 10:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: florian.schoppm...@emc.com (Florian Schoppmann) writes: In PostgreSQL 9.1 and 9.2 (possibly also in earlier versions), the query --8-- WITH source AS ( SELECT i FROM generate_series(1,10) AS i ) SELECT i FROM

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Sep 19, 2012 at 10:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: To do what you want, I'd suggest wrapping the join into a sub-select with an OFFSET 0 clause, which will serve as an optimization fence that prevents the random() call from being

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Robert Haas
On Wed, Sep 19, 2012 at 12:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Sep 19, 2012 at 10:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: To do what you want, I'd suggest wrapping the join into a sub-select with an OFFSET 0 clause, which will serve as

[HACKERS] Removal of AcceptInvalidationMessages broke things

2012-09-19 Thread Tom Lane
I looked into bug #7557, which demonstrates a case where a session fails to notice a just-committed change in table permissions. This is pretty obviously due to a failure to read the sinval message notifying other backends of the pg_class.relacl update. Some digging in the git history says it

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: It still seems like awfully weird behavior. Why? The WHERE condition relates only to the output of the _stats subquery, so why shouldn't it be evaluated there, rather than after the join? In a green field I might agree that we should de-optimize such

Re: [HACKERS] Re: [COMMITTERS] pgsql: Properly set relpersistence for fake relcache entries.

2012-09-19 Thread Tom Lane
BTW, what should our advice be for recovering from corruption due to this bug? As far as the btree and GIN problems go, we can tell people that REINDEX will fix it. And in 9.1, you don't really need to worry about the visibility map being bad. But what do you do in 9.2, if you have a bad

Re: [HACKERS] Removal of AcceptInvalidationMessages broke things

2012-09-19 Thread Noah Misch
On Wed, Sep 19, 2012 at 01:17:17PM -0400, Tom Lane wrote: I looked into bug #7557, which demonstrates a case where a session fails to notice a just-committed change in table permissions. - /* -* Check for shared-cache-inval messages before trying to open the -* relation. This is

Re: [HACKERS] Re: [COMMITTERS] pgsql: Properly set relpersistence for fake relcache entries.

2012-09-19 Thread Simon Riggs
On 19 September 2012 18:47, Tom Lane t...@sss.pgh.pa.us wrote: BTW, what should our advice be for recovering from corruption due to this bug? As far as the btree and GIN problems go, we can tell people that REINDEX will fix it. And in 9.1, you don't really need to worry about the visibility

Re: [HACKERS] Removal of AcceptInvalidationMessages broke things

2012-09-19 Thread Tom Lane
Noah Misch n...@leadboat.com writes: On Wed, Sep 19, 2012 at 01:17:17PM -0400, Tom Lane wrote: Since we have only a few hours before 9.2.1 is due to wrap, my inclination for a band-aid fix is to put back that code. There might be some more elegant answer, but we haven't got time to find it

Re: [HACKERS] Removal of AcceptInvalidationMessages broke things

2012-09-19 Thread Robert Haas
On Wed, Sep 19, 2012 at 2:17 PM, Noah Misch n...@leadboat.com wrote: Sounds fine for now. I suspect the better change would be to make AcceptInvalidationMessages() unconditional in LockRelationOid() and friends. There's no reason to desire recent ACLs only when opening by name. I agree, on

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Robert Haas
On Wed, Sep 19, 2012 at 1:26 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: It still seems like awfully weird behavior. Why? The WHERE condition relates only to the output of the _stats subquery, so why shouldn't it be evaluated there, rather than after the

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: It still seems like awfully weird behavior. Why? The WHERE condition relates only to the output of the _stats subquery, so why shouldn't it be evaluated there, rather than after the join? In another thread, Tom

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread k...@rice.edu
On Wed, Sep 19, 2012 at 02:39:12PM -0500, Kevin Grittner wrote: Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: It still seems like awfully weird behavior. Why? The WHERE condition relates only to the output of the _stats subquery, so why shouldn't it be

[HACKERS] CTE optimization fence on the todo list?

2012-09-19 Thread Daniel Browning
I would like to have the option of disabling the CTE optimization fence for certain CTEs and/or queries. Can that be added to the official todo list? If not, why not? I would find the option beneficial because large, complicated queries are often a lot clearer, simpler, and easier to read with

Re: [v9.3] Extra Daemons (Re: [HACKERS] elegant and effective way for running jobs inside a database)

2012-09-19 Thread Simon Riggs
On 12 September 2012 04:30, Amit Kapila amit.kap...@huawei.com wrote: On Tuesday, September 11, 2012 9:09 PM Alvaro Herrera wrote: Excerpts from Boszormenyi Zoltan's message of vie jun 29 09:11:23 -0400 2012: We have some use cases for this patch, when can you post a new version? I would test

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Kevin Grittner
k...@rice.edu k...@rice.edu wrote: On Wed, Sep 19, 2012 at 02:39:12PM -0500, Kevin Grittner wrote: In another thread, Tom Lane t...@sss.pgh.pa.us wrote: 2. Apply the WHERE condition to each row from 1, and drop rows that don't pass it. People expect that the results will be consistent

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: There is a workaround, if you don't mind ugly: Or, better: WITH source AS ( SELECT i, random() AS r FROM generate_series(1,10) AS i ) SELECT i FROM source, ( SELECT count(*) AS _n FROM source ) AS

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread David Johnston
-Original Message- There really needs to be some way to specify that when an expression is evaluated for each row in a set, a function used within that expression is not optimized away for some rows. Fortunately we have a way:

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Kevin Grittner
David Johnston pol...@yahoo.com wrote: | VOLATILE indicates that the function value can change even | within a single table scan, so no optimizations can be made. | Relatively few database functions are volatile in this sense; | some examples are random(), [...] What are the arguments

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread David Johnston
-Original Message- | VOLATILE indicates that the function value can change even within a | single table scan, so no optimizations can be made. | Relatively few database functions are volatile in this sense; some | examples are random(), [...] What are the arguments against

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Kevin Grittner
David Johnston pol...@yahoo.com wrote: VOLATILE: A Volatile function used in an ORDER BY or WHERE clause without referencing any columns from the query itself (i.e., no parameters or all constants) will be evaluated a single time and the result treated as a constant (i.e., all rows will have

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread David Johnston
-Original Message- From: Kevin Grittner [mailto:kevin.gritt...@wicourts.gov] Sent: Wednesday, September 19, 2012 5:51 PM To: k...@rice.edu; David Johnston Cc: 'Florian Schoppmann'; 'Robert Haas'; pgsql-hackers@postgresql.org; 'Tom Lane' Subject: RE: [HACKERS] Invalid optimization of

Re: [HACKERS] [COMMITTERS] pgsql: Fix bufmgr so CHECKPOINT_END_OF_RECOVERY behaves as a shutdown c

2012-09-19 Thread Andres Freund
On Tuesday, September 18, 2012 04:18:01 AM Robert Haas wrote: Maybe what we should do is - if this is an end-of-recovery checkpoint - *assert* that the BM_PERMANENT bit is set on every buffer we find. That would provide a useful cross-check that we don't have a bug similar to the one Jeff

Re: [HACKERS] Invalid optimization of VOLATILE function in WHERE clause?

2012-09-19 Thread Florian Schoppmann
Tom Lane t...@sss.pgh.pa.us wrote: florian.schoppm...@emc.com (Florian Schoppmann) writes: [VOLATILE function in WHERE clause *does* get optimized] I can't get excited about this. Any time you put a volatile function into WHERE, you're playing with fire. The docs warn against it:

[HACKERS] newline conversion in SQL command strings

2012-09-19 Thread Peter Eisentraut
I have received a number of bug reports about plsh choking on Windows-style line endings. The problem is that the user uses some Windows-based tool or other to execute an SQL command line this: CREATE FUNCTION foo() RETURNS somethingCRLF LANGUAGE plshCRLF AS $$CRLF #!/bin/shCRLF CRLF do

Re: [HACKERS] 64-bit API for large object

2012-09-19 Thread Kohei KaiGai
I checked this patch. It can be applied onto the latest master branch without any problems. My comments are below. 2012/9/11 Tatsuo Ishii is...@postgresql.org: Ok, here is the patch to implement 64-bit API for large object, to allow to use up to 4TB large objects(or 16TB if BLCKSZ changed to

Re: [v9.3] Extra Daemons (Re: [HACKERS] elegant and effective way for running jobs inside a database)

2012-09-19 Thread Amit Kapila
On Thursday, September 20, 2012 1:44 AM Simon Riggs wrote: On 12 September 2012 04:30, Amit Kapila amit.kap...@huawei.com wrote: On Tuesday, September 11, 2012 9:09 PM Alvaro Herrera wrote: Excerpts from Boszormenyi Zoltan's message of vie jun 29 09:11:23 -0400 2012: We have some use cases for