Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Boszormenyi Zoltan
2013-04-03 20:58 keltezéssel, Gavin Flower írta: On 04/04/13 05:36, David E. Wheeler wrote: On Apr 3, 2013, at 9:30 AM, Tom Lanet...@sss.pgh.pa.us wrote: Fortran ... Basic ... actually I'd have thought that zero was a minority position. Fashions change I guess. I say we turn the default

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 2:52 AM Robert Haas wrote: On Wed, Apr 3, 2013 at 2:54 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut pete...@gmx.net wrote: It's weird that SET LOCAL and SET SESSION actually

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 7:19 AM Greg Smith wrote: On 4/2/13 11:54 AM, Robert Haas wrote: But, having said that, I still think the best idea is what Andres proposed, which pretty much matches my own thoughts: the bgwriter needs to populate the free list, so that buffer allocations don't

Re: [HACKERS] Drastic performance loss in assert-enabled build in HEAD

2013-04-04 Thread Nicolas Barbier
2013/4/3 Tom Lane t...@sss.pgh.pa.us: Kevin Grittner kgri...@ymail.com writes: To be honest, I don't think I've personally seen a single use case for matviews where they could be used if you couldn't count on an error if attempting to use them without the contents reflecting a

Re: [HACKERS] CREATE EXTENSION BLOCKS

2013-04-04 Thread Dimitri Fontaine
Hi, I though we were more specific about an extension's object itself not living in a schema in our documentation, but I agree we still have room for progress here. David E. Wheeler da...@justatheory.com writes: +Note that only the extension objects will be placed into the named +

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Simon Riggs
On 4 April 2013 02:39, Andres Freund and...@2ndquadrant.com wrote: Ok, I think I see the bug. And I think its been introduced in the checkpoints patch. Well spotted. (I think you mean checksums patch). If by now the first backend has proceeded to PageSetLSN() we are writing different data

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-04 Thread Robert Haas
On Wed, Apr 3, 2013 at 9:49 PM, Greg Smith g...@2ndquadrant.com wrote: On 4/2/13 11:54 AM, Robert Haas wrote: But, having said that, I still think the best idea is what Andres proposed, which pretty much matches my own thoughts: the bgwriter needs to populate the free list, so that buffer

[HACKERS] Number of spinlocks

2013-04-04 Thread Heikki Linnakangas
With --disable-spinlocks, we need to know the number of spinlocks in the system at startup, so that we can reserve enough semaphores to mimic the spinlocks. It's calculated in SpinLockSemas(): /* * Report number of semaphores needed to support spinlocks. */ int SpinlockSemas(void) {

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Andres Freund
On 2013-04-04 13:30:40 +0100, Simon Riggs wrote: On 4 April 2013 02:39, Andres Freund and...@2ndquadrant.com wrote: Ok, I think I see the bug. And I think its been introduced in the checkpoints patch. Well spotted. (I think you mean checksums patch). Heh, yes. I was slightly tired at that

[HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Rodrigo Barboza
Hi guys. I am wondering when I can use the PG_GETARG_UINT32 and PG_RETURN_UINT32. If postgres has no unsigned int type, what is the use of these macros?

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Merlin Moncure
On Wed, Apr 3, 2013 at 11:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: In any case, the whole exercise is pointless if we don't change the visible behavior of array_dims et al. So I think the idea that this would be without visible consequence is silly. What's up for argument is just how much

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Robert Haas
On Wed, Apr 3, 2013 at 9:40 PM, Greg Stark st...@mit.edu wrote: I used to advocate a similar idea. But when questioned on list I tried to work out the details and ran into some problem coming up with a concrete plan. How do you compare a plan that you think has a 99% chance of running in 1ms

Re: [HACKERS] CREATE EXTENSION BLOCKS

2013-04-04 Thread David E. Wheeler
On Apr 4, 2013, at 5:16 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: David E. Wheeler da...@justatheory.com writes: +Note that only the extension objects will be placed into the named +schema; the extension itself is a database-global object. I think you're patching

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Simon Riggs
On 4 April 2013 15:53, Andres Freund and...@2ndquadrant.com wrote: Unfortunately I find that approach unacceptably ugly. Yeh. If we can confirm its a fix we can discuss a cleaner patch and that is much better. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] psql crash fix

2013-04-04 Thread Bruce Momjian
On Tue, Apr 2, 2013 at 08:48:53PM -0400, Bruce Momjian wrote: I found that psql will crash if given a PSQLRC value containing a tilde: $ PSQLRC=~/x psql test *** glibc detected *** psql: free(): invalid pointer: 0x7fffb7c933ec *** This is on Debian Squeeze 6.0.7. The

Re: [HACKERS] Regex with 32k different chars causes a backend crash

2013-04-04 Thread Heikki Linnakangas
On 04.04.2013 03:32, Noah Misch wrote: On Wed, Apr 03, 2013 at 08:09:15PM +0300, Heikki Linnakangas wrote: --- a/src/include/regex/regguts.h +++ b/src/include/regex/regguts.h @@ -148,6 +148,7 @@ typedef short color; /* colors of characters */ typedef int pcolor;

[HACKERS] puzzling JSON bug

2013-04-04 Thread Andrew Dunstan
David Wheeler has presented me with a nasty bug case. If I do this: select '{members: { add: [3, 4]}}'::json # '{members,add}'; then I get a crash. If I comment out the pfree() at json.c:parse_object_field() lines 378-9 then I get back the right result but instead get a warning like this:

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
Tom, all, * Tom Lane (t...@sss.pgh.pa.us) wrote: So that's at least going in the right direction. I agree that this is going in the right direction; it certainly would make the plan that I *expect* to be chosen more likely, however.. I've been fiddling with this on the very much larger overall

Re: [HACKERS] Drastic performance loss in assert-enabled build in HEAD

2013-04-04 Thread Kevin Grittner
Early versions of the matview patch had a relisvalid flag in pg_class to determine whether the relation was scannable.  The name was chosen based on a similarity to the purpose of indisvalid, although the proliferation of new bools for related issues left me wondering if a char would be a better

Re: [HACKERS] pg_dump selectively ignores extension configuration tables

2013-04-04 Thread Vibhor Kumar
On Mar 25, 2013, at 12:01 PM, Vibhor Kumar vibhor.ku...@enterprisedb.com wrote: On Mar 25, 2013, at 10:48 AM, Joe Conway m...@joeconway.com wrote: On 03/25/2013 08:12 AM, Vibhor Kumar wrote: Since, nobody has picked this one. If there is no objection,then I can test this patch against

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: I've been fiddling with this on the very much larger overall database where this test case came from and have found that hashing the large table can actually be *faster* and appears to cause a more consistent and constant amount of disk i/o (which is

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: What I'm trying to get at in this overall email is: why in the world is it so expensive to do hash lookups? perf or oprofile reveal anything? Working on a test case actually- I've got one now: http://snowman.net/~sfrost/test_case2.sql In this

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: for estimate_worstcase_fraction. So, when computing the cost of a path, we'd compute our current expected-case estimate, and also a worst-case estimate, and then compute the final cost as: There also was the idea for the executor to be able to handle

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: perf or oprofile reveal anything? Here's what we get from oprofile (perhaps not too surprising): Hash the small table / scan the big table: samples cum. samples %cum. % linenr info image name symbol name 167374

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: 8504125241524.3624 72.3115mcount.c:60 libc-2.15.so __mcount_internal 28370280785 8.1274 80.4389_mcount.S:33 libc-2.15.so mcount [...] And as a side-note, I'm

Re: [HACKERS] puzzling JSON bug

2013-04-04 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: David Wheeler has presented me with a nasty bug case. If I do this: select '{members: { add: [3, 4]}}'::json # '{members,add}'; then I get a crash. If I comment out the pfree() at json.c:parse_object_field() lines 378-9 then I get back the

Re: [HACKERS] puzzling JSON bug

2013-04-04 Thread Andrew Dunstan
On 04/04/2013 03:39 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: David Wheeler has presented me with a nasty bug case. If I do this: select '{members: { add: [3, 4]}}'::json # '{members,add}'; then I get a crash. If I comment out the pfree() at

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
Andres, Thank you for diagnosing this problem! On Thu, 2013-04-04 at 16:53 +0200, Andres Freund wrote: I think the route you quickly sketched is more realistic. That would remove all knowledge obout XLOG_HINT from generic code hich is a very good thing, I spent like 15minutes yesterday

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: It does look like reducing bucket depth, as I outlined before through the use of a 2-level hashing system, might help speed up ExecScanHashBucket, as it would hopefully have very few (eg: 1-2) entries to consider instead of more. Along those same

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Robert Haas
On Thu, Apr 4, 2013 at 2:53 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: for estimate_worstcase_fraction. So, when computing the cost of a path, we'd compute our current expected-case estimate, and also a worst-case estimate, and then compute

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: Looking with opannotate, there's two main hotspots in ExecScanHashBucket: 12846 17.4001 :hashTuple = hashtable-buckets[hjstate-hj_CurBucketNo]; and 22100 29.9348 :hashTuple = hashTuple-next; Those are, of course, pretty trivial

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Robert Haas
On Thu, Apr 4, 2013 at 11:10 AM, Merlin Moncure mmonc...@gmail.com wrote: The only reasonable answer for this (a provably used, non-security, non-standards violating, non-gross functionality breakage case) is *zero*. +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise

Re: [HACKERS] Clang compiler warning on 9.3 HEAD

2013-04-04 Thread Alvaro Herrera
Will Leinweber wrote: On ref 8507907 when compiling with clang on os x, I got this warning which seems like a possible bug. I thought to report this because I imagine clang isn't frequently used day-to-day by most. Ugh. My fault. Yes, this is a bug. I don't see any nice way to convert

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Andres Freund
On 2013-04-04 12:59:36 -0700, Jeff Davis wrote: Andres, Thank you for diagnosing this problem! On Thu, 2013-04-04 at 16:53 +0200, Andres Freund wrote: I think the route you quickly sketched is more realistic. That would remove all knowledge obout XLOG_HINT from generic code hich is a

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Apr 4, 2013 at 11:10 AM, Merlin Moncure mmonc...@gmail.com wrote: The only reasonable answer for this (a provably used, non-security, non-standards violating, non-gross functionality breakage case) is *zero*. +1. Well, if we're going to take

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread k...@rice.edu
On Thu, Apr 04, 2013 at 04:16:12PM -0400, Stephen Frost wrote: * Stephen Frost (sfr...@snowman.net) wrote: It does look like reducing bucket depth, as I outlined before through the use of a 2-level hashing system, might help speed up ExecScanHashBucket, as it would hopefully have very few

Re: [HACKERS] CREATE EXTENSION BLOCKS

2013-04-04 Thread Tom Lane
David E. Wheeler da...@justatheory.com writes: On Apr 3, 2013, at 11:41 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Right -- an extension is not considered to live within a schema, they are database-global. The objects might live in a particular schema (if it is relocatable), and

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Janes
On Thu, Apr 4, 2013 at 5:30 AM, Simon Riggs si...@2ndquadrant.com wrote: On 4 April 2013 02:39, Andres Freund and...@2ndquadrant.com wrote: If by now the first backend has proceeded to PageSetLSN() we are writing different data to disk than the one we computed the checksum of before.

Re: [HACKERS] Clang compiler warning on 9.3 HEAD

2013-04-04 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Now, it annoys me that we now have three places that know about object types supported by event triggers: there's a large struct of command tag substrings (event_trigger_support), then there's these two functions. It might be better to add

[HACKERS] matview scannability rehash (was Re: Drastic performance loss in assert-enabled build in HEAD)

2013-04-04 Thread Noah Misch
Subject updated to account for the wider topics now appearing. On Wed, Apr 03, 2013 at 05:49:18PM -0400, Tom Lane wrote: What I'd actually rather see us spending time on right now is making some provision for incremental updates, which I will boldly propose could be supported by user-written

Re: [HACKERS] Drastic performance loss in assert-enabled build in HEAD

2013-04-04 Thread Noah Misch
On Thu, Apr 04, 2013 at 12:28:01PM +0200, Nicolas Barbier wrote: 2013/4/3 Tom Lane t...@sss.pgh.pa.us: And if you're absolutely convinced that unlogged matviews mustn't work as I suggest, we can lose those from 9.3, too. +1. Having unlogged matviews without having incremental updates yet,

Re: [HACKERS] matview scannability rehash (was Re: Drastic performance loss in assert-enabled build in HEAD)

2013-04-04 Thread Tom Lane
Noah Misch n...@leadboat.com writes: On Wed, Apr 03, 2013 at 05:49:18PM -0400, Tom Lane wrote: No. This is an absolute disaster. It's taking something we have always considered to be an irrelevant implementation detail and making it into user-visible DDL state, despite the fact that it

Re: [HACKERS] matview scannability rehash (was Re: Drastic performance loss in assert-enabled build in HEAD)

2013-04-04 Thread Alvaro Herrera
Tom Lane escribió: Noah Misch n...@leadboat.com writes: A slight variation on the committed approach would be to add a _scannable relation fork. Not very transaction-safe, I think (consider crash midway through a transaction that adds or removes the fork), and in any case orders of

Re: [HACKERS] patch to add \watch to psql

2013-04-04 Thread Tom Lane
Will Leinweber w...@heroku.com writes: Here is an updated patch that addresses several of the points brought up so far, such as the sleep, internationalization banner, and zero wait check, and it removes the premature input check. I whacked this around some more, added basic docs, and

Re: [HACKERS] patch to add \watch to psql

2013-04-04 Thread Will Leinweber
On Thu, Apr 4, 2013 at 5:04 PM, Tom Lane t...@sss.pgh.pa.us wrote: I whacked this around some more, added basic docs, and committed it. Thanks! Unfortunately rl_clear_screen() is not included at all in libedit, causing compilation to fail, and I was completely unable to find a way to

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
On Thu, 2013-04-04 at 14:21 -0700, Jeff Janes wrote: This brings up a pretty frightening possibility to me, unrelated to data checksums. If a bit gets twiddled in the WAL file due to a hardware issue or a cosmic ray, and then a crash happens, automatic recovery will stop early with the

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
On Thu, 2013-04-04 at 22:39 +0200, Andres Freund wrote: I don't think its really slower. Earlier the code took WalInsertLock everytime, even if we ended up not logging anything. Thats far more epensive than a single spinlock. And the copy should also only be taken in the case we need to log.

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Thu, 2013-04-04 at 14:21 -0700, Jeff Janes wrote: This brings up a pretty frightening possibility to me, unrelated to data checksums. If a bit gets twiddled in the WAL file due to a hardware issue or a cosmic ray, and then a crash happens, automatic

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Brendan Jurd
On 5 April 2013 07:43, Tom Lane t...@sss.pgh.pa.us wrote: Well, if we're going to take that hard a line on it, then we can't change anything about array data storage or the existing functions' behavior; which leaves us with either doing nothing at all, or inventing new functions that have

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
On Thu, 2013-04-04 at 21:06 -0400, Tom Lane wrote: I can't escape the feeling that we'd just be reinventing software RAID. There's no reason to think that we can deal with this class of problems better than the storage system can. The goal would be to reliably detect a situation where WAL that

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Tom Lane
Brendan Jurd dire...@gmail.com writes: The other suggestion that had been tossed around elsewhere upthread was inventing a new type that serves the demand for a straightforward mutable list, which has exactly one dimension, and which may be sensibly empty. Those few who are interested in

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Brendan Jurd
On 5 April 2013 13:04, Tom Lane t...@sss.pgh.pa.us wrote: (There's been a remarkable lack of attention to the question of spec compliance in this thread, btw. Surely the standard has something to say on the matter of zero-length arrays?) From 4.10 in my draft copy of Foundation, arrays are

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Tom Lane
Brendan Jurd dire...@gmail.com writes: On 5 April 2013 13:04, Tom Lane t...@sss.pgh.pa.us wrote: (There's been a remarkable lack of attention to the question of spec compliance in this thread, btw. Surely the standard has something to say on the matter of zero-length arrays?) The language

Re: [HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 8:30 PM Rodrigo Barboza wrote: Hi guys. I am wondering when I can use the PG_GETARG_UINT32 and PG_RETURN_UINT32. If postgres has no unsigned int type, what is the use of these macros? They are mainly used for contrib module functionality or some built-in functions

Re: [HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Rodrigo Barboza
I am creating my own uint32 type and faced this function. But my args are always of type my_int and one of the signed int types as postgres doesn't hava unsigned. Could I use those functions in operations between those types? I can't see a place for this, I don't know if I am missing something

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Amit Kapila
On Friday, April 05, 2013 1:59 AM Robert Haas wrote: On Thu, Apr 4, 2013 at 2:53 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: for estimate_worstcase_fraction. So, when computing the cost of a path, we'd compute our current expected-case

Re: [PATCH] Exorcise zero-dimensional arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Brendan Jurd
On 5 April 2013 15:05, Tom Lane t...@sss.pgh.pa.us wrote: Brendan Jurd dire...@gmail.com writes: While I was in there I noticed CARDINALITY, which would be pretty easy to add and would at least provide a more productive way to get the real length of an array without disrupting existing

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 6:12 PM Robert Haas wrote: On Wed, Apr 3, 2013 at 9:49 PM, Greg Smith g...@2ndquadrant.com wrote: On 4/2/13 11:54 AM, Robert Haas wrote: But, having said that, I still think the best idea is what Andres proposed, which pretty much matches my own thoughts: the

Re: [HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Amit Kapila
On Friday, April 05, 2013 10:00 AM Rodrigo Barboza wrote: I am creating my own uint32 type and faced this function. But my args are always of type my_int and one of the signed int types as postgres doesn't hava unsigned. Could I use those functions in operations between those types? It should