Re: [HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Amit Kapila
On Friday, April 05, 2013 10:00 AM Rodrigo Barboza wrote: > I am creating my own uint32 type and faced this function. > But my args are always of type my_int and one of the signed int types as postgres doesn't hava unsigned. > Could I use those functions in operations between those types? It shou

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 6:12 PM Robert Haas wrote: > On Wed, Apr 3, 2013 at 9:49 PM, Greg Smith > wrote: > > On 4/2/13 11:54 AM, Robert Haas wrote: > >> But, having said that, I still think the best idea is what Andres > >> proposed, which pretty much matches my own thoughts: the bgwriter > >>

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Brendan Jurd
On 5 April 2013 15:05, Tom Lane wrote: > Brendan Jurd writes: >> While I was in there I noticed CARDINALITY, which would be pretty easy >> to add and would at least provide a more productive way to get the >> "real" length of an array without disrupting existing functionality: > > Yeah, that woul

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Amit Kapila
On Friday, April 05, 2013 1:59 AM Robert Haas wrote: > On Thu, Apr 4, 2013 at 2:53 PM, Dimitri Fontaine > wrote: > > Robert Haas writes: > >> for estimate_worstcase_fraction. So, when computing the cost of a > >> path, we'd compute our current expected-case estimate, and also a > >> worst-case e

Re: [HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Rodrigo Barboza
I am creating my own uint32 type and faced this function. But my args are always of type my_int and one of the signed int types as postgres doesn't hava unsigned. Could I use those functions in operations between those types? I can't see a place for this, I don't know if I am missing something

Re: [HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 8:30 PM Rodrigo Barboza wrote: > Hi guys. > I am wondering when I can use the PG_GETARG_UINT32 and PG_RETURN_UINT32. > If postgres has no unsigned int type, what is the use of these macros? They are mainly used for contrib module functionality or some built-in function

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Tom Lane
Brendan Jurd writes: > On 5 April 2013 13:04, Tom Lane wrote: >> (There's been a remarkable lack of attention to the question >> of spec compliance in this thread, btw. Surely the standard has >> something to say on the matter of zero-length arrays?) > The language specifically allows for zero

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Brendan Jurd
On 5 April 2013 13:04, Tom Lane wrote: > (There's been a remarkable lack of attention to the question > of spec compliance in this thread, btw. Surely the standard has > something to say on the matter of zero-length arrays?) From 4.10 in my draft copy of "Foundation", arrays are one of two "coll

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Tom Lane
Brendan Jurd writes: > The other suggestion that had been tossed around elsewhere upthread > was inventing a new type that serves the demand for a straightforward > mutable list, which has exactly one dimension, and which may be > sensibly empty. Those few who are interested in dimensions >= 2 co

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
On Thu, 2013-04-04 at 21:06 -0400, Tom Lane wrote: > I can't escape the feeling that we'd just be reinventing software RAID. > There's no reason to think that we can deal with this class of problems > better than the storage system can. The goal would be to reliably detect a situation where WAL th

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Brendan Jurd
On 5 April 2013 07:43, Tom Lane wrote: > Well, if we're going to take that hard a line on it, then we can't > change anything about array data storage or the existing functions' > behavior; which leaves us with either doing nothing at all, or > inventing new functions that have saner behavior whil

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Tom Lane
Jeff Davis writes: > On Thu, 2013-04-04 at 14:21 -0700, Jeff Janes wrote: >> This brings up a pretty frightening possibility to me, unrelated to >> data checksums. If a bit gets twiddled in the WAL file due to a >> hardware issue or a "cosmic ray", and then a crash happens, automatic >> recovery

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
On Thu, 2013-04-04 at 22:39 +0200, Andres Freund wrote: > I don't think its really slower. Earlier the code took WalInsertLock > everytime, even if we ended up not logging anything. Thats far more > epensive than a single spinlock. And the copy should also only be taken > in the case we need to log

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
On Thu, 2013-04-04 at 14:21 -0700, Jeff Janes wrote: > This brings up a pretty frightening possibility to me, unrelated to > data checksums. If a bit gets twiddled in the WAL file due to a > hardware issue or a "cosmic ray", and then a crash happens, automatic > recovery will stop early with the

Re: [HACKERS] patch to add \watch to psql

2013-04-04 Thread Will Leinweber
On Thu, Apr 4, 2013 at 5:04 PM, Tom Lane wrote: > > I whacked this around some more, added basic docs, and committed it. > > Thanks! > > Unfortunately rl_clear_screen() is not included at all in libedit, > causing > > compilation to fail, and I was completely unable to find a way to > > distin

Re: [HACKERS] patch to add \watch to psql

2013-04-04 Thread Tom Lane
Will Leinweber writes: > Here is an updated patch that addresses several of the points brought up so > far, such as the sleep, internationalization banner, and zero wait check, > and it removes the premature input check. I whacked this around some more, added basic docs, and committed it. > Unfo

Re: [HACKERS] matview scannability rehash (was Re: Drastic performance loss in assert-enabled build in HEAD)

2013-04-04 Thread Alvaro Herrera
Tom Lane escribió: > Noah Misch writes: > > A slight variation on the committed approach would be to add a "_scannable" > > relation fork. > > Not very transaction-safe, I think (consider crash midway through a > transaction that adds or removes the fork), and in any case orders of > magnitude m

Re: [HACKERS] matview scannability rehash (was Re: Drastic performance loss in assert-enabled build in HEAD)

2013-04-04 Thread Tom Lane
Noah Misch writes: > On Wed, Apr 03, 2013 at 05:49:18PM -0400, Tom Lane wrote: >> No. This is an absolute disaster. It's taking something we have always >> considered to be an irrelevant implementation detail and making it into >> user-visible DDL state, despite the fact that it doesn't begin to

Re: [HACKERS] Drastic performance loss in assert-enabled build in HEAD

2013-04-04 Thread Noah Misch
On Thu, Apr 04, 2013 at 12:28:01PM +0200, Nicolas Barbier wrote: > 2013/4/3 Tom Lane : > > And if you're absolutely convinced that unlogged matviews mustn't work as I > > suggest, we can lose those from 9.3, too. > > +1. Having unlogged matviews without having incremental updates yet, > isn't supe

[HACKERS] matview scannability rehash (was Re: Drastic performance loss in assert-enabled build in HEAD)

2013-04-04 Thread Noah Misch
Subject updated to account for the wider topics now appearing. On Wed, Apr 03, 2013 at 05:49:18PM -0400, Tom Lane wrote: > What I'd actually rather see us spending time on right now is making > some provision for incremental updates, which I will boldly propose > could be supported by user-written

Re: [HACKERS] Clang compiler warning on 9.3 HEAD

2013-04-04 Thread Tom Lane
Alvaro Herrera writes: > Now, it annoys me that we now have three places that know about object > types supported by event triggers: there's a large struct of command tag > substrings (event_trigger_support), then there's these two functions. > It might be better to add ObjectType and ObjectClass

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Janes
On Thu, Apr 4, 2013 at 5:30 AM, Simon Riggs wrote: > On 4 April 2013 02:39, Andres Freund wrote: > > > If by now the first backend has proceeded to PageSetLSN() we are writing > > different data to disk than the one we computed the checksum of > > before. Boom. > > Right, so nothing else we were

Re: [HACKERS] CREATE EXTENSION BLOCKS

2013-04-04 Thread Tom Lane
"David E. Wheeler" writes: > On Apr 3, 2013, at 11:41 AM, Alvaro Herrera wrote: >> Right -- an extension is not considered to live within a schema, they >> are database-global. The objects might live in a particular schema (if >> it is "relocatable"), and there's support to move those to a diffe

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread k...@rice.edu
On Thu, Apr 04, 2013 at 04:16:12PM -0400, Stephen Frost wrote: > * Stephen Frost (sfr...@snowman.net) wrote: > > It does look like reducing bucket depth, as I outlined before through > > the use of a 2-level hashing system, might help speed up > > ExecScanHashBucket, as it would hopefully have very

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Tom Lane
Robert Haas writes: > On Thu, Apr 4, 2013 at 11:10 AM, Merlin Moncure wrote: >> The only reasonable answer for this (a provably used, non-security, >> non-standards violating, non-gross functionality breakage case) is >> *zero*. > +1. Well, if we're going to take that hard a line on it, then we

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Andres Freund
On 2013-04-04 12:59:36 -0700, Jeff Davis wrote: > Andres, > > Thank you for diagnosing this problem! > > On Thu, 2013-04-04 at 16:53 +0200, Andres Freund wrote: > > I think the route you quickly sketched is more realistic. That would > > remove all knowledge obout XLOG_HINT from generic code hich

Re: [HACKERS] Clang compiler warning on 9.3 HEAD

2013-04-04 Thread Alvaro Herrera
Will Leinweber wrote: > On ref 8507907 when compiling with clang on os x, I got this warning which > seems like a possible bug. > > I thought to report this because I imagine clang isn't frequently used > day-to-day by most. Ugh. My fault. Yes, this is a bug. I don't see any nice way to conver

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Robert Haas
On Thu, Apr 4, 2013 at 11:10 AM, Merlin Moncure wrote: > The only reasonable answer for this (a provably used, non-security, > non-standards violating, non-gross functionality breakage case) is > *zero*. +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Compa

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Tom Lane
Stephen Frost writes: > Looking with opannotate, there's two main hotspots in > ExecScanHashBucket: > 12846 17.4001 :hashTuple = > hashtable->buckets[hjstate->hj_CurBucketNo]; > and > 22100 29.9348 :hashTuple = hashTuple->next; Those are, of course, pretty trivial statements;

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Robert Haas
On Thu, Apr 4, 2013 at 2:53 PM, Dimitri Fontaine wrote: > Robert Haas writes: >> for estimate_worstcase_fraction. So, when computing the cost of a >> path, we'd compute our current expected-case estimate, and also a >> worst-case estimate, and then compute the final cost as: > > There also was t

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: > It does look like reducing bucket depth, as I outlined before through > the use of a 2-level hashing system, might help speed up > ExecScanHashBucket, as it would hopefully have very few (eg: 1-2) > entries to consider instead of more. Along those same

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
Andres, Thank you for diagnosing this problem! On Thu, 2013-04-04 at 16:53 +0200, Andres Freund wrote: > I think the route you quickly sketched is more realistic. That would > remove all knowledge obout XLOG_HINT from generic code hich is a very > good thing, I spent like 15minutes yesterday wond

Re: [HACKERS] puzzling JSON bug

2013-04-04 Thread Andrew Dunstan
On 04/04/2013 03:39 PM, Tom Lane wrote: Andrew Dunstan writes: David Wheeler has presented me with a nasty bug case. If I do this: select '{"members": { "add": [3, 4]}}'::json #> '{members,add}'; then I get a crash. If I comment out the pfree() at json.c:parse_object_field() lines 378-9 t

Re: [HACKERS] puzzling JSON bug

2013-04-04 Thread Tom Lane
Andrew Dunstan writes: > David Wheeler has presented me with a nasty bug case. > If I do this: > select '{"members": { "add": [3, 4]}}'::json #> '{members,add}'; > then I get a crash. > If I comment out the pfree() at json.c:parse_object_field() lines 378-9 > then I get back the right resu

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: > 8504125241524.3624 72.3115mcount.c:60 > libc-2.15.so __mcount_internal > 28370280785 8.1274 80.4389_mcount.S:33 > libc-2.15.so mcount [...] And as a side-note, I

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > perf or oprofile reveal anything? Here's what we get from oprofile (perhaps not too surprising): Hash the small table / scan the big table: samples cum. samples %cum. % linenr info image name symbol name 167374

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Dimitri Fontaine
Robert Haas writes: > for estimate_worstcase_fraction. So, when computing the cost of a > path, we'd compute our current expected-case estimate, and also a > worst-case estimate, and then compute the final cost as: There also was the idea for the executor to be able to handle alternate plans and

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > > What I'm trying to get at in this overall email is: why in the world is > > it so expensive to do hash lookups? > > perf or oprofile reveal anything? Working on a test case actually- I've got one now: http://snowman.net/~sfrost/test_case2.sql In this ex

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Tom Lane
Stephen Frost writes: > I've been fiddling with this on the very much larger overall database > where this test case came from and have found that hashing the large > table can actually be *faster* and appears to cause a more consistent > and constant amount of disk i/o (which is good). Interesti

Re: [HACKERS] pg_dump selectively ignores extension configuration tables

2013-04-04 Thread Vibhor Kumar
On Mar 25, 2013, at 12:01 PM, Vibhor Kumar wrote: > > On Mar 25, 2013, at 10:48 AM, Joe Conway wrote: > >> On 03/25/2013 08:12 AM, Vibhor Kumar wrote: >>> Since, nobody has picked this one. >>> >>> If there is no objection,then I can test this patch against 9.1 & 9.2. >> >> Here are diffs

Re: [HACKERS] Drastic performance loss in assert-enabled build in HEAD

2013-04-04 Thread Kevin Grittner
Early versions of the matview patch had a relisvalid flag in pg_class to determine whether the relation was scannable.  The name was chosen based on a similarity to the purpose of indisvalid, although the proliferation of new bools for related issues left me wondering if a "char" would be a better

Re: [HACKERS] Hash Join cost estimates

2013-04-04 Thread Stephen Frost
Tom, all, * Tom Lane (t...@sss.pgh.pa.us) wrote: > So that's at least going in the right direction. I agree that this is going in the right direction; it certainly would make the plan that I *expect* to be chosen more likely, however.. I've been fiddling with this on the very much larger overall

[HACKERS] puzzling JSON bug

2013-04-04 Thread Andrew Dunstan
David Wheeler has presented me with a nasty bug case. If I do this: select '{"members": { "add": [3, 4]}}'::json #> '{members,add}'; then I get a crash. If I comment out the pfree() at json.c:parse_object_field() lines 378-9 then I get back the right result but instead get a warning like t

Re: [HACKERS] Regex with > 32k different chars causes a backend crash

2013-04-04 Thread Heikki Linnakangas
On 04.04.2013 03:32, Noah Misch wrote: On Wed, Apr 03, 2013 at 08:09:15PM +0300, Heikki Linnakangas wrote: --- a/src/include/regex/regguts.h +++ b/src/include/regex/regguts.h @@ -148,6 +148,7 @@ typedef short color; /* colors of characters */ typedef int pcolor;

Re: [HACKERS] psql crash fix

2013-04-04 Thread Bruce Momjian
On Tue, Apr 2, 2013 at 08:48:53PM -0400, Bruce Momjian wrote: > I found that psql will crash if given a PSQLRC value containing a tilde: > > $ PSQLRC="~/x" psql test > *** glibc detected *** psql: free(): invalid pointer: > 0x7fffb7c933ec *** > > This is on Debian Squeeze 6.0.7.

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Simon Riggs
On 4 April 2013 15:53, Andres Freund wrote: > Unfortunately I find that approach unacceptably ugly. Yeh. If we can confirm its a fix we can discuss a cleaner patch and that is much better. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Trai

Re: [HACKERS] CREATE EXTENSION BLOCKS

2013-04-04 Thread David E. Wheeler
On Apr 4, 2013, at 5:16 AM, Dimitri Fontaine wrote: > "David E. Wheeler" writes: >> +Note that only the extension objects will be placed into the named >> +schema; the extension itself is a database-global object. > > I think you're patching the right place, but I'm not sure abo

Re: [HACKERS] Multi-pass planner

2013-04-04 Thread Robert Haas
On Wed, Apr 3, 2013 at 9:40 PM, Greg Stark wrote: > I used to advocate a similar idea. But when questioned on list I tried to > work out the details and ran into some problem coming up with a concrete > plan. > > How do you compare a plan that you think has a 99% chance of running in 1ms > but a 1

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Merlin Moncure
On Wed, Apr 3, 2013 at 11:11 PM, Tom Lane wrote: > In any case, the whole exercise is pointless if we don't change the > visible behavior of array_dims et al. So I think the idea that this > would be without visible consequence is silly. What's up for argument > is just how much incompatibility

[HACKERS] Why there is a PG_GETARG_UINT32 and PG_RETURN_UINT32?

2013-04-04 Thread Rodrigo Barboza
Hi guys. I am wondering when I can use the PG_GETARG_UINT32 and PG_RETURN_UINT32. If postgres has no unsigned int type, what is the use of these macros?

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Andres Freund
On 2013-04-04 13:30:40 +0100, Simon Riggs wrote: > On 4 April 2013 02:39, Andres Freund wrote: > > > Ok, I think I see the bug. And I think its been introduced in the > > checkpoints patch. > > Well spotted. (I think you mean checksums patch). Heh, yes. I was slightly tired at that point ;) > >

[HACKERS] Number of spinlocks

2013-04-04 Thread Heikki Linnakangas
With --disable-spinlocks, we need to know the number of spinlocks in the system at startup, so that we can reserve enough semaphores to mimic the spinlocks. It's calculated in SpinLockSemas(): /* * Report number of semaphores needed to support spinlocks. */ int SpinlockSemas(void) {

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-04 Thread Robert Haas
On Wed, Apr 3, 2013 at 9:49 PM, Greg Smith wrote: > On 4/2/13 11:54 AM, Robert Haas wrote: >> But, having said that, I still think the best idea is what Andres >> proposed, which pretty much matches my own thoughts: the bgwriter >> needs to populate the free list, so that buffer allocations don't

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Simon Riggs
On 4 April 2013 02:39, Andres Freund wrote: > Ok, I think I see the bug. And I think its been introduced in the > checkpoints patch. Well spotted. (I think you mean checksums patch). > If by now the first backend has proceeded to PageSetLSN() we are writing > different data to disk than the one

Re: [HACKERS] CREATE EXTENSION BLOCKS

2013-04-04 Thread Dimitri Fontaine
Hi, I though we were more specific about an extension's object itself not living in a schema in our documentation, but I agree we still have room for progress here. "David E. Wheeler" writes: > +Note that only the extension objects will be placed into the named > +schema; the ext

Re: [HACKERS] Drastic performance loss in assert-enabled build in HEAD

2013-04-04 Thread Nicolas Barbier
2013/4/3 Tom Lane : > Kevin Grittner writes: > >> To be honest, I don't think I've personally seen a single use case >> for matviews where they could be used if you couldn't count on an >> error if attempting to use them without the contents reflecting a >> materialization of the associated query

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 7:19 AM Greg Smith wrote: > On 4/2/13 11:54 AM, Robert Haas wrote: > > But, having said that, I still think the best idea is what Andres > > proposed, which pretty much matches my own thoughts: the bgwriter > > needs to populate the free list, so that buffer allocations

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-04-04 Thread Amit Kapila
On Thursday, April 04, 2013 2:52 AM Robert Haas wrote: > On Wed, Apr 3, 2013 at 2:54 PM, Tom Lane wrote: > > Robert Haas writes: > >> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut > wrote: > >>> It's weird that SET LOCAL and SET SESSION actually *set* the value, > and > >>> the second key wo

Re: [PATCH] Exorcise "zero-dimensional" arrays (Was: Re: [HACKERS] Should array_length() Return NULL)

2013-04-04 Thread Boszormenyi Zoltan
2013-04-03 20:58 keltezéssel, Gavin Flower írta: On 04/04/13 05:36, David E. Wheeler wrote: On Apr 3, 2013, at 9:30 AM, Tom Lane wrote: Fortran ... Basic ... actually I'd have thought that zero was a minority position. Fashions change I guess. I say we turn the default lower bound up to 11.