Re: [HACKERS] CLOG extension

2012-05-04 Thread Simon Riggs
On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote:
 On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote:
 Your two paragraphs have roughly opposite arguments...

 Doing it every 32 pages would give you 30 seconds to complete the
 fsync, if you kicked it off when half way through the previous file -
 at current maximum rates. So there is utility in doing it in larger
 chunks.

 Maybe, but I'd like to try changing one thing at a time.  If we change
 too much at once, it's likely to be hard to figure out where the
 improvement is coming from.  Moving the task to a background process
 is one improvement; doing it in larger chunks is another.  Those
 deserve independent testing.

You gave a good argument why background pre-allocation wouldn't work
very well if we do it a page at a time. I believe you.

If we do it a file at a time, we can just write the file without
calling it in page by page into the SLRU, as long as we write the WAL
file for it first then we don't need to fsync either of them.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] index-only scans vs. Hot Standby, round two

2012-05-04 Thread Simon Riggs
On 2 May 2012 13:41, Robert Haas robertmh...@gmail.com wrote:

 So on further reflection I'm thinking it may be best just to stick
 with a hard conflict for now and see what feedback we get from beta
 testers.

Which is what I was expecting y'all to conclude once you'd looked at
the task in more detail.

And I'm happy with the concept of beta being a period where we listen
to feedback, not just bugs, and decide whether further refinements are
required.

What I think is possible is to alter the conflict as it hits the
backend. If a backend has enable_indexonly = off then it wouldn't be
affected by those conflicts anyhow. So if we simply record whether we
are using an index only plan then we'll know whether to ignore it or
abort. I think we can handle that by marking the snapshot at executor
startup time. Needs a few other pushups but not that hard.

The likelihood is that SQL that uses index only won't run for long
enough to be cancelled anyway.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Hannu Krosing
On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
 
 
 On Wed, May 2, 2012 at 2:29 AM, Hannu Krosing ha...@2ndquadrant.com
 wrote:
 
 
 
 I don't object to row_to_json() and array_to_json() functions
 being
 there as a convenience and as the two official functions
 guaranteed to
 return JSON text.
 
 
 So given that do we do anything about this now, or wait till 9.3?

Sorry for missing this mail, followed this only on list
I hope it is ok to CC this back to list



I'd like the json support in 9.2 updated as follows


Generic to_json(...) returning a JSON value
=

we should have a generic to_json(...) both for eas and use and for easy
extensibility, as explained below.

to_json(...) should work for all types, returning a json value similar
to what current json_in does, but for all types, not just cstring.

We could keep row_to_json() and array_to_json() as official json-text
returning functions 


Configurable and extensible to_json()
==

When working on structured types, always the first try for getting an
element-as-json should be running to_json(element) and only if this
fails fall back to current use text representation code.

this enables two important things

1) configurable json-coding of values not explicitly supported by
standard

You can read about an attempt to standardise json-date formart here

http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx .

By allowing developers just to define their own to_json(date) function
we give them the power do decide which one to use. And if we honour
search_path when looking up the to_json() functions, then they can even
choose to have different conventions for different applications.

2) flexibility in adding support for extension types, like representing
hstore as object/dict by just providing the to_json(hstore, ...)
functions in hstore extension

Pretty-printing
===

If we were to support prettyprinting of anything more complex than
single level structs (record or array), then we need to pass ident
into the to_json() function

my recommendation would be to have the signature 

to_json(datum any, ident int)

with ident = NULL meaning no prettyprint , ident =0 meaninf top level,
or starting at left margin and anything else meaning the amount of
spaces needed to be added to the beginning of all rows exept the first
one, for example the query 

hannu=# select array_to_json(array(select test from test limit 2),true);
array_to_json
-
 [{id:9,data:testdata,tstamp:2012-05-01 09:44:50.175189}, +
  {id:10,data:testdata,tstamp:2012-05-01 09:45:50.260276}]
(1 row)

could return this:

[{'data': 'testdata',
  'id': 9,
  'tstamp': '2012-05-01 09:44:50.175189'},
 {'data': 'testdata',
  'id': 10,
  'tstamp': '2012-05-01 09:45:50.260276'}]

if it would call to_json(row, 1) for getting each row prettyprinted with
ident 1


Getting a record _from_ json()
==

JSON support would be much more useful if we supported the function of
converting the other way as well, that is from json to record

The best model easy to support seems to be what Merlin suggested, that
is the populate_record(record,hstore) function and corresponding #=
operator (see: http://www.postgresql.org/docs/9.1/static/hstore.html)



The complete plan for 9.2 once more
===

What is needed to nicely round up a  simple and powerful json type is

1. the json TYPE

   an agreement tha json type represents a JSON value (this is what 
   json_in currently does)


2. json output

2.1 doing the right thing with internal values

   to_json() function for converting to this JSON value for any type.
   default implementation for 'non-json' types returns their 
   postgresql textual representation in double quotes 
(date - 2012-05-01 09:45:50.260276

   structured types use to_json() for getting values internally, 
   so that by defining to_json(hstore) you can automatically get 
   hstore represented in javascript object or dictionary representation

   hannu=# select row_to_json(r) 
   from (select 1::int id, 'foo=1, bar=baz'::hstore)r;

   should not return this:

row_to_json  
   --
{id:1,hstore:\bar\=\baz\, \foo\=\1\}
   (1 row)

   but this
row_to_json  
   --
{id:1,hstore:{bar: baz, foo:1}}
   (1 row)

2.1 getting the pretty-printing right for structured types 

   to_json(any, indent) functions for getting recursive prettyprinting.
   we might also need another argument telling the page width we want
   to pretty print to.


3. json input 

Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Hannu Krosing
On Thu, 2012-05-03 at 00:58 -0500, Jim Nasby wrote:
 On 4/29/12 6:03 AM, Simon Riggs wrote:
  The DML-WITH-LIMIT-1 is required to do single logical updates on tables
with non-unique rows.
And as for any logical updates we will have huge performance problem
when doing UPDATE or DELETE on large table with no indexes, but
fortunately this problem is on slave, not master;)
  While that is possible, I would favour the do-nothing approach. By
  making the default replication mode = none, we then require a PK to be
  assigned before allowing replication mode = on for a table. Trying to
  replicate tables without PKs is a problem that can wait basically.
 
 
 Something that a in-core method might be able to do that an external one 
 can't would be to support a method of uniquely identifying rows in tables
 with no PK's. A gross example (that undoubtedly wouldn't work in the real 
 world) would be using TID's. 
 A real-world implementation might be based on a hidden serial column.

For logical we don't really need to uniquely identify such rows - it
should sufficient if we just update exactly one of the matching rows.

The way to do this is to put all fields of the OLD.* tuple in the WHERE
clause and then update just one matching row.

IIRC updating (or deleting) CURRENT OF a cursor is currently supported
only in pl/pgsql so this needs to be done using a plpgsql cursor.

If the table has no indexes or index lookup returns lots of rows, then
this is bound to be slow, but in this case it was probably slow on
master too :)

 -- 
 Jim C. Nasby, Database Architect   j...@nasby.net
 512.569.9461 (cell) http://jim.nasby.net
 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] CLOG extension

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 3:35 AM, Simon Riggs si...@2ndquadrant.com wrote:
 On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote:
 On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote:
 Your two paragraphs have roughly opposite arguments...

 Doing it every 32 pages would give you 30 seconds to complete the
 fsync, if you kicked it off when half way through the previous file -
 at current maximum rates. So there is utility in doing it in larger
 chunks.

 Maybe, but I'd like to try changing one thing at a time.  If we change
 too much at once, it's likely to be hard to figure out where the
 improvement is coming from.  Moving the task to a background process
 is one improvement; doing it in larger chunks is another.  Those
 deserve independent testing.

 You gave a good argument why background pre-allocation wouldn't work
 very well if we do it a page at a time. I believe you.

Your confidence is sort of gratifying, but in this case I believe it's
misplaced.  On more careful analysis, it seems that ExtendCLOG() does
just two things: (1) evict a CLOG buffer and replace it with a zero'd
page representing the new page and (2)  write an XLOG record for the
change.  Apparently, extending CLOG doesn't actually involve
extending anything on disk at all.  We rely on the future buffer
eviction to do that, which is surprisingly different from the way
relation extension is handled.

So CLOG extension is normally fast, but occasionally something goes
wrong.  So far I see two ways that can happen: (1) the WAL insertion
stalls because wal_buffers are full, and we're forced to wait for WAL
to be written (and perhaps fsync'd, since both are covered by the same
lock) or (2) the page we choose to evict happens to be dirty, and we
have to write+fsync it before repurposing it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 8:32 AM, Hannu Krosing ha...@krosing.net wrote:
 For logical we don't really need to uniquely identify such rows - it
 should sufficient if we just update exactly one of the matching rows.

 The way to do this is to put all fields of the OLD.* tuple in the WHERE
 clause and then update just one matching row.

 IIRC updating (or deleting) CURRENT OF a cursor is currently supported
 only in pl/pgsql so this needs to be done using a plpgsql cursor.

 If the table has no indexes or index lookup returns lots of rows, then
 this is bound to be slow, but in this case it was probably slow on
 master too :)

I was about to write a reply saying exactly this, but you said it
better than I would have been able to manage.

I think this is all exactly right.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Robert Haas
On Thu, May 3, 2012 at 8:22 PM, Greg Smith g...@2ndquadrant.com wrote:
 On 05/01/2012 09:09 AM, Robert Haas wrote:

 I think we ought to be sharing and debugging designs in
 public, not internally within 2ndQuadrant - or any other company, or
 any other mailing list other than this one.

 OK.  You go first.

Are you requesting more transparency in general, asking for my
thoughts on logical replication specifically, or something else?

I try pretty hard not to go off and do large amounts of work in a
vacuum.  If something is more than a couple days work, I post the
design on hackers and wait for feedback before writing a line of code.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] CLOG extension

2012-05-04 Thread Simon Riggs
On 4 May 2012 13:59, Robert Haas robertmh...@gmail.com wrote:
 On Fri, May 4, 2012 at 3:35 AM, Simon Riggs si...@2ndquadrant.com wrote:
 On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote:
 On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote:
 Your two paragraphs have roughly opposite arguments...

 Doing it every 32 pages would give you 30 seconds to complete the
 fsync, if you kicked it off when half way through the previous file -
 at current maximum rates. So there is utility in doing it in larger
 chunks.

 Maybe, but I'd like to try changing one thing at a time.  If we change
 too much at once, it's likely to be hard to figure out where the
 improvement is coming from.  Moving the task to a background process
 is one improvement; doing it in larger chunks is another.  Those
 deserve independent testing.

 You gave a good argument why background pre-allocation wouldn't work
 very well if we do it a page at a time. I believe you.

 Your confidence is sort of gratifying, but in this case I believe it's
 misplaced.  On more careful analysis, it seems that ExtendCLOG() does
 just two things: (1) evict a CLOG buffer and replace it with a zero'd
 page representing the new page and (2)  write an XLOG record for the
 change.  Apparently, extending CLOG doesn't actually involve
 extending anything on disk at all.  We rely on the future buffer
 eviction to do that, which is surprisingly different from the way
 relation extension is handled.

 So CLOG extension is normally fast, but occasionally something goes
 wrong.

I don't agree its normally fast.

WALInsert contention is high, so there is usually a long queue. As
we've discussed this can be done offline and and so (2) can completely
avoided in the main line. Considering that all new xids wait for this
action, any wait at all is bad and takes time to drain once it clears.

Evicting a clog has cost because the tail is almost always dirty when
we switch pages.

Doing both of those will ensure switch to new page requires zero wait time.

So you have the solution. Not sure what else you're looking for.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Simon Riggs
On 4 May 2012 14:01, Robert Haas robertmh...@gmail.com wrote:
 On Fri, May 4, 2012 at 8:32 AM, Hannu Krosing ha...@krosing.net wrote:
 For logical we don't really need to uniquely identify such rows - it
 should sufficient if we just update exactly one of the matching rows.

 The way to do this is to put all fields of the OLD.* tuple in the WHERE
 clause and then update just one matching row.

 IIRC updating (or deleting) CURRENT OF a cursor is currently supported
 only in pl/pgsql so this needs to be done using a plpgsql cursor.

 If the table has no indexes or index lookup returns lots of rows, then
 this is bound to be slow, but in this case it was probably slow on
 master too :)

 I was about to write a reply saying exactly this, but you said it
 better than I would have been able to manage.

 I think this is all exactly right.

Yes, but its not a high priority for inclusion. Many things like this
will need to wait behind the really critical additional features.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Advisory locks seem rather broken

2012-05-04 Thread Robert Haas
On Thu, May 3, 2012 at 5:18 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 ... btw, it appears to me that the fast path patch has broken things
 rather badly in LockReleaseAll.  AFAICS it's not honoring either the
 lockmethodid restriction nor the allLocks restriction with respect to
 fastpath locks.  Perhaps user locks and session locks are never taken
 fast path, but still it would be better to be making those checks
 further up, no?

User locks are never taken fast path, but session locks can be, so I
think you're right that there is a bug here.  I think what we should
probably do is put the nLocks == 0 test before the lockmethodid and
allLocks checks, and then the fast path stuff after those two checks.

In 9.1, we just did this:

if (locallock-proclock == NULL || locallock-lock == NULL)
{
/*
 * We must've run out of shared memory while
trying to set up this
 * lock.  Just forget the local entry.
 */
Assert(locallock-nLocks == 0);
RemoveLocalLock(locallock);
continue;
}

...and I just shoved the new logic into that stanza without thinking
hard enough about what order to do things in.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Advisory locks seem rather broken

2012-05-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 In 9.1, we just did this:

 if (locallock-proclock == NULL || locallock-lock == NULL)
 {
 /*
  * We must've run out of shared memory while
 trying to set up this
  * lock.  Just forget the local entry.
  */
 Assert(locallock-nLocks == 0);
 RemoveLocalLock(locallock);
 continue;
 }

 ...and I just shoved the new logic into that stanza without thinking
 hard enough about what order to do things in.

Right.  The other thing that was bothering me about that was that it's
not clear now how to tell a broken locallock entry (which is what this
logic originally intended to clean up) from a fastpath one.  Perhaps
it'd be a good idea to add a valid flag?  And while I'm wondering
about such things, what happens when it's necessary to convert a
fastpath lock to a regular one, but there's no room in shared memory
for more lock objects?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes:
 On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
 So given that do we do anything about this now, or wait till 9.3?

 I'd like the json support in 9.2 updated as follows

I think it's too late to be entertaining proposals for such changes in
9.2.  If we had concluded that the existing functions were actively
wrong or a bad idea, then of course we'd need to do something; but they
are not, so we can just as well consider additions in the 9.3 cycle
rather than now.  I am not convinced that this proposal is fully baked
yet, anyway; not to mention that right now we need to have our heads
down on resolving the remaining open issues, not designing,
implementing, and reviewing a pile of brand new code for json.

 By allowing developers just to define their own to_json(date) function
 we give them the power do decide which one to use. And if we honour
 search_path when looking up the to_json() functions, then they can even
 choose to have different conventions for different applications.

This is not going to work anywhere near as nicely as you think.  If
somebody tries to define multiple to_json() functions that override a
generic to_json(anyelement) one, he will start getting function is not
unique parse failures.  The parser will only successfully decide which
function to call when the input data type exactly matches one of the
specialized functions, which means you might as well not have the
generic one at all.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] CLOG extension

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 9:11 AM, Simon Riggs si...@2ndquadrant.com wrote:
 On 4 May 2012 13:59, Robert Haas robertmh...@gmail.com wrote:
 On Fri, May 4, 2012 at 3:35 AM, Simon Riggs si...@2ndquadrant.com wrote:
 On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote:
 On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote:
 Your two paragraphs have roughly opposite arguments...

 Doing it every 32 pages would give you 30 seconds to complete the
 fsync, if you kicked it off when half way through the previous file -
 at current maximum rates. So there is utility in doing it in larger
 chunks.

 Maybe, but I'd like to try changing one thing at a time.  If we change
 too much at once, it's likely to be hard to figure out where the
 improvement is coming from.  Moving the task to a background process
 is one improvement; doing it in larger chunks is another.  Those
 deserve independent testing.

 You gave a good argument why background pre-allocation wouldn't work
 very well if we do it a page at a time. I believe you.

 Your confidence is sort of gratifying, but in this case I believe it's
 misplaced.  On more careful analysis, it seems that ExtendCLOG() does
 just two things: (1) evict a CLOG buffer and replace it with a zero'd
 page representing the new page and (2)  write an XLOG record for the
 change.  Apparently, extending CLOG doesn't actually involve
 extending anything on disk at all.  We rely on the future buffer
 eviction to do that, which is surprisingly different from the way
 relation extension is handled.

 So CLOG extension is normally fast, but occasionally something goes
 wrong.

 I don't agree its normally fast.

 WALInsert contention is high, so there is usually a long queue. As
 we've discussed this can be done offline and and so (2) can completely
 avoided in the main line. Considering that all new xids wait for this
 action, any wait at all is bad and takes time to drain once it clears.

 Evicting a clog has cost because the tail is almost always dirty when
 we switch pages.

 Doing both of those will ensure switch to new page requires zero wait time.

 So you have the solution. Not sure what else you're looking for.

Nothing, really.  I was just mooting some ideas before I went and
started coding, to see what people thought.  I've got your opinion and
Tom's, and of course my own, so now I'm off to test some different
approaches.  At the moment I'm running a battery of tests on
background-writing CLOG, which I will post about when they are
complete, and I intend to play around with some of the ideas from this
thread as well.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Advisory locks seem rather broken

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 9:25 AM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 In 9.1, we just did this:

                 if (locallock-proclock == NULL || locallock-lock == NULL)
                 {
                         /*
                          * We must've run out of shared memory while
 trying to set up this
                          * lock.  Just forget the local entry.
                          */
                         Assert(locallock-nLocks == 0);
                         RemoveLocalLock(locallock);
                         continue;
                 }

 ...and I just shoved the new logic into that stanza without thinking
 hard enough about what order to do things in.

 Right.  The other thing that was bothering me about that was that it's
 not clear now how to tell a broken locallock entry (which is what this
 logic originally intended to clean up) from a fastpath one.  Perhaps
 it'd be a good idea to add a valid flag?

Well, I think nLocks == 0 should serve that purpose adequately.

 And while I'm wondering
 about such things, what happens when it's necessary to convert a
 fastpath lock to a regular one, but there's no room in shared memory
 for more lock objects?

Then you error out.  Of course, if the fast path mechanism didn't
exist at all, you would have started erroring out much sooner.  Now,
there is some rub here, because the mechanism isn't fair: strong
lockers will error out instead of weak lockers, and in the worst case
where the lock table remains perpetually on the edge of overflowing,
strong lock requests could be fail repeatedly, essentially a DOS.
Originally, I thought that the patch should include some kind of
accounting mechanism to prevent that from happening, where we'd keep
track of the number of fast-path locks that were outstanding and make
sure to keep that many slots free in the main lock table, but Noah
talked me out of it, on theory that (1) it was very unlikely to occur
in practice and (2) if it did occur, then you probably need to bump up
max_locks_per_transaction anyway and (3) it amounted to forcing
failures in cases where that might not be strictly necessary, which is
usually not a great thing to do.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Advisory locks seem rather broken

2012-05-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 Originally, I thought that the patch should include some kind of
 accounting mechanism to prevent that from happening, where we'd keep
 track of the number of fast-path locks that were outstanding and make
 sure to keep that many slots free in the main lock table, but Noah
 talked me out of it, on theory that (1) it was very unlikely to occur
 in practice and (2) if it did occur, then you probably need to bump up
 max_locks_per_transaction anyway and (3) it amounted to forcing
 failures in cases where that might not be strictly necessary, which is
 usually not a great thing to do.

I agree with that, as long as we can be sure that the system behaves
sanely (doesn't leave the data structures in a corrupt state) when an
out-of-memory condition does occur.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Advisory locks seem rather broken

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 Originally, I thought that the patch should include some kind of
 accounting mechanism to prevent that from happening, where we'd keep
 track of the number of fast-path locks that were outstanding and make
 sure to keep that many slots free in the main lock table, but Noah
 talked me out of it, on theory that (1) it was very unlikely to occur
 in practice and (2) if it did occur, then you probably need to bump up
 max_locks_per_transaction anyway and (3) it amounted to forcing
 failures in cases where that might not be strictly necessary, which is
 usually not a great thing to do.

 I agree with that, as long as we can be sure that the system behaves
 sanely (doesn't leave the data structures in a corrupt state) when an
 out-of-memory condition does occur.

OK.  I believe that commit 53c5b869b464d567c3b8f617201b49a395f437ab
robustified this code path quite a bit; but it is certainly possible
that there are remaining oversights, and I would certainly appreciate
any further review you have time to do.  Basically, it seems like the
likely failure modes, if there are further bugs, would be either (1)
failing to track the strong lock counts properly, leading to
performance degradation if the counters become permanently stuck at a
value other than zero even after all the locks are gone or (2) somehow
muffing the migration of a lock from the fast-path mechanism to the
regular mechanism.

When taking a strong lock, the idea is that the strong locker first
bumps the strong lock count.  That bump must be unwound if we fail to
acquire the lock, which means it has to be cleaned up in the error
path and any case where we give up (e.g. conditional acquire of a
contended lock).  Next, we iterate through all the backend slots and
transfer fast path locks for each one individually.  If we fail midway
through, the strong locker must simply make sure to unwind the strong
lock count.  The weak lockers whose locks got transferred are fine:
they need to know how to cope with releasing a transferred lock
anyway; whether the backend that did the transfer subsequently blew up
is not something they have any need to care about.  Once all the
transfers are complete, the strong locker queues for the lock using
the main mechanism, which now includes all possible conflicting locks.
 Again, if we blow up while waiting for the lock, the only extra thing
we need to do is unwind the strong lock count acquisition.

Of course, I may be missing some other kind of bug altogether...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Greg Smith

On 05/04/2012 09:03 AM, Robert Haas wrote:

I try pretty hard not to go off and do large amounts of work in a
vacuum.  If something is more than a couple days work, I post the
design on hackers and wait for feedback before writing a line of code.


That is an excellent luxury to have.  You've worked very hard to earn 
it.  Not everyone is in the position where that's possible though.



Are you requesting more transparency in general, asking for my
thoughts on logical replication specifically, or something else?


The straw man argument here would require 100% transparency on 
everything you do in regards to PostgreSQL and related software.  Before 
doing any development on any code, first post here to ask for design 
review.  And if someone asks you to work on a program that isn't open 
source from day one, refuse unless you can operate that transparently.


That standard is nice if you can pull it off.  But I don't give you a 
hard time if you have to make some compromises from that ideal to keep 
yourself gainfully employed.  You do a ton of good work for the 
PostgreSQL community in a transparent way, so I assume that you're doing 
the best you can.  I would like to see that assumption presumed on our 
side, too.


Here are the individual straw men in this area I'd like to see put out 
of their misery:


You're developing things in secret:  if that's the case, we're pretty 
bad at it, given the history I outlined at 
http://archives.postgresql.org/message-id/4f9b1b6c.5010...@2ndquadrant.com


That discussion didn't happen in the right place:  it's not our fault 
that the cluster-hackers list exists.  Go joust at getting that list 
shut down and their meeting during PGCon canceled if you think it's 
unproductive for discussions to happen there.  I've been trying to 
bridge that gap for over two years now; note how many times I appear in 
the edit history at 
http://wiki.postgresql.org/index.php?title=ClusterFeaturesaction=history


You might do too much development in the wrong direction and not build 
the right thing:  and?  Yes, there are people who develop into a corner 
and end up doing unproductive work as a result.  And there are others 
who submit things and give up when faced with feedback on them.  Last 
time I checked, there wasn't anyone who flat-out rejects on-list 
feedback working for 2ndQuadrant.   Instead, I see features that go 
through extensive and numerous review cycles based on what we hear back.


Designs should be presented on-list before doing any development: 
this is not always practical for those of us who are doing feature 
development.  Some feature sponsors are still getting used to open 
development.  If we have a private development milestone date to hit *in 
order to get more funding for public PostgreSQL work*, which is often 
the case here, we try not to miss it.  We'd be bad community members to 
do so.  And sometimes that involves building a proof of concept or 
prototype here first, then submitting it to the community once it's 
moved onto being a proven concept.  Since the community has a clear set 
of guidelines for how and when to submit new features, we make sure the 
development plans line up with them.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 11:06 AM, Greg Smith g...@2ndquadrant.com wrote:
 The straw man argument here would require 100% transparency on everything
 you do in regards to PostgreSQL and related software.  Before doing any
 development on any code, first post here to ask for design review.  And if
 someone asks you to work on a program that isn't open source from day one,
 refuse unless you can operate that transparently.

Well, look.  At the end of the day, I don't really care whether you
post your designs before writing code or not - unless it turns out
that we get to the end of the development cycle, a gigantic patch
shows up at the last minute, it gets rejected because people aren't
satisfied with the design, and then massive bitching ensues because
the author(s) put a lot of work into that patch.  Then I care, because
now the fact that no design consensus was sought at the outset has
been transformed into a defect in the community process, which does in
fact have defects, but that isn't one of them.  We all know that
design review is going to have to happen at some point, and if there's
not an adequate opportunity to do that before the code is written then
it will happen after the code is written.  If that means the code has
to be thrown out, then that's the risk you take by writing the code
first.  As long as everybody understands that, do it in whatever order
you like.

I think the real straw man here is the idea that it will somehow save
time to skip the design phase and start writing code.  I have never
worked on a project, open source or otherwise, where that was true,
and I believe that any textbook on software engineering you pick up is
likely to tell you that in fact exactly the opposite is the case.

Obviously, there are times when you need to write some throw-away code
just to see how things shake out, and I do that all the time, and it
makes complete sense, and I'm not knocking it.  But if any of that
code makes it into the committed patch, I count that as unusually
lucky.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Andrew Dunstan



On 05/04/2012 09:52 AM, Tom Lane wrote:

Hannu Krosingha...@2ndquadrant.com  writes:

On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:

So given that do we do anything about this now, or wait till 9.3?

I'd like the json support in 9.2 updated as follows

I think it's too late to be entertaining proposals for such changes in
9.2.  If we had concluded that the existing functions were actively
wrong or a bad idea, then of course we'd need to do something; but they
are not, so we can just as well consider additions in the 9.3 cycle
rather than now.  I am not convinced that this proposal is fully baked
yet, anyway; not to mention that right now we need to have our heads
down on resolving the remaining open issues, not designing,
implementing, and reviewing a pile of brand new code for json.



Yeah, that was my feeling. We usually take a release or two to get 
things right, fill in what's missing, etc. and I don't think this will 
be ant different.






By allowing developers just to define their own to_json(date) function
we give them the power do decide which one to use. And if we honour
search_path when looking up the to_json() functions, then they can even
choose to have different conventions for different applications.

This is not going to work anywhere near as nicely as you think.  If
somebody tries to define multiple to_json() functions that override a
generic to_json(anyelement) one, he will start getting function is not
unique parse failures.  The parser will only successfully decide which
function to call when the input data type exactly matches one of the
specialized functions, which means you might as well not have the
generic one at all.




Yeah, what I've been thinking about in conjunction with similar problems 
is some sort of type registry, so that we could code for non-builtin 
types in certain cases. Maybe we should add that the the developers' 
meeting agenda.


cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Beta time?

2012-05-04 Thread Josh Berkus

 Next week, I thought.
 
 How are we handling the Monday release with everyone at PGCon?  Was that
 resolved?

I have yet to see a confirmed date, guys.  If we expect any support from
the packagers and/or the advocacy volunteers, then people need at least
a week's notice, probably more.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] c-function variants running time

2012-05-04 Thread Armando
Hi everybody.

First of all I have to thank you for your wonderful job! PostgreSQL rocks!

I am writing you because I am interested in understanding some specifics related
to PostgreSQL internals. More precisely, I am investigating the running time
of the different function implementation approaches, which is part of my BSc
thesis.

Here is the thing: I have implemented as a proof of concept three functions,
which are
a) text[] arraypoc(text, int); this returns an array of the form 'text,int'
   where the text is the copy of the text passed as parameter and int is a
   simple counter. The number of tuples is specified by the integer parameter.
   I have estimated the running time in this manner:
   SELECT *
 FROM unnest(arraypoc('abcdefghilmnopqrstuvz',100));
   The estimated running time is after 10 executions is:
   (791.571 + 797.163 + 677.331 + 686.674 + 686.691 + 686.438 +
797.910 + 795.955 + 793.459 + 794.110)/10 = 750.7302

b) TABLE(text,int) srfpoc(text, int); is similar as the previous one but this
   is a set returning function, which returns a table of a similar shape as in
   the previous case. Instead of a string, I return a text and an integer. Again
   text is just the copy of the parameter and int is a counter.
   I have estimated the running time in this manner:
   SELECT *
 FROM srfpoc('abcdefghilmnopqrstuvz',100);
   The estimated running time is after 10 executions is: 
   (665.016 + 778.100 + 640.605 + 787.102 + 785.501 + 791.307 +
784.780 + 793.222 + 794.624 + 790.357)/10 = 761.0614

c) TABLE(text,int) srfmatpoc(text, int); this does the same as the previous one,
   but in this case I wrote a SRF_Materialized using the SPI interface.  I have
   estimated the running time in this manner:
   SELECT *
 FROM srfmatpoc('abcdefghilmnopqrstuvz',100);
   The estimated running time is after 10 executions is: 
   (747.095 + 703.894 + 762.310 + 763.299 + 764.582 + 760.991 + 763.427 +
764.033 + 731.292 + 770.895)/10   = 753.1818

I have executed all the tests on the same server. The functions are compiled
using the -O3 compilation parameter. I am using PostgreSQL 9.1.3.

I would have expected the version a) to be slower than b) and c) but it turns
out that it is actually the fastest (the difference is not so big anyway). What
am I doing wrong? What can I do to improve the functions? Have I misunderstood
something?

Attached you find the code of all three functions.

Thanks a lot!
Armando
#include postgres.h
#include string.h

#include utils/array.h
#include utils/builtins.h	/* text_to_cstring */
#include catalog/pg_type.h
#include utils/lsyscache.h
#include fmgr.h

#ifdef PG_MODULE_MAGIC
PG_MODULE_MAGIC;
#endif

PG_FUNCTION_INFO_V1(arraypoc);

Datum arraypoc(PG_FUNCTION_ARGS) {
  ArrayType   *result;
  Datum   *result_data = NULL;
  int16o_typlen;
  bool o_typbyval;
  char o_typalign;
  text*t,
  *cpyt;
  uint32   c;
  int i, t_len;

  t = (text *)PG_GETARG_TEXT_PP(0);
  c = (uint32)PG_GETARG_INT32(1);

  if(c0) result_data = (Datum *)palloc(sizeof(Datum) * c);

  for(i=0; ic; ++i) {
t_len = VARSIZE_ANY_EXHDR(t);

cpyt = (text *)palloc(VARHDRSZ + t_len + 1 + 16);
SET_VARSIZE(cpyt, VARHDRSZ + t_len + 1 + 16);
memcpy((void *)VARDATA(cpyt), (void *)VARDATA(t), t_len);
memcpy((void *)VARDATA(cpyt) + t_len, ,, 1);
snprintf((void *)VARDATA(cpyt) + t_len + 1, 16, %d, i);

result_data[i] = PointerGetDatum(cpyt);
  }

  if(result_data == NULL || c = 0) {
PG_RETURN_NULL();

  } else {
get_typlenbyvalalign(TEXTOID, o_typlen, o_typbyval, o_typalign);
result = construct_array(result_data, c, TEXTOID, o_typlen, o_typbyval,
 o_typalign);
PG_RETURN_ARRAYTYPE_P(result);
  }
}
#include postgres.h

#include stdio.h/* snprintf */
#include stdlib.h/* malloc, free */
#include string.h/* strncpy, strlen */
#include sys/types.h  /* for uint32 and size_t*/

#include catalog/pg_type.h
#include utils/builtins.h  /* text_to_cstring */
#include funcapi.h
#include fmgr.h

#ifdef PG_MODULE_MAGIC
PG_MODULE_MAGIC;
#endif

PG_FUNCTION_INFO_V1(srfpoc);

Datum srfpoc(PG_FUNCTION_ARGS) {
  FuncCallContext *funcctx;
  TupleDesctupdesc;
  text*t;
  int  max_calls,
   call_cntr;

  t = (text *)PG_GETARG_TEXT_PP(0);
  
  if (SRF_IS_FIRSTCALL()) {
MemoryContextoldcontext;

funcctx = SRF_FIRSTCALL_INIT();
oldcontext = MemoryContextSwitchTo(funcctx-multi_call_memory_ctx);

funcctx-max_calls = (uint32)PG_GETARG_INT32(1);

tupdesc = CreateTemplateTupleDesc(2, false);
TupleDescInitEntry(tupdesc, (AttrNumber) 1, t, TEXTOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2, count, INT4OID, -1, 0);

funcctx-user_fctx = BlessTupleDesc(tupdesc);

MemoryContextSwitchTo(oldcontext);
  }

  funcctx = 

Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes:
 Yeah, what I've been thinking about in conjunction with similar problems 
 is some sort of type registry, so that we could code for non-builtin 
 types in certain cases. Maybe we should add that the the developers' 
 meeting agenda.

Maybe.  I don't want to see a json-specific hack for this, but some sort
of generic way to add type knowledge could be useful, if we could figure
out what we want.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Beta time?

2012-05-04 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes:
 How are we handling the Monday release with everyone at PGCon?  Was that
 resolved?

 I have yet to see a confirmed date, guys.  If we expect any support from
 the packagers and/or the advocacy volunteers, then people need at least
 a week's notice, probably more.

I haven't seen anybody positively say we can't do a wrap next Thursday
and release Monday, so I've been assuming that's what will happen.
If there are reasons to think it won't work, let's hear 'em now.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Andres Freund
Hi Robert, Hi all,

On Friday, May 04, 2012 06:29:33 PM Robert Haas wrote:
 On Fri, May 4, 2012 at 11:06 AM, Greg Smith g...@2ndquadrant.com wrote:
  The straw man argument here would require 100% transparency on everything
  you do in regards to PostgreSQL and related software.  Before doing any
  development on any code, first post here to ask for design review.  And
  if someone asks you to work on a program that isn't open source from day
  one, refuse unless you can operate that transparently.
 
 Well, look.  At the end of the day, I don't really care whether you
 post your designs before writing code or not - unless it turns out
 that we get to the end of the development cycle, a gigantic patch
 shows up at the last minute, it gets rejected because people aren't
 satisfied with the design, and then massive bitching ensues because
 the author(s) put a lot of work into that patch.  Then I care, because
 now the fact that no design consensus was sought at the outset has
 been transformed into a defect in the community process, which does in
 fact have defects, but that isn't one of them.  We all know that
 design review is going to have to happen at some point, and if there's
 not an adequate opportunity to do that before the code is written then
 it will happen after the code is written.  If that means the code has
 to be thrown out, then that's the risk you take by writing the code
 first.  As long as everybody understands that, do it in whatever order
 you like.
In my understanding - as the person doing quite a bit of the coding atm - the 
point is to provide a very minimal *early* prototype to have a sensible basis 
for design decisions/discussions. On one side thats useful to get a feeling 
for the problems involved. On the other side doing design discussions without 
an underlaying basic patch  design on -hackers tends to often go into 
directions of feature creep and bikeshedding. It also helps against this is 
impossible claims.
Parts of this thread and related ones are a somewhat good example of this.

The plan is to show the early prototype around pgcon and send design documents 
and split-up patches (of that prototype) a holiday and some cleanup later to -
hackers. I/We aim to have individual, independently usable, parts of the patch 
submitted to the first 9.3 commitfest.

I definitely do not wish to screw anyone over doing this or such. And I am 
sure thats the same with others working on this. Even if sometimes emotions 
get in the way/into play.

Greetings,

Andres
-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 12:59 PM, Andres Freund and...@2ndquadrant.com wrote:
 In my understanding - as the person doing quite a bit of the coding atm - the
 point is to provide a very minimal *early* prototype to have a sensible basis
 for design decisions/discussions. On one side thats useful to get a feeling
 for the problems involved. On the other side doing design discussions without
 an underlaying basic patch  design on -hackers tends to often go into
 directions of feature creep and bikeshedding. It also helps against this is
 impossible claims.
 Parts of this thread and related ones are a somewhat good example of this.

 The plan is to show the early prototype around pgcon and send design documents
 and split-up patches (of that prototype) a holiday and some cleanup later to -
 hackers. I/We aim to have individual, independently usable, parts of the patch
 submitted to the first 9.3 commitfest.

 I definitely do not wish to screw anyone over doing this or such. And I am
 sure thats the same with others working on this. Even if sometimes emotions
 get in the way/into play.

Thanks.  I think we are basically all on the same page, here.  As I
said upthread, I am not trying to accuse anything of doing anything
wrong, and as you and Simon and Greg have all said, you're not trying
to do anything wrong.  So, no problem.  :-)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 12:56 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Andrew Dunstan and...@dunslane.net writes:
 Yeah, what I've been thinking about in conjunction with similar problems
 is some sort of type registry, so that we could code for non-builtin
 types in certain cases. Maybe we should add that the the developers'
 meeting agenda.

 Maybe.  I don't want to see a json-specific hack for this, but some sort
 of generic way to add type knowledge could be useful, if we could figure
 out what we want.

For this particular case, I think you just need some place to store a
pg_type - pg_proc mapping.  I'm not exactly sure how to make that not
a JSON-specific hack, since I certainly don't think we'd want to add a
new catalog just for that.

In general, I think it would be very useful to have some way of
identifying particular types - and versions of types - independently
of a particular installation - e.g. by assigning each type a UUID that
never changes and a version number that we bump when we change
something about that type.  That seems like it would be very useful
for schema comparison tools, or for logical replication, where you
want to know whether two types are the same type even though they
are in different clusters.  pg_upgrade has had past needs in this area
as well.  However, I'm not sure that'd help solve this particular
problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] remove dead ports?

2012-05-04 Thread Peter Eisentraut
On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote:
 On 3 May 2012 17:21, Bruce Momjian br...@momjian.us wrote:
  I think I was the only user left;  I have never heard from a BSD/OS user
  in the past 5-7 years.
 
 I'm inclined to agree with Bruce. While it's not reasonable to assume
 that the lack of a BSD/OS user complaining on -general indicates that
 there are none, it's also not reasonable for them to expect us to
 support their operating system for 8 years after the original
 proprietary vendor. Better to not support BSD/OS than to supply a port
 that no one really has any confidence in. It's not as if we've ceased
 support in release branches.

I'm not so much opposed to removing the port.  I am more concerned about
the manner in which it was done.  The other ports I removed were known
to not work anyway, for years, and there were at least several days of
discussion.  The bsdi case was removing a working port with less than 24
hours notice.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-05-04 Thread Peter Eisentraut
On tor, 2012-05-03 at 15:47 -0400, Bruce Momjian wrote:
 Peter, where are we on this?

I hadn't received any clear feedback, but if no one objects, I can
commit it.

 ---
 
 On Fri, Mar 30, 2012 at 08:16:59PM +0300, Peter Eisentraut wrote:
  On fre, 2012-03-23 at 07:52 -0700, David Fetter wrote:
   On Thu, Mar 22, 2012 at 06:05:30PM -0400, Andrew Dunstan wrote:
On 03/22/2012 05:49 PM, Bruce Momjian wrote:
Robert Haas and I are disappointed by this change.  I liked the
fact that I could post nice-looking SQL queries without having to
use my capslock key (which I use as a second control key).  Any
chance of reverting this change?


Should it be governed by a setting?
   
   Something like (upper|lower|preserve) ?
  
  How about this patch then?  (There are actually four possible settings,
  see patch.)



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] PL/Python result set slicing broken in Python 3

2012-05-04 Thread Jan Urbański

On 03/05/12 11:04, Jan Urbański wrote:

On 02/05/12 20:18, Peter Eisentraut wrote:

This doesn't work anymore with Python 3:

rv = plpy.execute(...)
do_something(rv[0:1])

Apparently, they changed the C API for doing slicing, or rather made one
of the two APIs for it silently do nothing. Details are difficult to
find, but this email message seems to contain something:
http://mail.python.org/pipermail/python-3000/2007-August/009851.html.

I'll try to sort this out sometime, but if someone wants to take a shot
at it, go ahead.


Sounds ugly. I'll take a look.


I found some instructions on how to deal with the Python 2/Python 3 
slicing mess:


http://renesd.blogspot.com/2009/07/python3-c-api-simple-slicing-sqslice.html

Apparently you need that egregious hack in order to avoid code 
duplication. I'll try to produce a patch over the weekend.


Cheers,
Jan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Hannu Krosing
On Fri, 2012-05-04 at 13:43 -0400, Robert Haas wrote:
 On Fri, May 4, 2012 at 12:56 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  Andrew Dunstan and...@dunslane.net writes:
  Yeah, what I've been thinking about in conjunction with similar problems
  is some sort of type registry, so that we could code for non-builtin
  types in certain cases. Maybe we should add that the the developers'
  meeting agenda.
 
  Maybe.  I don't want to see a json-specific hack for this, but some sort
  of generic way to add type knowledge could be useful, if we could figure
  out what we want.
 
 For this particular case, I think you just need some place to store a
 pg_type - pg_proc mapping.  I'm not exactly sure how to make that not
 a JSON-specific hack, since I certainly don't think we'd want to add a
 new catalog just for that.

This was my initial proposal to have casts to ::json for all types.

I backed out from this in favot of generic to_json(datum, indent) in
order to support prettyprinting.

 -- 
 Robert Haas
 EnterpriseDB: http://www.enterprisedb.com
 The Enterprise PostgreSQL Company
 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Hannu Krosing
On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
 Hannu Krosing ha...@2ndquadrant.com writes:
  On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
  So given that do we do anything about this now, or wait till 9.3?
 
  I'd like the json support in 9.2 updated as follows
 
 I think it's too late to be entertaining proposals for such changes in
 9.2.  If we had concluded that the existing functions were actively
 wrong or a bad idea,

I think that hard-coding postgresql text representation as our json
representation without a possibility for the user tio easily fix it
without rewriting foll xx_to_json() functions is borderline actively
wrong.

Can we at least have the xxx_to_json() functions try cast to json first
and fall back to text if the cast fails.

This would address my worst problem, all the rest can be easily defined
in user functions.

  then of course we'd need to do something; but they
 are not, so we can just as well consider additions in the 9.3 cycle
 rather than now.  I am not convinced that this proposal is fully baked
 yet, anyway; not to mention that right now we need to have our heads
 down on resolving the remaining open issues, not designing,
 implementing, and reviewing a pile of brand new code for json.
 
  By allowing developers just to define their own to_json(date) function
  we give them the power do decide which one to use. And if we honour
  search_path when looking up the to_json() functions, then they can even
  choose to have different conventions for different applications.
 
 This is not going to work anywhere near as nicely as you think.  If
 somebody tries to define multiple to_json() functions that override a
 generic to_json(anyelement) one, he will start getting function is not
 unique parse failures.  The parser will only successfully decide which
 function to call when the input data type exactly matches one of the
 specialized functions, which means you might as well not have the
 generic one at all.
 
   regards, tom lane
 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net wrote:
 On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
 Hannu Krosing ha...@2ndquadrant.com writes:
  On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
  So given that do we do anything about this now, or wait till 9.3?

  I'd like the json support in 9.2 updated as follows

 I think it's too late to be entertaining proposals for such changes in
 9.2.  If we had concluded that the existing functions were actively
 wrong or a bad idea,

 I think that hard-coding postgresql text representation as our json
 representation without a possibility for the user tio easily fix it
 without rewriting foll xx_to_json() functions is borderline actively
 wrong.

 Can we at least have the xxx_to_json() functions try cast to json first
 and fall back to text if the cast fails.

I think the idea that you can involve the casting machinery in this is
misguided.  sometextval::json has got to mean that sometextval is
expected to be in the form of a syntactically correct JSON value - and
NOT that we wrap it in a JSON string.  We can have constructors for
JSON, but they've got to be separate from the casting machinery.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net wrote:
 Can we at least have the xxx_to_json() functions try cast to json first
 and fall back to text if the cast fails.

 I think the idea that you can involve the casting machinery in this is
 misguided.

It is possible that that can be made to work, but it's a research
project, not something to be crammed into 9.2 at the last possible
minute.  In any case, I really dislike the idea that array_to_json
and row_to_json would contain two entirely different behaviors.
Leave the extensibility ideas for a future to_json() function.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread Hannu Krosing
On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote:
 On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net wrote:
  On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
  Hannu Krosing ha...@2ndquadrant.com writes:
   On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
   So given that do we do anything about this now, or wait till 9.3?
 
   I'd like the json support in 9.2 updated as follows
 
  I think it's too late to be entertaining proposals for such changes in
  9.2.  If we had concluded that the existing functions were actively
  wrong or a bad idea,
 
  I think that hard-coding postgresql text representation as our json
  representation without a possibility for the user tio easily fix it
  without rewriting foll xx_to_json() functions is borderline actively
  wrong.
 
  Can we at least have the xxx_to_json() functions try cast to json first
  and fall back to text if the cast fails.
 
 I think the idea that you can involve the casting machinery in this is
 misguided.  sometextval::json has got to mean that sometextval is
 expected to be in the form of a syntactically correct JSON value - and
 NOT that we wrap it in a JSON string.  We can have constructors for
 JSON, but they've got to be separate from the casting machinery.

on the contrary - the string representation of textual value a is a

casting should _not_ neam syntax check, casting is by definition a
conversion.

if we cast text to int, we return value of type int , if we cast int to
numeric(5,2) we return value of type numeric(5,2)

why should casring to json work differntly ?

 -- 
 Robert Haas
 EnterpriseDB: http://www.enterprisedb.com
 The Enterprise PostgreSQL Company
 



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] remove dead ports?

2012-05-04 Thread Marko Kreen
On Fri, May 4, 2012 at 8:45 PM, Peter Eisentraut pete...@gmx.net wrote:
 On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote:
 On 3 May 2012 17:21, Bruce Momjian br...@momjian.us wrote:
  I think I was the only user left;  I have never heard from a BSD/OS user
  in the past 5-7 years.

 I'm inclined to agree with Bruce. While it's not reasonable to assume
 that the lack of a BSD/OS user complaining on -general indicates that
 there are none, it's also not reasonable for them to expect us to
 support their operating system for 8 years after the original
 proprietary vendor. Better to not support BSD/OS than to supply a port
 that no one really has any confidence in. It's not as if we've ceased
 support in release branches.

 I'm not so much opposed to removing the port.  I am more concerned about
 the manner in which it was done.  The other ports I removed were known
 to not work anyway, for years, and there were at least several days of
 discussion.  The bsdi case was removing a working port with less than 24
 hours notice.

There is enough time until 9.2-final for a BSD/OS user raise complaints.

-- 
marko

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

2012-05-04 Thread David Johnston
 -Original Message-
 From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-
 ow...@postgresql.org] On Behalf Of Hannu Krosing
 Sent: Friday, May 04, 2012 4:40 PM
 To: Robert Haas
 Cc: Tom Lane; Andrew Dunstan; PostgreSQL-development; Merlin Moncure
 Subject: Re: [HACKERS] JSON in 9.2 - Could we have just one to_json()
 function instead of two separate versions ?
 
 On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote:
  On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net
 wrote:
   On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
   Hannu Krosing ha...@2ndquadrant.com writes:
On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
So given that do we do anything about this now, or wait till 9.3?
  
I'd like the json support in 9.2 updated as follows
  
   I think it's too late to be entertaining proposals for such changes
   in 9.2.  If we had concluded that the existing functions were
   actively wrong or a bad idea,
  
   I think that hard-coding postgresql text representation as our
   json representation without a possibility for the user tio easily
   fix it without rewriting foll xx_to_json() functions is borderline
   actively wrong.
  
   Can we at least have the xxx_to_json() functions try cast to json
   first and fall back to text if the cast fails.
 
  I think the idea that you can involve the casting machinery in this is
  misguided.  sometextval::json has got to mean that sometextval is
  expected to be in the form of a syntactically correct JSON value - and
  NOT that we wrap it in a JSON string.  We can have constructors for
  JSON, but they've got to be separate from the casting machinery.
 
 on the contrary - the string representation of textual value a is a
 
 casting should _not_ neam syntax check, casting is by definition a conversion.
 
 if we cast text to int, we return value of type int , if we cast int to
 numeric(5,2) we return value of type numeric(5,2)
 
 why should casring to json work differntly ?
 

What is the distinction between what you are thinking regarding JSON and this 
example?

SELECT '1a'::integer;
SQL Error: ERROR:  invalid input syntax for integer: 1a
LINE 1: SELECT '1a'::integer

As a user if I cast something to something else I want the result to be of the 
correct type and deterministic; otherwise throw me some kind of invalid input 
format exception (or syntax exception).  Casting vs. Constructors is really a 
meaningless distinction to a lay person.  When I cast I do so by constructing a 
new value using my existing value for input.  When I use an explicit CAST I am 
unable to supply additional parameters to configure the casting whereas a 
constructor function gives me that possibility.  But a constructor function 
without any additional parameters is not semantically different than a cast.

I guess the concern to address is something like:

SELECT '{key: value}'::json OR SELECT '[1.25]'::json;  Do you interpret this as 
already being valid JSON and thus output object/array constructs (i.e., JSON 
Text) or do you treat them as string literals and output scalars (i.e., JSON 
Value).  Even if you feel these are artificial constructs the concepts holds 
that there may be ambiguous data that can be interpreted in multiple ways (this 
applies even to function forms, though in the function form you could specify 
which one you want to output using a separate DEFAULTed parameter). 



I can see the primary use-case for JSON Value casting as being queries of the 
following forms (since the record and array forms are going through the 
record/array_to_json function):

SELECT COUNT(*)::json FROM table [WHERE ...];
SELECT single_col::json FROM table WHERE id = ?;

Where the single provided value can be sent directly back to the web-caller 
JavaScript and used as-is because it is valid JSON.  Though, honestly, both 
SELECT to_json(single_col) and SELECT to_json(COUNT(*)) are equally usable so 
any distinction between them is a pure technical issue to me.

Am I correct in assuming the following expected behavior (the forgive the 
blatantly wrong syntax but you should get the point)?

RAISE NOTICE '%', SELECT 'A'::text  =  A
RAISE NOTICE '%', SELECT 'A'::json = A

David J



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] remove dead ports?

2012-05-04 Thread Bruce Momjian
On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote:
 On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote:
  On 3 May 2012 17:21, Bruce Momjian br...@momjian.us wrote:
   I think I was the only user left;  I have never heard from a BSD/OS user
   in the past 5-7 years.
  
  I'm inclined to agree with Bruce. While it's not reasonable to assume
  that the lack of a BSD/OS user complaining on -general indicates that
  there are none, it's also not reasonable for them to expect us to
  support their operating system for 8 years after the original
  proprietary vendor. Better to not support BSD/OS than to supply a port
  that no one really has any confidence in. It's not as if we've ceased
  support in release branches.
 
 I'm not so much opposed to removing the port.  I am more concerned about
 the manner in which it was done.  The other ports I removed were known
 to not work anyway, for years, and there were at least several days of
 discussion.  The bsdi case was removing a working port with less than 24
 hours notice.

Not sure where you got 24 hours:

  Tues  http://archives.postgresql.org/pgsql-hackers/2012-05/msg00061.php
  Wed   http://archives.postgresql.org/pgsql-general/2012-05/msg00060.php
  Thur  http://archives.postgresql.org/pgsql-committers/2012-05/msg00023.php

Basically, we have beta next week so I wanted to do it before then, and
I have my head down doing the release notes, so I wanted to do it before
I started that too.  I kind of knew the bsdi answer before I even asked.

If you are objecting to me short-circuiting this, I will revert the
patch.  If we can't short-circuiting thinks when we already know the
answer, everyone's work will take more time.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

2012-05-04 Thread Jim Nasby

On 5/3/12 2:54 PM, Josh Berkus wrote:

(2)  If logical transactions had been implemented as additions to
  the WAL stream, and Slony was using that, do you think they would
  still have been usable for this recovery?

Quite possibly not.


The key advantage that I see in londiste/slony replication is that your data 
stream has absolutely nothing to do with anything binary or internal to 
Postgres. That means that the only way corruption will travel from a master to 
a slave is if the corruption is in the actual fields being updated, and even 
that's not a given (ie: UPDATING a field to a completely new value would not 
propagate corruption even if the old value of the field was corrupted).

So, embedding a logical stream into WAL is not inherently bad... what would be bad is if 
that logical stream was susceptible to corruption due to something like full 
page writes. Simply embedding the exact same info slony or londiste captures into the WAL 
should be fine (though likely defeats the purpose). Translating binary WAL data into DML 
statements would very likely allow corruption to travel from master to slave.
--
Jim C. Nasby, Database Architect   j...@nasby.net
512.569.9461 (cell) http://jim.nasby.net

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] remove dead ports?

2012-05-04 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote:
 I'm not so much opposed to removing the port.  I am more concerned about
 the manner in which it was done.  The other ports I removed were known
 to not work anyway, for years, and there were at least several days of
 discussion.  The bsdi case was removing a working port with less than 24
 hours notice.

What's the grounds for asserting they were known not to work?  Not
actual testing, I assume.

 Basically, we have beta next week so I wanted to do it before then, and
 I have my head down doing the release notes, so I wanted to do it before
 I started that too.  I kind of knew the bsdi answer before I even asked.

 If you are objecting to me short-circuiting this, I will revert the
 patch.  If we can't short-circuiting thinks when we already know the
 answer, everyone's work will take more time.

Leave it as-is.  I agree with the upthread comment that we can revert
the patch during beta (or even later than that), if anyone complains.
Furthermore, I would want to insist that a complainer provide a
buildfarm member as the price of us continuing to support an old
uncommon platform.  Otherwise the apparent support is hollow.  The BSDI
port was viable for us to support as long as Bruce was using it daily,
but with that gone, we need somebody else to be testing it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] remove dead ports?

2012-05-04 Thread Bruce Momjian
On Fri, May 04, 2012 at 06:25:24PM -0400, Tom Lane wrote:
 Bruce Momjian br...@momjian.us writes:
  On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote:
  I'm not so much opposed to removing the port.  I am more concerned about
  the manner in which it was done.  The other ports I removed were known
  to not work anyway, for years, and there were at least several days of
  discussion.  The bsdi case was removing a working port with less than 24
  hours notice.
 
 What's the grounds for asserting they were known not to work?  Not
 actual testing, I assume.
 
  Basically, we have beta next week so I wanted to do it before then, and
  I have my head down doing the release notes, so I wanted to do it before
  I started that too.  I kind of knew the bsdi answer before I even asked.
 
  If you are objecting to me short-circuiting this, I will revert the
  patch.  If we can't short-circuiting thinks when we already know the
  answer, everyone's work will take more time.
 
 Leave it as-is.  I agree with the upthread comment that we can revert
 the patch during beta (or even later than that), if anyone complains.
 Furthermore, I would want to insist that a complainer provide a
 buildfarm member as the price of us continuing to support an old
 uncommon platform.  Otherwise the apparent support is hollow.  The BSDI
 port was viable for us to support as long as Bruce was using it daily,
 but with that gone, we need somebody else to be testing it.

Yes, it was an odd port that probably would have been removed five years
ago if I had not been using it, which I am no longer.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Documentation for temp_file_limit

2012-05-04 Thread Bruce Momjian
The new 9.2 GUC parameter temp_file_limit says it restricts temporary
file usage per session, but it doesn't say what happens if a session
needs to exceed that value --- it throws an error.  Shouldn't we mention
that?

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers