Re: [HACKERS] CLOG extension
On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote: Your two paragraphs have roughly opposite arguments... Doing it every 32 pages would give you 30 seconds to complete the fsync, if you kicked it off when half way through the previous file - at current maximum rates. So there is utility in doing it in larger chunks. Maybe, but I'd like to try changing one thing at a time. If we change too much at once, it's likely to be hard to figure out where the improvement is coming from. Moving the task to a background process is one improvement; doing it in larger chunks is another. Those deserve independent testing. You gave a good argument why background pre-allocation wouldn't work very well if we do it a page at a time. I believe you. If we do it a file at a time, we can just write the file without calling it in page by page into the SLRU, as long as we write the WAL file for it first then we don't need to fsync either of them. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] index-only scans vs. Hot Standby, round two
On 2 May 2012 13:41, Robert Haas robertmh...@gmail.com wrote: So on further reflection I'm thinking it may be best just to stick with a hard conflict for now and see what feedback we get from beta testers. Which is what I was expecting y'all to conclude once you'd looked at the task in more detail. And I'm happy with the concept of beta being a period where we listen to feedback, not just bugs, and decide whether further refinements are required. What I think is possible is to alter the conflict as it hits the backend. If a backend has enable_indexonly = off then it wouldn't be affected by those conflicts anyhow. So if we simply record whether we are using an index only plan then we'll know whether to ignore it or abort. I think we can handle that by marking the snapshot at executor startup time. Needs a few other pushups but not that hard. The likelihood is that SQL that uses index only won't run for long enough to be cancelled anyway. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
On Wed, May 2, 2012 at 2:29 AM, Hannu Krosing ha...@2ndquadrant.com
wrote:
I don't object to row_to_json() and array_to_json() functions
being
there as a convenience and as the two official functions
guaranteed to
return JSON text.
So given that do we do anything about this now, or wait till 9.3?
Sorry for missing this mail, followed this only on list
I hope it is ok to CC this back to list
I'd like the json support in 9.2 updated as follows
Generic to_json(...) returning a JSON value
=
we should have a generic to_json(...) both for eas and use and for easy
extensibility, as explained below.
to_json(...) should work for all types, returning a json value similar
to what current json_in does, but for all types, not just cstring.
We could keep row_to_json() and array_to_json() as official json-text
returning functions
Configurable and extensible to_json()
==
When working on structured types, always the first try for getting an
element-as-json should be running to_json(element) and only if this
fails fall back to current use text representation code.
this enables two important things
1) configurable json-coding of values not explicitly supported by
standard
You can read about an attempt to standardise json-date formart here
http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx .
By allowing developers just to define their own to_json(date) function
we give them the power do decide which one to use. And if we honour
search_path when looking up the to_json() functions, then they can even
choose to have different conventions for different applications.
2) flexibility in adding support for extension types, like representing
hstore as object/dict by just providing the to_json(hstore, ...)
functions in hstore extension
Pretty-printing
===
If we were to support prettyprinting of anything more complex than
single level structs (record or array), then we need to pass ident
into the to_json() function
my recommendation would be to have the signature
to_json(datum any, ident int)
with ident = NULL meaning no prettyprint , ident =0 meaninf top level,
or starting at left margin and anything else meaning the amount of
spaces needed to be added to the beginning of all rows exept the first
one, for example the query
hannu=# select array_to_json(array(select test from test limit 2),true);
array_to_json
-
[{id:9,data:testdata,tstamp:2012-05-01 09:44:50.175189}, +
{id:10,data:testdata,tstamp:2012-05-01 09:45:50.260276}]
(1 row)
could return this:
[{'data': 'testdata',
'id': 9,
'tstamp': '2012-05-01 09:44:50.175189'},
{'data': 'testdata',
'id': 10,
'tstamp': '2012-05-01 09:45:50.260276'}]
if it would call to_json(row, 1) for getting each row prettyprinted with
ident 1
Getting a record _from_ json()
==
JSON support would be much more useful if we supported the function of
converting the other way as well, that is from json to record
The best model easy to support seems to be what Merlin suggested, that
is the populate_record(record,hstore) function and corresponding #=
operator (see: http://www.postgresql.org/docs/9.1/static/hstore.html)
The complete plan for 9.2 once more
===
What is needed to nicely round up a simple and powerful json type is
1. the json TYPE
an agreement tha json type represents a JSON value (this is what
json_in currently does)
2. json output
2.1 doing the right thing with internal values
to_json() function for converting to this JSON value for any type.
default implementation for 'non-json' types returns their
postgresql textual representation in double quotes
(date - 2012-05-01 09:45:50.260276
structured types use to_json() for getting values internally,
so that by defining to_json(hstore) you can automatically get
hstore represented in javascript object or dictionary representation
hannu=# select row_to_json(r)
from (select 1::int id, 'foo=1, bar=baz'::hstore)r;
should not return this:
row_to_json
--
{id:1,hstore:\bar\=\baz\, \foo\=\1\}
(1 row)
but this
row_to_json
--
{id:1,hstore:{bar: baz, foo:1}}
(1 row)
2.1 getting the pretty-printing right for structured types
to_json(any, indent) functions for getting recursive prettyprinting.
we might also need another argument telling the page width we want
to pretty print to.
3. json input
Re: [HACKERS] Future In-Core Replication
On Thu, 2012-05-03 at 00:58 -0500, Jim Nasby wrote: On 4/29/12 6:03 AM, Simon Riggs wrote: The DML-WITH-LIMIT-1 is required to do single logical updates on tables with non-unique rows. And as for any logical updates we will have huge performance problem when doing UPDATE or DELETE on large table with no indexes, but fortunately this problem is on slave, not master;) While that is possible, I would favour the do-nothing approach. By making the default replication mode = none, we then require a PK to be assigned before allowing replication mode = on for a table. Trying to replicate tables without PKs is a problem that can wait basically. Something that a in-core method might be able to do that an external one can't would be to support a method of uniquely identifying rows in tables with no PK's. A gross example (that undoubtedly wouldn't work in the real world) would be using TID's. A real-world implementation might be based on a hidden serial column. For logical we don't really need to uniquely identify such rows - it should sufficient if we just update exactly one of the matching rows. The way to do this is to put all fields of the OLD.* tuple in the WHERE clause and then update just one matching row. IIRC updating (or deleting) CURRENT OF a cursor is currently supported only in pl/pgsql so this needs to be done using a plpgsql cursor. If the table has no indexes or index lookup returns lots of rows, then this is bound to be slow, but in this case it was probably slow on master too :) -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CLOG extension
On Fri, May 4, 2012 at 3:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote: Your two paragraphs have roughly opposite arguments... Doing it every 32 pages would give you 30 seconds to complete the fsync, if you kicked it off when half way through the previous file - at current maximum rates. So there is utility in doing it in larger chunks. Maybe, but I'd like to try changing one thing at a time. If we change too much at once, it's likely to be hard to figure out where the improvement is coming from. Moving the task to a background process is one improvement; doing it in larger chunks is another. Those deserve independent testing. You gave a good argument why background pre-allocation wouldn't work very well if we do it a page at a time. I believe you. Your confidence is sort of gratifying, but in this case I believe it's misplaced. On more careful analysis, it seems that ExtendCLOG() does just two things: (1) evict a CLOG buffer and replace it with a zero'd page representing the new page and (2) write an XLOG record for the change. Apparently, extending CLOG doesn't actually involve extending anything on disk at all. We rely on the future buffer eviction to do that, which is surprisingly different from the way relation extension is handled. So CLOG extension is normally fast, but occasionally something goes wrong. So far I see two ways that can happen: (1) the WAL insertion stalls because wal_buffers are full, and we're forced to wait for WAL to be written (and perhaps fsync'd, since both are covered by the same lock) or (2) the page we choose to evict happens to be dirty, and we have to write+fsync it before repurposing it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
On Fri, May 4, 2012 at 8:32 AM, Hannu Krosing ha...@krosing.net wrote: For logical we don't really need to uniquely identify such rows - it should sufficient if we just update exactly one of the matching rows. The way to do this is to put all fields of the OLD.* tuple in the WHERE clause and then update just one matching row. IIRC updating (or deleting) CURRENT OF a cursor is currently supported only in pl/pgsql so this needs to be done using a plpgsql cursor. If the table has no indexes or index lookup returns lots of rows, then this is bound to be slow, but in this case it was probably slow on master too :) I was about to write a reply saying exactly this, but you said it better than I would have been able to manage. I think this is all exactly right. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
On Thu, May 3, 2012 at 8:22 PM, Greg Smith g...@2ndquadrant.com wrote: On 05/01/2012 09:09 AM, Robert Haas wrote: I think we ought to be sharing and debugging designs in public, not internally within 2ndQuadrant - or any other company, or any other mailing list other than this one. OK. You go first. Are you requesting more transparency in general, asking for my thoughts on logical replication specifically, or something else? I try pretty hard not to go off and do large amounts of work in a vacuum. If something is more than a couple days work, I post the design on hackers and wait for feedback before writing a line of code. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CLOG extension
On 4 May 2012 13:59, Robert Haas robertmh...@gmail.com wrote: On Fri, May 4, 2012 at 3:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote: Your two paragraphs have roughly opposite arguments... Doing it every 32 pages would give you 30 seconds to complete the fsync, if you kicked it off when half way through the previous file - at current maximum rates. So there is utility in doing it in larger chunks. Maybe, but I'd like to try changing one thing at a time. If we change too much at once, it's likely to be hard to figure out where the improvement is coming from. Moving the task to a background process is one improvement; doing it in larger chunks is another. Those deserve independent testing. You gave a good argument why background pre-allocation wouldn't work very well if we do it a page at a time. I believe you. Your confidence is sort of gratifying, but in this case I believe it's misplaced. On more careful analysis, it seems that ExtendCLOG() does just two things: (1) evict a CLOG buffer and replace it with a zero'd page representing the new page and (2) write an XLOG record for the change. Apparently, extending CLOG doesn't actually involve extending anything on disk at all. We rely on the future buffer eviction to do that, which is surprisingly different from the way relation extension is handled. So CLOG extension is normally fast, but occasionally something goes wrong. I don't agree its normally fast. WALInsert contention is high, so there is usually a long queue. As we've discussed this can be done offline and and so (2) can completely avoided in the main line. Considering that all new xids wait for this action, any wait at all is bad and takes time to drain once it clears. Evicting a clog has cost because the tail is almost always dirty when we switch pages. Doing both of those will ensure switch to new page requires zero wait time. So you have the solution. Not sure what else you're looking for. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
On 4 May 2012 14:01, Robert Haas robertmh...@gmail.com wrote: On Fri, May 4, 2012 at 8:32 AM, Hannu Krosing ha...@krosing.net wrote: For logical we don't really need to uniquely identify such rows - it should sufficient if we just update exactly one of the matching rows. The way to do this is to put all fields of the OLD.* tuple in the WHERE clause and then update just one matching row. IIRC updating (or deleting) CURRENT OF a cursor is currently supported only in pl/pgsql so this needs to be done using a plpgsql cursor. If the table has no indexes or index lookup returns lots of rows, then this is bound to be slow, but in this case it was probably slow on master too :) I was about to write a reply saying exactly this, but you said it better than I would have been able to manage. I think this is all exactly right. Yes, but its not a high priority for inclusion. Many things like this will need to wait behind the really critical additional features. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Advisory locks seem rather broken
On Thu, May 3, 2012 at 5:18 PM, Tom Lane t...@sss.pgh.pa.us wrote:
... btw, it appears to me that the fast path patch has broken things
rather badly in LockReleaseAll. AFAICS it's not honoring either the
lockmethodid restriction nor the allLocks restriction with respect to
fastpath locks. Perhaps user locks and session locks are never taken
fast path, but still it would be better to be making those checks
further up, no?
User locks are never taken fast path, but session locks can be, so I
think you're right that there is a bug here. I think what we should
probably do is put the nLocks == 0 test before the lockmethodid and
allLocks checks, and then the fast path stuff after those two checks.
In 9.1, we just did this:
if (locallock-proclock == NULL || locallock-lock == NULL)
{
/*
* We must've run out of shared memory while
trying to set up this
* lock. Just forget the local entry.
*/
Assert(locallock-nLocks == 0);
RemoveLocalLock(locallock);
continue;
}
...and I just shoved the new logic into that stanza without thinking
hard enough about what order to do things in.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Advisory locks seem rather broken
Robert Haas robertmh...@gmail.com writes:
In 9.1, we just did this:
if (locallock-proclock == NULL || locallock-lock == NULL)
{
/*
* We must've run out of shared memory while
trying to set up this
* lock. Just forget the local entry.
*/
Assert(locallock-nLocks == 0);
RemoveLocalLock(locallock);
continue;
}
...and I just shoved the new logic into that stanza without thinking
hard enough about what order to do things in.
Right. The other thing that was bothering me about that was that it's
not clear now how to tell a broken locallock entry (which is what this
logic originally intended to clean up) from a fastpath one. Perhaps
it'd be a good idea to add a valid flag? And while I'm wondering
about such things, what happens when it's necessary to convert a
fastpath lock to a regular one, but there's no room in shared memory
for more lock objects?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote: So given that do we do anything about this now, or wait till 9.3? I'd like the json support in 9.2 updated as follows I think it's too late to be entertaining proposals for such changes in 9.2. If we had concluded that the existing functions were actively wrong or a bad idea, then of course we'd need to do something; but they are not, so we can just as well consider additions in the 9.3 cycle rather than now. I am not convinced that this proposal is fully baked yet, anyway; not to mention that right now we need to have our heads down on resolving the remaining open issues, not designing, implementing, and reviewing a pile of brand new code for json. By allowing developers just to define their own to_json(date) function we give them the power do decide which one to use. And if we honour search_path when looking up the to_json() functions, then they can even choose to have different conventions for different applications. This is not going to work anywhere near as nicely as you think. If somebody tries to define multiple to_json() functions that override a generic to_json(anyelement) one, he will start getting function is not unique parse failures. The parser will only successfully decide which function to call when the input data type exactly matches one of the specialized functions, which means you might as well not have the generic one at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CLOG extension
On Fri, May 4, 2012 at 9:11 AM, Simon Riggs si...@2ndquadrant.com wrote: On 4 May 2012 13:59, Robert Haas robertmh...@gmail.com wrote: On Fri, May 4, 2012 at 3:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, May 3, 2012 at 9:56 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 3, 2012 at 3:20 PM, Simon Riggs si...@2ndquadrant.com wrote: Your two paragraphs have roughly opposite arguments... Doing it every 32 pages would give you 30 seconds to complete the fsync, if you kicked it off when half way through the previous file - at current maximum rates. So there is utility in doing it in larger chunks. Maybe, but I'd like to try changing one thing at a time. If we change too much at once, it's likely to be hard to figure out where the improvement is coming from. Moving the task to a background process is one improvement; doing it in larger chunks is another. Those deserve independent testing. You gave a good argument why background pre-allocation wouldn't work very well if we do it a page at a time. I believe you. Your confidence is sort of gratifying, but in this case I believe it's misplaced. On more careful analysis, it seems that ExtendCLOG() does just two things: (1) evict a CLOG buffer and replace it with a zero'd page representing the new page and (2) write an XLOG record for the change. Apparently, extending CLOG doesn't actually involve extending anything on disk at all. We rely on the future buffer eviction to do that, which is surprisingly different from the way relation extension is handled. So CLOG extension is normally fast, but occasionally something goes wrong. I don't agree its normally fast. WALInsert contention is high, so there is usually a long queue. As we've discussed this can be done offline and and so (2) can completely avoided in the main line. Considering that all new xids wait for this action, any wait at all is bad and takes time to drain once it clears. Evicting a clog has cost because the tail is almost always dirty when we switch pages. Doing both of those will ensure switch to new page requires zero wait time. So you have the solution. Not sure what else you're looking for. Nothing, really. I was just mooting some ideas before I went and started coding, to see what people thought. I've got your opinion and Tom's, and of course my own, so now I'm off to test some different approaches. At the moment I'm running a battery of tests on background-writing CLOG, which I will post about when they are complete, and I intend to play around with some of the ideas from this thread as well. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Advisory locks seem rather broken
On Fri, May 4, 2012 at 9:25 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Robert Haas robertmh...@gmail.com writes:
In 9.1, we just did this:
if (locallock-proclock == NULL || locallock-lock == NULL)
{
/*
* We must've run out of shared memory while
trying to set up this
* lock. Just forget the local entry.
*/
Assert(locallock-nLocks == 0);
RemoveLocalLock(locallock);
continue;
}
...and I just shoved the new logic into that stanza without thinking
hard enough about what order to do things in.
Right. The other thing that was bothering me about that was that it's
not clear now how to tell a broken locallock entry (which is what this
logic originally intended to clean up) from a fastpath one. Perhaps
it'd be a good idea to add a valid flag?
Well, I think nLocks == 0 should serve that purpose adequately.
And while I'm wondering
about such things, what happens when it's necessary to convert a
fastpath lock to a regular one, but there's no room in shared memory
for more lock objects?
Then you error out. Of course, if the fast path mechanism didn't
exist at all, you would have started erroring out much sooner. Now,
there is some rub here, because the mechanism isn't fair: strong
lockers will error out instead of weak lockers, and in the worst case
where the lock table remains perpetually on the edge of overflowing,
strong lock requests could be fail repeatedly, essentially a DOS.
Originally, I thought that the patch should include some kind of
accounting mechanism to prevent that from happening, where we'd keep
track of the number of fast-path locks that were outstanding and make
sure to keep that many slots free in the main lock table, but Noah
talked me out of it, on theory that (1) it was very unlikely to occur
in practice and (2) if it did occur, then you probably need to bump up
max_locks_per_transaction anyway and (3) it amounted to forcing
failures in cases where that might not be strictly necessary, which is
usually not a great thing to do.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Advisory locks seem rather broken
Robert Haas robertmh...@gmail.com writes: Originally, I thought that the patch should include some kind of accounting mechanism to prevent that from happening, where we'd keep track of the number of fast-path locks that were outstanding and make sure to keep that many slots free in the main lock table, but Noah talked me out of it, on theory that (1) it was very unlikely to occur in practice and (2) if it did occur, then you probably need to bump up max_locks_per_transaction anyway and (3) it amounted to forcing failures in cases where that might not be strictly necessary, which is usually not a great thing to do. I agree with that, as long as we can be sure that the system behaves sanely (doesn't leave the data structures in a corrupt state) when an out-of-memory condition does occur. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Advisory locks seem rather broken
On Fri, May 4, 2012 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Originally, I thought that the patch should include some kind of accounting mechanism to prevent that from happening, where we'd keep track of the number of fast-path locks that were outstanding and make sure to keep that many slots free in the main lock table, but Noah talked me out of it, on theory that (1) it was very unlikely to occur in practice and (2) if it did occur, then you probably need to bump up max_locks_per_transaction anyway and (3) it amounted to forcing failures in cases where that might not be strictly necessary, which is usually not a great thing to do. I agree with that, as long as we can be sure that the system behaves sanely (doesn't leave the data structures in a corrupt state) when an out-of-memory condition does occur. OK. I believe that commit 53c5b869b464d567c3b8f617201b49a395f437ab robustified this code path quite a bit; but it is certainly possible that there are remaining oversights, and I would certainly appreciate any further review you have time to do. Basically, it seems like the likely failure modes, if there are further bugs, would be either (1) failing to track the strong lock counts properly, leading to performance degradation if the counters become permanently stuck at a value other than zero even after all the locks are gone or (2) somehow muffing the migration of a lock from the fast-path mechanism to the regular mechanism. When taking a strong lock, the idea is that the strong locker first bumps the strong lock count. That bump must be unwound if we fail to acquire the lock, which means it has to be cleaned up in the error path and any case where we give up (e.g. conditional acquire of a contended lock). Next, we iterate through all the backend slots and transfer fast path locks for each one individually. If we fail midway through, the strong locker must simply make sure to unwind the strong lock count. The weak lockers whose locks got transferred are fine: they need to know how to cope with releasing a transferred lock anyway; whether the backend that did the transfer subsequently blew up is not something they have any need to care about. Once all the transfers are complete, the strong locker queues for the lock using the main mechanism, which now includes all possible conflicting locks. Again, if we blow up while waiting for the lock, the only extra thing we need to do is unwind the strong lock count acquisition. Of course, I may be missing some other kind of bug altogether... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
On 05/04/2012 09:03 AM, Robert Haas wrote: I try pretty hard not to go off and do large amounts of work in a vacuum. If something is more than a couple days work, I post the design on hackers and wait for feedback before writing a line of code. That is an excellent luxury to have. You've worked very hard to earn it. Not everyone is in the position where that's possible though. Are you requesting more transparency in general, asking for my thoughts on logical replication specifically, or something else? The straw man argument here would require 100% transparency on everything you do in regards to PostgreSQL and related software. Before doing any development on any code, first post here to ask for design review. And if someone asks you to work on a program that isn't open source from day one, refuse unless you can operate that transparently. That standard is nice if you can pull it off. But I don't give you a hard time if you have to make some compromises from that ideal to keep yourself gainfully employed. You do a ton of good work for the PostgreSQL community in a transparent way, so I assume that you're doing the best you can. I would like to see that assumption presumed on our side, too. Here are the individual straw men in this area I'd like to see put out of their misery: You're developing things in secret: if that's the case, we're pretty bad at it, given the history I outlined at http://archives.postgresql.org/message-id/4f9b1b6c.5010...@2ndquadrant.com That discussion didn't happen in the right place: it's not our fault that the cluster-hackers list exists. Go joust at getting that list shut down and their meeting during PGCon canceled if you think it's unproductive for discussions to happen there. I've been trying to bridge that gap for over two years now; note how many times I appear in the edit history at http://wiki.postgresql.org/index.php?title=ClusterFeaturesaction=history You might do too much development in the wrong direction and not build the right thing: and? Yes, there are people who develop into a corner and end up doing unproductive work as a result. And there are others who submit things and give up when faced with feedback on them. Last time I checked, there wasn't anyone who flat-out rejects on-list feedback working for 2ndQuadrant. Instead, I see features that go through extensive and numerous review cycles based on what we hear back. Designs should be presented on-list before doing any development: this is not always practical for those of us who are doing feature development. Some feature sponsors are still getting used to open development. If we have a private development milestone date to hit *in order to get more funding for public PostgreSQL work*, which is often the case here, we try not to miss it. We'd be bad community members to do so. And sometimes that involves building a proof of concept or prototype here first, then submitting it to the community once it's moved onto being a proven concept. Since the community has a clear set of guidelines for how and when to submit new features, we make sure the development plans line up with them. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
On Fri, May 4, 2012 at 11:06 AM, Greg Smith g...@2ndquadrant.com wrote: The straw man argument here would require 100% transparency on everything you do in regards to PostgreSQL and related software. Before doing any development on any code, first post here to ask for design review. And if someone asks you to work on a program that isn't open source from day one, refuse unless you can operate that transparently. Well, look. At the end of the day, I don't really care whether you post your designs before writing code or not - unless it turns out that we get to the end of the development cycle, a gigantic patch shows up at the last minute, it gets rejected because people aren't satisfied with the design, and then massive bitching ensues because the author(s) put a lot of work into that patch. Then I care, because now the fact that no design consensus was sought at the outset has been transformed into a defect in the community process, which does in fact have defects, but that isn't one of them. We all know that design review is going to have to happen at some point, and if there's not an adequate opportunity to do that before the code is written then it will happen after the code is written. If that means the code has to be thrown out, then that's the risk you take by writing the code first. As long as everybody understands that, do it in whatever order you like. I think the real straw man here is the idea that it will somehow save time to skip the design phase and start writing code. I have never worked on a project, open source or otherwise, where that was true, and I believe that any textbook on software engineering you pick up is likely to tell you that in fact exactly the opposite is the case. Obviously, there are times when you need to write some throw-away code just to see how things shake out, and I do that all the time, and it makes complete sense, and I'm not knocking it. But if any of that code makes it into the committed patch, I count that as unusually lucky. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On 05/04/2012 09:52 AM, Tom Lane wrote: Hannu Krosingha...@2ndquadrant.com writes: On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote: So given that do we do anything about this now, or wait till 9.3? I'd like the json support in 9.2 updated as follows I think it's too late to be entertaining proposals for such changes in 9.2. If we had concluded that the existing functions were actively wrong or a bad idea, then of course we'd need to do something; but they are not, so we can just as well consider additions in the 9.3 cycle rather than now. I am not convinced that this proposal is fully baked yet, anyway; not to mention that right now we need to have our heads down on resolving the remaining open issues, not designing, implementing, and reviewing a pile of brand new code for json. Yeah, that was my feeling. We usually take a release or two to get things right, fill in what's missing, etc. and I don't think this will be ant different. By allowing developers just to define their own to_json(date) function we give them the power do decide which one to use. And if we honour search_path when looking up the to_json() functions, then they can even choose to have different conventions for different applications. This is not going to work anywhere near as nicely as you think. If somebody tries to define multiple to_json() functions that override a generic to_json(anyelement) one, he will start getting function is not unique parse failures. The parser will only successfully decide which function to call when the input data type exactly matches one of the specialized functions, which means you might as well not have the generic one at all. Yeah, what I've been thinking about in conjunction with similar problems is some sort of type registry, so that we could code for non-builtin types in certain cases. Maybe we should add that the the developers' meeting agenda. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beta time?
Next week, I thought. How are we handling the Monday release with everyone at PGCon? Was that resolved? I have yet to see a confirmed date, guys. If we expect any support from the packagers and/or the advocacy volunteers, then people need at least a week's notice, probably more. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] c-function variants running time
Hi everybody.
First of all I have to thank you for your wonderful job! PostgreSQL rocks!
I am writing you because I am interested in understanding some specifics related
to PostgreSQL internals. More precisely, I am investigating the running time
of the different function implementation approaches, which is part of my BSc
thesis.
Here is the thing: I have implemented as a proof of concept three functions,
which are
a) text[] arraypoc(text, int); this returns an array of the form 'text,int'
where the text is the copy of the text passed as parameter and int is a
simple counter. The number of tuples is specified by the integer parameter.
I have estimated the running time in this manner:
SELECT *
FROM unnest(arraypoc('abcdefghilmnopqrstuvz',100));
The estimated running time is after 10 executions is:
(791.571 + 797.163 + 677.331 + 686.674 + 686.691 + 686.438 +
797.910 + 795.955 + 793.459 + 794.110)/10 = 750.7302
b) TABLE(text,int) srfpoc(text, int); is similar as the previous one but this
is a set returning function, which returns a table of a similar shape as in
the previous case. Instead of a string, I return a text and an integer. Again
text is just the copy of the parameter and int is a counter.
I have estimated the running time in this manner:
SELECT *
FROM srfpoc('abcdefghilmnopqrstuvz',100);
The estimated running time is after 10 executions is:
(665.016 + 778.100 + 640.605 + 787.102 + 785.501 + 791.307 +
784.780 + 793.222 + 794.624 + 790.357)/10 = 761.0614
c) TABLE(text,int) srfmatpoc(text, int); this does the same as the previous one,
but in this case I wrote a SRF_Materialized using the SPI interface. I have
estimated the running time in this manner:
SELECT *
FROM srfmatpoc('abcdefghilmnopqrstuvz',100);
The estimated running time is after 10 executions is:
(747.095 + 703.894 + 762.310 + 763.299 + 764.582 + 760.991 + 763.427 +
764.033 + 731.292 + 770.895)/10 = 753.1818
I have executed all the tests on the same server. The functions are compiled
using the -O3 compilation parameter. I am using PostgreSQL 9.1.3.
I would have expected the version a) to be slower than b) and c) but it turns
out that it is actually the fastest (the difference is not so big anyway). What
am I doing wrong? What can I do to improve the functions? Have I misunderstood
something?
Attached you find the code of all three functions.
Thanks a lot!
Armando
#include postgres.h
#include string.h
#include utils/array.h
#include utils/builtins.h /* text_to_cstring */
#include catalog/pg_type.h
#include utils/lsyscache.h
#include fmgr.h
#ifdef PG_MODULE_MAGIC
PG_MODULE_MAGIC;
#endif
PG_FUNCTION_INFO_V1(arraypoc);
Datum arraypoc(PG_FUNCTION_ARGS) {
ArrayType *result;
Datum *result_data = NULL;
int16o_typlen;
bool o_typbyval;
char o_typalign;
text*t,
*cpyt;
uint32 c;
int i, t_len;
t = (text *)PG_GETARG_TEXT_PP(0);
c = (uint32)PG_GETARG_INT32(1);
if(c0) result_data = (Datum *)palloc(sizeof(Datum) * c);
for(i=0; ic; ++i) {
t_len = VARSIZE_ANY_EXHDR(t);
cpyt = (text *)palloc(VARHDRSZ + t_len + 1 + 16);
SET_VARSIZE(cpyt, VARHDRSZ + t_len + 1 + 16);
memcpy((void *)VARDATA(cpyt), (void *)VARDATA(t), t_len);
memcpy((void *)VARDATA(cpyt) + t_len, ,, 1);
snprintf((void *)VARDATA(cpyt) + t_len + 1, 16, %d, i);
result_data[i] = PointerGetDatum(cpyt);
}
if(result_data == NULL || c = 0) {
PG_RETURN_NULL();
} else {
get_typlenbyvalalign(TEXTOID, o_typlen, o_typbyval, o_typalign);
result = construct_array(result_data, c, TEXTOID, o_typlen, o_typbyval,
o_typalign);
PG_RETURN_ARRAYTYPE_P(result);
}
}
#include postgres.h
#include stdio.h/* snprintf */
#include stdlib.h/* malloc, free */
#include string.h/* strncpy, strlen */
#include sys/types.h /* for uint32 and size_t*/
#include catalog/pg_type.h
#include utils/builtins.h /* text_to_cstring */
#include funcapi.h
#include fmgr.h
#ifdef PG_MODULE_MAGIC
PG_MODULE_MAGIC;
#endif
PG_FUNCTION_INFO_V1(srfpoc);
Datum srfpoc(PG_FUNCTION_ARGS) {
FuncCallContext *funcctx;
TupleDesctupdesc;
text*t;
int max_calls,
call_cntr;
t = (text *)PG_GETARG_TEXT_PP(0);
if (SRF_IS_FIRSTCALL()) {
MemoryContextoldcontext;
funcctx = SRF_FIRSTCALL_INIT();
oldcontext = MemoryContextSwitchTo(funcctx-multi_call_memory_ctx);
funcctx-max_calls = (uint32)PG_GETARG_INT32(1);
tupdesc = CreateTemplateTupleDesc(2, false);
TupleDescInitEntry(tupdesc, (AttrNumber) 1, t, TEXTOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2, count, INT4OID, -1, 0);
funcctx-user_fctx = BlessTupleDesc(tupdesc);
MemoryContextSwitchTo(oldcontext);
}
funcctx =
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
Andrew Dunstan and...@dunslane.net writes: Yeah, what I've been thinking about in conjunction with similar problems is some sort of type registry, so that we could code for non-builtin types in certain cases. Maybe we should add that the the developers' meeting agenda. Maybe. I don't want to see a json-specific hack for this, but some sort of generic way to add type knowledge could be useful, if we could figure out what we want. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beta time?
Josh Berkus j...@agliodbs.com writes: How are we handling the Monday release with everyone at PGCon? Was that resolved? I have yet to see a confirmed date, guys. If we expect any support from the packagers and/or the advocacy volunteers, then people need at least a week's notice, probably more. I haven't seen anybody positively say we can't do a wrap next Thursday and release Monday, so I've been assuming that's what will happen. If there are reasons to think it won't work, let's hear 'em now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
Hi Robert, Hi all, On Friday, May 04, 2012 06:29:33 PM Robert Haas wrote: On Fri, May 4, 2012 at 11:06 AM, Greg Smith g...@2ndquadrant.com wrote: The straw man argument here would require 100% transparency on everything you do in regards to PostgreSQL and related software. Before doing any development on any code, first post here to ask for design review. And if someone asks you to work on a program that isn't open source from day one, refuse unless you can operate that transparently. Well, look. At the end of the day, I don't really care whether you post your designs before writing code or not - unless it turns out that we get to the end of the development cycle, a gigantic patch shows up at the last minute, it gets rejected because people aren't satisfied with the design, and then massive bitching ensues because the author(s) put a lot of work into that patch. Then I care, because now the fact that no design consensus was sought at the outset has been transformed into a defect in the community process, which does in fact have defects, but that isn't one of them. We all know that design review is going to have to happen at some point, and if there's not an adequate opportunity to do that before the code is written then it will happen after the code is written. If that means the code has to be thrown out, then that's the risk you take by writing the code first. As long as everybody understands that, do it in whatever order you like. In my understanding - as the person doing quite a bit of the coding atm - the point is to provide a very minimal *early* prototype to have a sensible basis for design decisions/discussions. On one side thats useful to get a feeling for the problems involved. On the other side doing design discussions without an underlaying basic patch design on -hackers tends to often go into directions of feature creep and bikeshedding. It also helps against this is impossible claims. Parts of this thread and related ones are a somewhat good example of this. The plan is to show the early prototype around pgcon and send design documents and split-up patches (of that prototype) a holiday and some cleanup later to - hackers. I/We aim to have individual, independently usable, parts of the patch submitted to the first 9.3 commitfest. I definitely do not wish to screw anyone over doing this or such. And I am sure thats the same with others working on this. Even if sometimes emotions get in the way/into play. Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Future In-Core Replication
On Fri, May 4, 2012 at 12:59 PM, Andres Freund and...@2ndquadrant.com wrote: In my understanding - as the person doing quite a bit of the coding atm - the point is to provide a very minimal *early* prototype to have a sensible basis for design decisions/discussions. On one side thats useful to get a feeling for the problems involved. On the other side doing design discussions without an underlaying basic patch design on -hackers tends to often go into directions of feature creep and bikeshedding. It also helps against this is impossible claims. Parts of this thread and related ones are a somewhat good example of this. The plan is to show the early prototype around pgcon and send design documents and split-up patches (of that prototype) a holiday and some cleanup later to - hackers. I/We aim to have individual, independently usable, parts of the patch submitted to the first 9.3 commitfest. I definitely do not wish to screw anyone over doing this or such. And I am sure thats the same with others working on this. Even if sometimes emotions get in the way/into play. Thanks. I think we are basically all on the same page, here. As I said upthread, I am not trying to accuse anything of doing anything wrong, and as you and Simon and Greg have all said, you're not trying to do anything wrong. So, no problem. :-) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On Fri, May 4, 2012 at 12:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: Yeah, what I've been thinking about in conjunction with similar problems is some sort of type registry, so that we could code for non-builtin types in certain cases. Maybe we should add that the the developers' meeting agenda. Maybe. I don't want to see a json-specific hack for this, but some sort of generic way to add type knowledge could be useful, if we could figure out what we want. For this particular case, I think you just need some place to store a pg_type - pg_proc mapping. I'm not exactly sure how to make that not a JSON-specific hack, since I certainly don't think we'd want to add a new catalog just for that. In general, I think it would be very useful to have some way of identifying particular types - and versions of types - independently of a particular installation - e.g. by assigning each type a UUID that never changes and a version number that we bump when we change something about that type. That seems like it would be very useful for schema comparison tools, or for logical replication, where you want to know whether two types are the same type even though they are in different clusters. pg_upgrade has had past needs in this area as well. However, I'm not sure that'd help solve this particular problem. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] remove dead ports?
On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote: On 3 May 2012 17:21, Bruce Momjian br...@momjian.us wrote: I think I was the only user left; I have never heard from a BSD/OS user in the past 5-7 years. I'm inclined to agree with Bruce. While it's not reasonable to assume that the lack of a BSD/OS user complaining on -general indicates that there are none, it's also not reasonable for them to expect us to support their operating system for 8 years after the original proprietary vendor. Better to not support BSD/OS than to supply a port that no one really has any confidence in. It's not as if we've ceased support in release branches. I'm not so much opposed to removing the port. I am more concerned about the manner in which it was done. The other ports I removed were known to not work anyway, for years, and there were at least several days of discussion. The bsdi case was removing a working port with less than 24 hours notice. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Uppercase tab completion keywords in psql?
On tor, 2012-05-03 at 15:47 -0400, Bruce Momjian wrote: Peter, where are we on this? I hadn't received any clear feedback, but if no one objects, I can commit it. --- On Fri, Mar 30, 2012 at 08:16:59PM +0300, Peter Eisentraut wrote: On fre, 2012-03-23 at 07:52 -0700, David Fetter wrote: On Thu, Mar 22, 2012 at 06:05:30PM -0400, Andrew Dunstan wrote: On 03/22/2012 05:49 PM, Bruce Momjian wrote: Robert Haas and I are disappointed by this change. I liked the fact that I could post nice-looking SQL queries without having to use my capslock key (which I use as a second control key). Any chance of reverting this change? Should it be governed by a setting? Something like (upper|lower|preserve) ? How about this patch then? (There are actually four possible settings, see patch.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PL/Python result set slicing broken in Python 3
On 03/05/12 11:04, Jan Urbański wrote: On 02/05/12 20:18, Peter Eisentraut wrote: This doesn't work anymore with Python 3: rv = plpy.execute(...) do_something(rv[0:1]) Apparently, they changed the C API for doing slicing, or rather made one of the two APIs for it silently do nothing. Details are difficult to find, but this email message seems to contain something: http://mail.python.org/pipermail/python-3000/2007-August/009851.html. I'll try to sort this out sometime, but if someone wants to take a shot at it, go ahead. Sounds ugly. I'll take a look. I found some instructions on how to deal with the Python 2/Python 3 slicing mess: http://renesd.blogspot.com/2009/07/python3-c-api-simple-slicing-sqslice.html Apparently you need that egregious hack in order to avoid code duplication. I'll try to produce a patch over the weekend. Cheers, Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On Fri, 2012-05-04 at 13:43 -0400, Robert Haas wrote: On Fri, May 4, 2012 at 12:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: Yeah, what I've been thinking about in conjunction with similar problems is some sort of type registry, so that we could code for non-builtin types in certain cases. Maybe we should add that the the developers' meeting agenda. Maybe. I don't want to see a json-specific hack for this, but some sort of generic way to add type knowledge could be useful, if we could figure out what we want. For this particular case, I think you just need some place to store a pg_type - pg_proc mapping. I'm not exactly sure how to make that not a JSON-specific hack, since I certainly don't think we'd want to add a new catalog just for that. This was my initial proposal to have casts to ::json for all types. I backed out from this in favot of generic to_json(datum, indent) in order to support prettyprinting. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote: So given that do we do anything about this now, or wait till 9.3? I'd like the json support in 9.2 updated as follows I think it's too late to be entertaining proposals for such changes in 9.2. If we had concluded that the existing functions were actively wrong or a bad idea, I think that hard-coding postgresql text representation as our json representation without a possibility for the user tio easily fix it without rewriting foll xx_to_json() functions is borderline actively wrong. Can we at least have the xxx_to_json() functions try cast to json first and fall back to text if the cast fails. This would address my worst problem, all the rest can be easily defined in user functions. then of course we'd need to do something; but they are not, so we can just as well consider additions in the 9.3 cycle rather than now. I am not convinced that this proposal is fully baked yet, anyway; not to mention that right now we need to have our heads down on resolving the remaining open issues, not designing, implementing, and reviewing a pile of brand new code for json. By allowing developers just to define their own to_json(date) function we give them the power do decide which one to use. And if we honour search_path when looking up the to_json() functions, then they can even choose to have different conventions for different applications. This is not going to work anywhere near as nicely as you think. If somebody tries to define multiple to_json() functions that override a generic to_json(anyelement) one, he will start getting function is not unique parse failures. The parser will only successfully decide which function to call when the input data type exactly matches one of the specialized functions, which means you might as well not have the generic one at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net wrote: On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote: So given that do we do anything about this now, or wait till 9.3? I'd like the json support in 9.2 updated as follows I think it's too late to be entertaining proposals for such changes in 9.2. If we had concluded that the existing functions were actively wrong or a bad idea, I think that hard-coding postgresql text representation as our json representation without a possibility for the user tio easily fix it without rewriting foll xx_to_json() functions is borderline actively wrong. Can we at least have the xxx_to_json() functions try cast to json first and fall back to text if the cast fails. I think the idea that you can involve the casting machinery in this is misguided. sometextval::json has got to mean that sometextval is expected to be in the form of a syntactically correct JSON value - and NOT that we wrap it in a JSON string. We can have constructors for JSON, but they've got to be separate from the casting machinery. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
Robert Haas robertmh...@gmail.com writes: On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net wrote: Can we at least have the xxx_to_json() functions try cast to json first and fall back to text if the cast fails. I think the idea that you can involve the casting machinery in this is misguided. It is possible that that can be made to work, but it's a research project, not something to be crammed into 9.2 at the last possible minute. In any case, I really dislike the idea that array_to_json and row_to_json would contain two entirely different behaviors. Leave the extensibility ideas for a future to_json() function. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote: On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net wrote: On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote: So given that do we do anything about this now, or wait till 9.3? I'd like the json support in 9.2 updated as follows I think it's too late to be entertaining proposals for such changes in 9.2. If we had concluded that the existing functions were actively wrong or a bad idea, I think that hard-coding postgresql text representation as our json representation without a possibility for the user tio easily fix it without rewriting foll xx_to_json() functions is borderline actively wrong. Can we at least have the xxx_to_json() functions try cast to json first and fall back to text if the cast fails. I think the idea that you can involve the casting machinery in this is misguided. sometextval::json has got to mean that sometextval is expected to be in the form of a syntactically correct JSON value - and NOT that we wrap it in a JSON string. We can have constructors for JSON, but they've got to be separate from the casting machinery. on the contrary - the string representation of textual value a is a casting should _not_ neam syntax check, casting is by definition a conversion. if we cast text to int, we return value of type int , if we cast int to numeric(5,2) we return value of type numeric(5,2) why should casring to json work differntly ? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] remove dead ports?
On Fri, May 4, 2012 at 8:45 PM, Peter Eisentraut pete...@gmx.net wrote: On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote: On 3 May 2012 17:21, Bruce Momjian br...@momjian.us wrote: I think I was the only user left; I have never heard from a BSD/OS user in the past 5-7 years. I'm inclined to agree with Bruce. While it's not reasonable to assume that the lack of a BSD/OS user complaining on -general indicates that there are none, it's also not reasonable for them to expect us to support their operating system for 8 years after the original proprietary vendor. Better to not support BSD/OS than to supply a port that no one really has any confidence in. It's not as if we've ceased support in release branches. I'm not so much opposed to removing the port. I am more concerned about the manner in which it was done. The other ports I removed were known to not work anyway, for years, and there were at least several days of discussion. The bsdi case was removing a working port with less than 24 hours notice. There is enough time until 9.2-final for a BSD/OS user raise complaints. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?
-Original Message-
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-
ow...@postgresql.org] On Behalf Of Hannu Krosing
Sent: Friday, May 04, 2012 4:40 PM
To: Robert Haas
Cc: Tom Lane; Andrew Dunstan; PostgreSQL-development; Merlin Moncure
Subject: Re: [HACKERS] JSON in 9.2 - Could we have just one to_json()
function instead of two separate versions ?
On Fri, 2012-05-04 at 15:59 -0400, Robert Haas wrote:
On Fri, May 4, 2012 at 3:49 PM, Hannu Krosing ha...@krosing.net
wrote:
On Fri, 2012-05-04 at 09:52 -0400, Tom Lane wrote:
Hannu Krosing ha...@2ndquadrant.com writes:
On Wed, 2012-05-02 at 12:06 -0700, Andrew Dunstan wrote:
So given that do we do anything about this now, or wait till 9.3?
I'd like the json support in 9.2 updated as follows
I think it's too late to be entertaining proposals for such changes
in 9.2. If we had concluded that the existing functions were
actively wrong or a bad idea,
I think that hard-coding postgresql text representation as our
json representation without a possibility for the user tio easily
fix it without rewriting foll xx_to_json() functions is borderline
actively wrong.
Can we at least have the xxx_to_json() functions try cast to json
first and fall back to text if the cast fails.
I think the idea that you can involve the casting machinery in this is
misguided. sometextval::json has got to mean that sometextval is
expected to be in the form of a syntactically correct JSON value - and
NOT that we wrap it in a JSON string. We can have constructors for
JSON, but they've got to be separate from the casting machinery.
on the contrary - the string representation of textual value a is a
casting should _not_ neam syntax check, casting is by definition a conversion.
if we cast text to int, we return value of type int , if we cast int to
numeric(5,2) we return value of type numeric(5,2)
why should casring to json work differntly ?
What is the distinction between what you are thinking regarding JSON and this
example?
SELECT '1a'::integer;
SQL Error: ERROR: invalid input syntax for integer: 1a
LINE 1: SELECT '1a'::integer
As a user if I cast something to something else I want the result to be of the
correct type and deterministic; otherwise throw me some kind of invalid input
format exception (or syntax exception). Casting vs. Constructors is really a
meaningless distinction to a lay person. When I cast I do so by constructing a
new value using my existing value for input. When I use an explicit CAST I am
unable to supply additional parameters to configure the casting whereas a
constructor function gives me that possibility. But a constructor function
without any additional parameters is not semantically different than a cast.
I guess the concern to address is something like:
SELECT '{key: value}'::json OR SELECT '[1.25]'::json; Do you interpret this as
already being valid JSON and thus output object/array constructs (i.e., JSON
Text) or do you treat them as string literals and output scalars (i.e., JSON
Value). Even if you feel these are artificial constructs the concepts holds
that there may be ambiguous data that can be interpreted in multiple ways (this
applies even to function forms, though in the function form you could specify
which one you want to output using a separate DEFAULTed parameter).
I can see the primary use-case for JSON Value casting as being queries of the
following forms (since the record and array forms are going through the
record/array_to_json function):
SELECT COUNT(*)::json FROM table [WHERE ...];
SELECT single_col::json FROM table WHERE id = ?;
Where the single provided value can be sent directly back to the web-caller
JavaScript and used as-is because it is valid JSON. Though, honestly, both
SELECT to_json(single_col) and SELECT to_json(COUNT(*)) are equally usable so
any distinction between them is a pure technical issue to me.
Am I correct in assuming the following expected behavior (the forgive the
blatantly wrong syntax but you should get the point)?
RAISE NOTICE '%', SELECT 'A'::text = A
RAISE NOTICE '%', SELECT 'A'::json = A
David J
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] remove dead ports?
On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote: On tor, 2012-05-03 at 17:39 +0100, Peter Geoghegan wrote: On 3 May 2012 17:21, Bruce Momjian br...@momjian.us wrote: I think I was the only user left; I have never heard from a BSD/OS user in the past 5-7 years. I'm inclined to agree with Bruce. While it's not reasonable to assume that the lack of a BSD/OS user complaining on -general indicates that there are none, it's also not reasonable for them to expect us to support their operating system for 8 years after the original proprietary vendor. Better to not support BSD/OS than to supply a port that no one really has any confidence in. It's not as if we've ceased support in release branches. I'm not so much opposed to removing the port. I am more concerned about the manner in which it was done. The other ports I removed were known to not work anyway, for years, and there were at least several days of discussion. The bsdi case was removing a working port with less than 24 hours notice. Not sure where you got 24 hours: Tues http://archives.postgresql.org/pgsql-hackers/2012-05/msg00061.php Wed http://archives.postgresql.org/pgsql-general/2012-05/msg00060.php Thur http://archives.postgresql.org/pgsql-committers/2012-05/msg00023.php Basically, we have beta next week so I wanted to do it before then, and I have my head down doing the release notes, so I wanted to do it before I started that too. I kind of knew the bsdi answer before I even asked. If you are objecting to me short-circuiting this, I will revert the patch. If we can't short-circuiting thinks when we already know the answer, everyone's work will take more time. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)
On 5/3/12 2:54 PM, Josh Berkus wrote: (2) If logical transactions had been implemented as additions to the WAL stream, and Slony was using that, do you think they would still have been usable for this recovery? Quite possibly not. The key advantage that I see in londiste/slony replication is that your data stream has absolutely nothing to do with anything binary or internal to Postgres. That means that the only way corruption will travel from a master to a slave is if the corruption is in the actual fields being updated, and even that's not a given (ie: UPDATING a field to a completely new value would not propagate corruption even if the old value of the field was corrupted). So, embedding a logical stream into WAL is not inherently bad... what would be bad is if that logical stream was susceptible to corruption due to something like full page writes. Simply embedding the exact same info slony or londiste captures into the WAL should be fine (though likely defeats the purpose). Translating binary WAL data into DML statements would very likely allow corruption to travel from master to slave. -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] remove dead ports?
Bruce Momjian br...@momjian.us writes: On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote: I'm not so much opposed to removing the port. I am more concerned about the manner in which it was done. The other ports I removed were known to not work anyway, for years, and there were at least several days of discussion. The bsdi case was removing a working port with less than 24 hours notice. What's the grounds for asserting they were known not to work? Not actual testing, I assume. Basically, we have beta next week so I wanted to do it before then, and I have my head down doing the release notes, so I wanted to do it before I started that too. I kind of knew the bsdi answer before I even asked. If you are objecting to me short-circuiting this, I will revert the patch. If we can't short-circuiting thinks when we already know the answer, everyone's work will take more time. Leave it as-is. I agree with the upthread comment that we can revert the patch during beta (or even later than that), if anyone complains. Furthermore, I would want to insist that a complainer provide a buildfarm member as the price of us continuing to support an old uncommon platform. Otherwise the apparent support is hollow. The BSDI port was viable for us to support as long as Bruce was using it daily, but with that gone, we need somebody else to be testing it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] remove dead ports?
On Fri, May 04, 2012 at 06:25:24PM -0400, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Fri, May 04, 2012 at 08:45:10PM +0300, Peter Eisentraut wrote: I'm not so much opposed to removing the port. I am more concerned about the manner in which it was done. The other ports I removed were known to not work anyway, for years, and there were at least several days of discussion. The bsdi case was removing a working port with less than 24 hours notice. What's the grounds for asserting they were known not to work? Not actual testing, I assume. Basically, we have beta next week so I wanted to do it before then, and I have my head down doing the release notes, so I wanted to do it before I started that too. I kind of knew the bsdi answer before I even asked. If you are objecting to me short-circuiting this, I will revert the patch. If we can't short-circuiting thinks when we already know the answer, everyone's work will take more time. Leave it as-is. I agree with the upthread comment that we can revert the patch during beta (or even later than that), if anyone complains. Furthermore, I would want to insist that a complainer provide a buildfarm member as the price of us continuing to support an old uncommon platform. Otherwise the apparent support is hollow. The BSDI port was viable for us to support as long as Bruce was using it daily, but with that gone, we need somebody else to be testing it. Yes, it was an odd port that probably would have been removed five years ago if I had not been using it, which I am no longer. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Documentation for temp_file_limit
The new 9.2 GUC parameter temp_file_limit says it restricts temporary file usage per session, but it doesn't say what happens if a session needs to exceed that value --- it throws an error. Shouldn't we mention that? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
