Re: [HACKERS] Careful PL/Perl Release Not Required
On Thu, Feb 10, 2011 at 21:53, David E. Wheeler wrote:
> On Feb 10, 2011, at 5:28 PM, Alex Hunsaker wrote:
>> The other thing that changed is non UTF-8 databases now also get
>> character semantics. That is we convert from the database encoding
>> into utf8 and visa versa on output. That probably should be noted
>> somewhere...
>
> Oh. I see. And Oleg's database wasn't utf-8 then, I guess. I'll have to
> re-read the JSON docs, I guess. Erm…feh. Okay. I have to pass the false value
> to utf8() *now*. Okay, at least that's more consistent.
I'd like to quibble with you over this point if I may. :-)
Per perldoc: JSON::XS
"utf8" flag disabled
When "utf8" is disabled (the default), then
"encode"/"decode" generate and expect Unicode strings ...
So
- If you are on < 9.1 and a utf8 database you want to pass
utf8(false), as you have a Unicode string.
- If you are on < 9.1 and on a non utf8 database you would want to
pass utf8(false) as the string is *not* Unicode, its byte soup. Its in
some _other_ encoding say EUC_JP. You would need to decode() it into
Unicode first.
- If you are on 9.1 and a utf8 database you still want to pass
utf8(false) as the string is still unicode.
- if you are on 9.1 and a non utf8 database you want to pass
utf8(false) as the string is _now_ unicode.
So... it seems you always want to pass false. The only case I can
where you would want to pass true is you are on < 9.1 with a SQL_ASCII
database and you know for a fact the string represents a utf8 byte
sequence.
Or am I missing something obvious?
>> If you do have to change your semantics/functions, could you post an
>> example? I'd like to make sure its because you were hitting one of
>> those nasty corner cases and not something new is broken.
>
> I think that people who have non-utf-8 databases might be surprised.
Yeah, surprised it does the right thing and its actually usable now ;).
>>> This probably won't be that common, but Oleg, for example, will need to
>>> convert his fixed function from:
> No, he had to add the decode line, IIRC:
>
> CREATE OR REPLACE FUNCTION url_decode(Vkw varchar) RETURNS varchar AS $$
> use strict;
> use URI::Escape;
> utf8::decode($_[0]);
> return uri_unescape($_[0]); $$ LANGUAGE plperlu;
>
> Because uri_unescape() needs its argument to be decoded to Perl's internal
> form. On 9.1, it will be, so he won't need to call utf8::decode(). That is,
> in a latin-1 database:
Meh, no, not really. He will still need to call decode. The problem is
uri_unescape() does not assume an encoding on the URI. It could be
UTF-16 encoded for all it knows (UTF-8 is probably standard, but thats
not the point, it knows nothing about Unicode or encodings).
For example, lets say you have a latin-1 accented e "é" the byte
sequence is the one byte: 0xe9. If you were to uri_escape that you get
the 3 byte ascii string "%E9":
$ perl -E 'use URI::Escape; my $str = "\xe9"; say uri_escape($str)'
%E9
If you uri_unescape "%E9" you get 1 byte back with a hex value of 0xe9:
$ perl -E 'use URI::Escape; my $str = uri_unescape("%E9"); say
sprintf("chr: %s hex: %s, len: %s", $str, unpack("H*", $str), length
$str)'
chr: é hex: e9, len: 1
What if we want to uri_escape a UTF-16 accented e? Thats two hex bytes 0x00e9:
$ perl -E 'use URI::Escape; my $str = "\x00\xe9"; say uri_escape($str)'
%00%E9
What happens we uri_unescape that? Do we get back a Unicode string
that has one character? No. And why should we? How is uri_unescape
supposed to know what %00%E9 represent? All it knows is thats 2
separate bytes:
$ perl -E 'use URI::Escape; my $str = uri_unescape("%00%E9"); say
sprintf("chr: %s hex: %s, len: %s", $str, unpack("H*", $str), length
$str)'
chr: é hex: 00e9, len: 2
Now, lets say you want to uri_escape a utf8 accented e, thats the two
byte sequence: 0xc3 0xa9:
$ perl -E 'use URI::Escape; my $str = "\xc3\xa9"; say uri_escape($str)'
%C3%A9
Ok, what happens when we uri_unescape those?:
$ perl -E 'use URI::Escape; my $str = uri_unescape("%C3%A9"); say
sprintf("chr: %s hex: %s, len: %s", $str, unpack("H*", $str), length
$str)'
chr: é hex: c3a9, len: 2
So, plperl will also return 2 characters here.
In the the cited case he was passing "%C3%A9" to uri_unescape() and
expecting it to return 1 character. The additional utf8::decode() will
tell perl the string is in utf8 so it will then return 1 char. The
point being, decode is needed and with it, the function will work pre
and post 9.1.
In-fact on a latin-1 database it sure as heck better return two
characters, it would be a bug if it only returned 1 as that would mean
it would be treating a series of latin1 bytes as a series of utf8
bytes!
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error attribution in foreign scans
On Wed, Feb 09, 2011 at 10:55:05AM +0900, Itagaki Takahiro wrote: > On Mon, Feb 7, 2011 at 22:47, Heikki Linnakangas > wrote: > > On Mon, Feb 7, 2011 at 21:17, Noah Misch wrote: > >> The message does not show which foreign table yielded the error. ??We > >> could evade > >> the problem in this case by adding a file name to the error message in the > >> COPY > >> code, > > > Yeah, an error context callback like that makes sense. In the case of the > > file FDW, though, just including the filename in the error message seems > > even better. Especially if the error is directly related to failure in > > reading the file. > > What do you think about filenames in terms of security? We will allow > non-superusers to use existing foreign tables of file_fdw. > For reference, we hide some path settings in GUC variables. Comprehensively hiding the name from non-superusers is ideal, but it seems adequate to document that the name will not be kept secret. The superuser could always mask the name by creating a symbolic link in $PGDATA and referencing that in the foreign table configuration. > We also reconsider privilege of fdwoptions, umoptions, etc. They could > contain password or server-side path, but all users can retrieve the > values. It's an existing issue, but will be more serious in 9.1. This would be good to get right by 9.1 (not sure what "right" is, though). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 9:15 PM, Tom Lane wrote: > Hmm, you're right. The word bootstrap implies that we're starting from > nothing, which is exactly what we're *not* doing (starting from nothing > is the easy "clean install" case). By the same token, FROM NOTHING > isn't the right phrase either. An accurate description would be > something like FROM UNPACKAGED OBJECTS, but I'm not seriously proposing > that ... > > Other ideas anyone? Implicit was the closest I saw in the reserved word list, if you're limiting things to that list. If not then, erm, LEGACY? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
"David E. Wheeler" writes: > On Feb 10, 2011, at 7:05 PM, Tom Lane wrote: >> (I'm not wedded to the phrase "FROM OLD" in particular, but it does >> reuse already existing keywords. Also, maybe it'd be better to reserve >> a version string such as "old" or "bootstrap", so that the bootstrap >> script could be called something more legible like foo-bootstrap-1.0.sql.) > Well, it's not really a bootstrap, is it? FROM OLD is okay, though not great. > FROM BEFORE would be better. Or IMPLICIT? (It was implicitly an extension > before.) Or, hey, FROM NOTHING! :-) Hmm, you're right. The word bootstrap implies that we're starting from nothing, which is exactly what we're *not* doing (starting from nothing is the easy "clean install" case). By the same token, FROM NOTHING isn't the right phrase either. An accurate description would be something like FROM UNPACKAGED OBJECTS, but I'm not seriously proposing that ... Other ideas anyone? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 7:05 PM, Tom Lane wrote: > I like this because (a) it's one less step, and one less concept for > users to deal with, and (b) it's much harder to screw up. If you forget > FROM OLD when you needed it, the CREATE will fail with "object already > exists" errors. If you use FROM OLD when you shouldn't have, it will > fail with "object doesn't exist" errors. There's no way for the command > to apparently succeed while not actually creating the desired state. +1 > (I'm not wedded to the phrase "FROM OLD" in particular, but it does > reuse already existing keywords. Also, maybe it'd be better to reserve > a version string such as "old" or "bootstrap", so that the bootstrap > script could be called something more legible like foo-bootstrap-1.0.sql.) Well, it's not really a bootstrap, is it? FROM OLD is okay, though not great. FROM BEFORE would be better. Or IMPLICIT? (It was implicitly an extension before.) Or, hey, FROM NOTHING! :-) >> That reminds me (OT), it's currently impossible to write an uninstall script >> for a custom data type because of the circular dependency between a type and >> its I/O functions. There's no way around that sort of DROP EXTENSION >> CASCADE, is there? > > Yeah, DROP TYPE CASCADE is currently the accepted way to do that, and > it's definitely a tad risky in that you might zap more than just the > type and the I/O functions. But I don't feel a need to do anything > special to fix that, because grouping the type and the functions into > an extension will take care of the problem. You will not need to say > CASCADE unless you're actually wanting to delete objects outside the > extension. Fair enough. Thanks, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Careful PL/Perl Release Not Required
On Feb 10, 2011, at 5:28 PM, Alex Hunsaker wrote:
> Hrm? For UTF-8 databases, in practice, nothing should have changed--
> we already passed strings in as utf8. What I fixed was some corner
> cases where some strings did not always have character semantics. See
> The "Unicode Bug" and "Forcing Unicode in Perl" in perldoc perlunicode
> for the problem and more or less how I fixed it.
Uh…
try=# create function is_utf8(text) returns boolean language plperl AS
'utf8::is_utf8(shift)';
CREATE FUNCTION
try=# select is_utf8('whatever');
is_utf8
─
t
(1 row)
try=# select is_utf8(U&'\0441\043B\043E\043D');
is_utf8
─
t
(1 row)
Damn, I guess you're right. How did I miss that?
> The other thing that changed is non UTF-8 databases now also get
> character semantics. That is we convert from the database encoding
> into utf8 and visa versa on output. That probably should be noted
> somewhere...
Oh. I see. And Oleg's database wasn't utf-8 then, I guess. I'll have to re-read
the JSON docs, I guess. Erm…feh. Okay. I have to pass the false value to utf8()
*now*. Okay, at least that's more consistent.
> If you do have to change your semantics/functions, could you post an
> example? I'd like to make sure its because you were hitting one of
> those nasty corner cases and not something new is broken.
I think that people who have non-utf-8 databases might be surprised.
>> This probably won't be that common, but Oleg, for example, will need to
>> convert his fixed function from:
>> ...
>
> Well assuming he fixed his bug by encoding uri_unescape's output he
> should not have to do anything. IIRC the problem was basically double
> encoded utf8, not a postgres bug.
No, the problem was that the string was passed to his pl/perl function encoded
in utf-8. He added a line to decode it to Perl's internal form. Once he goes to
9.1, unless the database is SQL_ASCII, he can dump the decode() line. I think.
> [ he had %3A%4A or something, uri_decode() decodes that to _two_
> characters because _it_ knows nothing about utf8. so you would need to
> call utf8::decode() on the result to turn those two bytes into a
> character ]
No, he had to add the decode line, IIRC:
CREATE OR REPLACE FUNCTION url_decode(Vkw varchar) RETURNS varchar AS $$
use strict;
use URI::Escape;
utf8::decode($_[0]);
return uri_unescape($_[0]); $$ LANGUAGE plperlu;
Because uri_unescape() needs its argument to be decoded to Perl's internal
form. On 9.1, it will be, so he won't need to call utf8::decode(). That is, in
a latin-1 database:
latin=# create or replace function is_utf8(text) returns boolean language
plperl AS 'utf8::is_utf8(shift) ? 1 : 0';
CREATE FUNCTION
Time: 1.934 ms
latin=# select is_utf8('whatever');
is_utf8
─
f
(1 row)
That will change, if I understand correctly.
>> So this needs to be highlighted in the release notes: If a PL/Perl function
>> is currently relying on a parameter passed in bytes, it will >need to be
>> modified to deal with utf8 strings, instead.
>
> FYI Andrew did add some docs.
Yeah, I was thinking of the release notes. Those who have non-uft-8 databases
might be surprised if their PL/Perl functions expect strings to be passed as
bytes.
> Thanks for keeping a sharp eye out.
>
> [ P.S. This stuff is confusing as hell, im just glad I got a sucker to
> commit it *waves* at Andrew :-) ]
Heh, well done. Frankly, though, this stuff isn't *that* hard. It's Perl's
terminology that's really bad.
Best,
David
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Spread checkpoint sync
On Thu, Feb 10, 2011 at 10:30 PM, Greg Smith wrote: > 3) The existing write spreading code in the background writer needs to be > overhauled, too, before spreading the syncs around is going to give the > benefits I was hoping for. I've been thinking about this problem a bit. It strikes me that the whole notion of a background writer delay is probably wrong-headed. Instead of having fixed-length cycles, we might want to make the delay dependent on whether we're actually keeping up. So during each cycle, we decide how many buffers we want to clean, and we write 'em. Then we go to sleep. When we wake up again, we figure out whether we kept up. If the number of buffers we wrote during the prior cycle was more than the required number, then we'll sleep longer the next time, up to some maximum; if we we didn't write enough, we'll reduce the sleep. Along with this, we'd want to change the minimum rate of writing checkpoint buffers from 1 per cycle to 1 for every 200 ms, or something like that. We could even possibly have a system where backends wake the background writer up early if they notice that it's not keeping up, although it's not exactly clear what a good algorithm would be. Another thing that would be really nice is if backends could somehow let the background writer know when they're using a BufferAccessStrategy, and somehow convince the background writer to write those buffers out to the OS at top speed. > I want to make this problem go away, but as you can see spreading the sync > calls around isn't enough. I think the main write loop needs to get spread > out more, too, so that the background writer is trying to work at a more > reasonable pace. I am pleased I've been able to reproduce this painful > behavior at home using test data, because that much improves my odds of > being able to isolate its cause and test solutions. But it's a tricky > problem, and I'm certainly not going to fix it in the next week. Thanks for working on this. I hope we get a better handle on it for 9.2. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Spread checkpoint sync
Looks like it's time to close the book on this one for 9.1 development...the unfortunate results are at http://www.2ndquadrant.us/pgbench-results/index.htm Test set #12 is the one with spread sync I was hoping would turn out better than #9, the reference I was trying to improve on. TPS is about 5% slower on the scale=500 and 15% slower on the scale=1000 tests with sync spread out. Even worse, maximum latency went up a lot. I am convinced of a couple of things now: 1) Most of the benefit we were seeing from the original patch I submitted was simply from doing much better at absorbing fsync requests from backends while the checkpoint sync was running. The already committed fsync compaction patch effectively removes that problem though, to the extent it's possible to do so, making the remaining pieces here not as useful in its wake. 2) I need to start over testing here with something that isn't 100% write all of the time the way pgbench is. It's really hard to isolate out latency improvements when the test program guarantees all associated write caches will be completely filled at every moment. Also, I can't see any benefit if I make changes that improve performance only for readers with it, which is quite unrealistic relative to real-world workloads. 3) The existing write spreading code in the background writer needs to be overhauled, too, before spreading the syncs around is going to give the benefits I was hoping for. Given all that, I'm going to take my feedback and give the test server a much deserved break. I'm happy that the fsync compaction patch has made 9.1 much more tolerant of write-heavy loads than earlier versions, so it's not like no progress was made in this release. For anyone who wants more details here...the news on this spread sync implementation is not all bad. If you compare this result from HEAD, with scale=1000 and clients=256: http://www.2ndquadrant.us/pgbench-results/611/index.html Against its identically configured result with spread sync: http://www.2ndquadrant.us/pgbench-results/708/index.html There are actually significantly less times in the >2000 ms latency area. That shows up as a reduction in the 90th percentile latency figures I compute, and you can see it in the graph if you look at how much denser the points are in the 2000 - 4000 ms are on #611. But that's a pretty weak change. But the most disappointing part here relative to what I was hoping is what happens with bigger buffer caches. The main idea driving this approach was that it would enable larger values of shared_buffers without the checkpoint spikes being as bad. Test set #13 tries that out, by increasing shared_buffers from 256MB to 4GB, along with a big enough increase in checkpoint_segments to make most checkpoints time based. Not only did smaller scale TPS drop in half, all kinds of bad things happened to latency. Here's a sample of the sort of dysfunctional checkpoints that came out of that: 2011-02-10 02:41:17 EST: LOG: checkpoint starting: xlog 2011-02-10 02:53:15 EST: DEBUG: checkpoint sync: estimated segments=22 2011-02-10 02:53:15 EST: DEBUG: checkpoint sync: number=1 file=base/16384/16768 time=150.008 msec 2011-02-10 02:53:15 EST: DEBUG: checkpoint sync: number=2 file=base/16384/16749 time=0.002 msec 2011-02-10 02:53:15 EST: DEBUG: checkpoint sync: number=3 file=base/16384/16749_fsm time=0.001 msec 2011-02-10 02:53:23 EST: DEBUG: checkpoint sync: number=4 file=base/16384/16761 time=8014.102 msec 2011-02-10 02:53:23 EST: DEBUG: checkpoint sync: number=5 file=base/16384/16752_vm time=0.002 msec 2011-02-10 02:53:35 EST: DEBUG: checkpoint sync: number=6 file=base/16384/16761.5 time=11739.038 msec 2011-02-10 02:53:37 EST: DEBUG: checkpoint sync: number=7 file=base/16384/16761.6 time=2205.721 msec 2011-02-10 02:53:45 EST: DEBUG: checkpoint sync: number=8 file=base/16384/16761.2 time=8273.849 msec 2011-02-10 02:54:06 EST: DEBUG: checkpoint sync: number=9 file=base/16384/16766 time=20874.167 msec 2011-02-10 02:54:06 EST: DEBUG: checkpoint sync: number=10 file=base/16384/16762 time=0.002 msec 2011-02-10 02:54:08 EST: DEBUG: checkpoint sync: number=11 file=base/16384/16761.3 time=2440.441 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=12 file=base/16384/16766.1 time=635.839 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=13 file=base/16384/16752_fsm time=0.001 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=14 file=base/16384/16764 time=0.001 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=15 file=base/16384/16768_fsm time=0.001 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=16 file=base/16384/16761_vm time=0.001 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=17 file=base/16384/16761.4 time=150.702 msec 2011-02-10 02:54:09 EST: DEBUG: checkpoint sync: number=18 file=base/16384/16752 time=0.002 msec 2011-02-10 02:54:09 EST: DEB
Re: [HACKERS] [PERFORM] pgbench to the MAXINT
Greg Smith writes: > Poking around a bit more, I just discovered another possible approach is > to use erand48 instead of rand in pgbench, which is either provided by > the OS or emulated in src/port/erand48.c That's way more resolution > than needed here, given that 2^48 pgbench accounts would be a scale of > 2.8M, which makes for a database of about 42 petabytes. I think that might be a good idea --- it'd reduce the cross-platform variability of the results quite a bit, I suspect. random() is not to be trusted everywhere, but I think erand48 is pretty much the same wherever it exists at all (and src/port/ provides it elsewhere). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
"David E. Wheeler" writes: > On Feb 10, 2011, at 4:11 PM, Tom Lane wrote: >> It's to tell it to create an empty extension in preparation for >> absorbing pre-existing objects from an old-style contrib module. >> See what I mean? WRAPPER is not a useful keyword here. > Reminds me of creating a "shell type" so you can create I/O functions before > *actually* creating the type. I don't suppose "SHELL" is available. Actually, I was having second thoughts about that while at dinner. What is the value of separating the bootstrap-an-extension-from-old-objects operation into two steps? It's certainly not convenient for users, and I don't see that the intermediate state with an empty extension has any redeeming social value for developers either. (If you need such a thing, just make an empty creation script.) So: let's forget the concept of a special "null version" altogether, at least from the user's-eye viewpoint. Instead, the way to bootstrap from loose objects is something like CREATE EXTENSION foo [ VERSION '1.0' ] [ FROM OLD ] When you specify FROM OLD, this runs foo--1.0.sql instead of foo-1.0.sql as it normally would. As before, that script contains ALTER EXTENSION ADD commands instead of CREATE commands. I like this because (a) it's one less step, and one less concept for users to deal with, and (b) it's much harder to screw up. If you forget FROM OLD when you needed it, the CREATE will fail with "object already exists" errors. If you use FROM OLD when you shouldn't have, it will fail with "object doesn't exist" errors. There's no way for the command to apparently succeed while not actually creating the desired state. (I'm not wedded to the phrase "FROM OLD" in particular, but it does reuse already existing keywords. Also, maybe it'd be better to reserve a version string such as "old" or "bootstrap", so that the bootstrap script could be called something more legible like foo-bootstrap-1.0.sql.) > That reminds me (OT), it's currently impossible to write an uninstall script > for a custom data type because of the circular dependency between a type and > its I/O functions. There's no way around that sort of DROP EXTENSION CASCADE, > is there? Yeah, DROP TYPE CASCADE is currently the accepted way to do that, and it's definitely a tad risky in that you might zap more than just the type and the I/O functions. But I don't feel a need to do anything special to fix that, because grouping the type and the functions into an extension will take care of the problem. You will not need to say CASCADE unless you're actually wanting to delete objects outside the extension. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: [JDBC] Weird issues when reading UDT from stored function
On Tue, Jan 25, 2011 at 2:39 AM, Lukas Eder wrote: > So what you're suggesting is that the plpgsql code is causing the issues? > Are there any indications about how I could re-write this code? The > important thing for me is to have the aforementioned signature of the > plpgsql function with one UDT OUT parameter. Even if this is a bit awkward > in general, in this case, I don't mind rewriting the plpgsql function > content to create a workaround for this problem... Possibly something like address := (SELECT ...) rather than SELECT ... INTO address? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PERFORM] pgbench to the MAXINT
Stephen Frost wrote: Just wondering, did you consider just calling random() twice and smashing the result together..? I did. The problem is that even within the 32 bits that random() returns, it's not uniformly distributed. Combining two of them isn't really going to solve the distribution problem, just move it around. Some number of lower-order bits are less random than the others, and which they are is implementation dependent. Poking around a bit more, I just discovered another possible approach is to use erand48 instead of rand in pgbench, which is either provided by the OS or emulated in src/port/erand48.c That's way more resolution than needed here, given that 2^48 pgbench accounts would be a scale of 2.8M, which makes for a database of about 42 petabytes. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python invalidate functions with composite arguments
On Wed, Feb 9, 2011 at 02:09, Jan Urbański wrote:
> On 27/01/11 22:42, Jan Urbański wrote:
>> On 23/12/10 14:50, Jan Urbański wrote:
>>> Here's a patch implementing properly invalidating functions that have
>>> composite type arguments after the type changes, as mentioned in
>>> http://archives.postgresql.org/pgsql-hackers/2010-12/msg01991.php. It's
>>> an incremental patch on top of the plpython-refactor patch sent eariler.
>>
>> Updated to master.
>
> Again.
Looks good, it works as described, the code is clean and well
documented and it passes the added regression tests.
I took the liberty of looking at the other pls to see how they handled
this to find they don't cache them in the first place. For fun, I
hacked plpython to not cache to see if there was any performance
difference pre patch, post patch and in the non-cached cases. I
couldn't find any probably due to:
1) my simple test case (select
count(test_composite_table_input('(John, 100, "(10)")')) FROM
generate_series(1, 100);)
2) things being cached
3) cache invalidation having to do most of the work that the non
caching cache does. I think there is one or two more SearchSysCall's.
4) overhead from cassert
It seems a bit heavy handed to invalidate and remake the entire
plpython function whenever we hit this case. I think we could get away
with setting ->is_rowtype = 2 in PLy_procedure_valid() instead. I
suppose it should be fairly rare case anyway so... *shrug*.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Careful PL/Perl Release Not Required
On Thu, Feb 10, 2011 at 16:28, David E. Wheeler wrote: > Hackers, > > With regard to this (very welcome) commit: > >> commit 50d89d422f9c68a52a6964e5468e8eb4f90b1d95 >> Author: Andrew Dunstan >> Date: Sun Feb 6 17:29:26 2011 -0500 >> >> Force strings passed to and from plperl to be in UTF8 encoding. >> >> String are converted to UTF8 on the way into perl and to the >> database encoding on the way back. This avoids a number of >> observed anomalies, and ensures Perl a consistent view of the >> world. >> >> Some minor code cleanups are also accomplished. >> >> Alex Hunsaker, reviewed by Andy Colson. > > I just want to emphasize that this needs to be highlighted as a compatibility > change in the release notes. As an example, I currently have this code in > PGXN to process a TEXT param to a function: > > my $dist_meta = JSON::XS->new->utf8->decode(shift); > > After I upgrade to 9.0, I will have to change that to: > > my $dist_meta = JSON::XS->new->utf8(0)->decode(shift); Hrm? For UTF-8 databases, in practice, nothing should have changed-- we already passed strings in as utf8. What I fixed was some corner cases where some strings did not always have character semantics. See The "Unicode Bug" and "Forcing Unicode in Perl" in perldoc perlunicode for the problem and more or less how I fixed it. The other thing that changed is non UTF-8 databases now also get character semantics. That is we convert from the database encoding into utf8 and visa versa on output. That probably should be noted somewhere... If you do have to change your semantics/functions, could you post an example? I'd like to make sure its because you were hitting one of those nasty corner cases and not something new is broken. > This probably won't be that common, but Oleg, for example, will need to > convert his fixed function from: > ... Well assuming he fixed his bug by encoding uri_unescape's output he should not have to do anything. IIRC the problem was basically double encoded utf8, not a postgres bug. [ he had %3A%4A or something, uri_decode() decodes that to _two_ characters because _it_ knows nothing about utf8. so you would need to call utf8::decode() on the result to turn those two bytes into a character ] > So this needs to be highlighted in the release notes: If a PL/Perl function > is currently relying on a parameter passed in bytes, it will >need to be > modified to deal with utf8 strings, instead. FYI Andrew did add some docs. Thanks for keeping a sharp eye out. [ P.S. This stuff is confusing as hell, im just glad I got a sucker to commit it *waves* at Andrew :-) ] -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sorting. When?
On Thu, Feb 10, 2011 at 6:21 PM, Nicolas Barbier wrote: > 2011/2/10 mac_man2...@yahoo.it : > >> Which operations invoke the sorting algorithms implemented in the sorting >> module (tuplesort.c) ? >> Of course, one of those operations invoking sorting is the ORDER BY clause >> and the DISTINCT too. >> >> Moreover, the Merge Join should be implemented invoking sorting. >> >> Is there any other operation invoking sorting? > > AFAIK, all set operators except for UNION ALL. (I am probably missing > a whole boatload of other things.) Merge joins don't necessarily involve a sort - you could do a merge over a pair of index scans, for example. Set operations can be implemented using hashing or sorting, too. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 6:50 PM, Tom Lane wrote: > Since we've agreed that there should be a version-to-install option > in CREATE EXTENSION, it seems to me that a workable solution is to > have a special convention for an "empty extension" version name. > Let's suppose that we choose the empty string as this reserved version > name. Then you would write > > CREATE EXTENSION foo VERSION '' [ SCHEMA whatever ]; > > as the equivalent of CREATE WRAPPER EXTENSION. This would create the > extension's entry in pg_extension, but not run any script, and the > extension would initially have no members. After that you could do > > ALTER EXTENSION foo UPGRADE TO '1.0'; > > and this would run the upgrade script "foo--1.0.sql", which would most > likely consist of just "ALTER EXTENSION foo ADD object" commands to > absorb the objects from the old-style contrib module into the extension. You don't really need any core support for this at all. People could simply ship an empty file called foo-.sql, and then foo--1.0.sql to upgrade to version 1.0. (Or if you want to pick 0 or bootstrap or null to represent the loose object situation, that works too.) > Third, I'm also not thrilled with the syntax "ALTER EXTENSION foo > UPGRADE". UPGRADE isn't an existing keyword (note that VERSION is). > And I don't see any strong reason to assume that the version change > is an "upgrade". Authors might well choose to support sidegrades or > downgrades, especially with experimental modules. I suggest either > > ALTER EXTENSION foo UPDATE [ TO 'version' ] > > ALTER EXTENSION foo VERSION [ 'version' ] > > the main excuse for the latter being that it's closer to the comparable > syntax in CREATE EXTENSION. > > OK, that's enough bikeshedding for today ... > > Comments? Generally, +1. Like David, I prefer the UPDATE syntax. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] SSI bug?
hi, it seems that PredicateLockTupleRowVersionLink sometimes create a loop of targets (it founds an existing 'newtarget' whose nextVersionOfRow chain points to the 'oldtarget') and it later causes CheckTargetForConflictsIn loop forever. YAMAMOTO Takashi -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 4:11 PM, Tom Lane wrote: > It's to tell it to create an empty extension in preparation for > absorbing pre-existing objects from an old-style contrib module. > See what I mean? WRAPPER is not a useful keyword here. Reminds me of creating a "shell type" so you can create I/O functions before *actually* creating the type. I don't suppose "SHELL" is available. That reminds me (OT), it's currently impossible to write an uninstall script for a custom data type because of the circular dependency between a type and its I/O functions. There's no way around that sort of DROP EXTENSION CASCADE, is there? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
"David E. Wheeler" writes: > On Feb 10, 2011, at 3:50 PM, Tom Lane wrote: >> First off, I don't much care for the name "CREATE WRAPPER EXTENSION". > What's the WRAPPER bit for? I've forgotten. It's to tell it to create an empty extension in preparation for absorbing pre-existing objects from an old-style contrib module. See what I mean? WRAPPER is not a useful keyword here. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 3:50 PM, Tom Lane wrote: > It seems that we've mostly got consensus on the ideas of having a separate > script file for each installable version of an extension, and for each > basic version-upgrade action, with version numbers embedded in the file > names so that the control files don't need to be involved in identifying > what's what. And the core system is expected to be able to figure out how > to chain upgrade scripts together when necessary. Therefore, I'm now > ready to start kibitzing on syntax details :-) Damn, I thought you were going to get rid of the control file there for a sec (in favor of Makefile variables). ;-P > First off, I don't much care for the name "CREATE WRAPPER EXTENSION". > WRAPPER is a misnomer in this case --- it's not wrapping anything. > I think Dimitri stated that he chose WRAPPER just because it was an > already existing keyword, but that isn't much of an excuse. What's the WRAPPER bit for? I've forgotten. > One minor objection to this idea is that "foo--1.0.sql" looks more like a > typo than anything else. We could alternatively decide that the special > reserved version name is '0', so that bootstrap script names look like > "foo-0-1.0.sql". But if you don't want to have any built-in assumptions > about what version names mean, you might not like that idea. I'm fine with either of these. "foo-0-1.0.sql" might lead to fewer questions being asked. But I otherwise have no preference. > Third, I'm also not thrilled with the syntax "ALTER EXTENSION foo > UPGRADE". UPGRADE isn't an existing keyword (note that VERSION is). > And I don't see any strong reason to assume that the version change > is an "upgrade". Authors might well choose to support sidegrades or > downgrades, especially with experimental modules. I suggest either > > ALTER EXTENSION foo UPDATE [ TO 'version' ] > > ALTER EXTENSION foo VERSION [ 'version' ] > > the main excuse for the latter being that it's closer to the comparable > syntax in CREATE EXTENSION. > > OK, that's enough bikeshedding for today ... The former reads much more clearly to me. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
It seems that we've mostly got consensus on the ideas of having a separate script file for each installable version of an extension, and for each basic version-upgrade action, with version numbers embedded in the file names so that the control files don't need to be involved in identifying what's what. And the core system is expected to be able to figure out how to chain upgrade scripts together when necessary. Therefore, I'm now ready to start kibitzing on syntax details :-) First off, I don't much care for the name "CREATE WRAPPER EXTENSION". WRAPPER is a misnomer in this case --- it's not wrapping anything. I think Dimitri stated that he chose WRAPPER just because it was an already existing keyword, but that isn't much of an excuse. Second, I don't like anything about the term "null version" for the case of bootstrapping from an old-style contrib module. Null implies unknown, which isn't what we've got here --- the upgrade script is going to make very definite assumptions about what's already there. Also, given that we're trying to minimize assumptions about what the version strings mean, reserving the string "null" for this purpose doesn't seem like a good idea. I *definitely* don't want to allow pg_extension.extversion to ever be a real SQL NULL. Since we've agreed that there should be a version-to-install option in CREATE EXTENSION, it seems to me that a workable solution is to have a special convention for an "empty extension" version name. Let's suppose that we choose the empty string as this reserved version name. Then you would write CREATE EXTENSION foo VERSION '' [ SCHEMA whatever ]; as the equivalent of CREATE WRAPPER EXTENSION. This would create the extension's entry in pg_extension, but not run any script, and the extension would initially have no members. After that you could do ALTER EXTENSION foo UPGRADE TO '1.0'; and this would run the upgrade script "foo--1.0.sql", which would most likely consist of just "ALTER EXTENSION foo ADD object" commands to absorb the objects from the old-style contrib module into the extension. One minor objection to this idea is that "foo--1.0.sql" looks more like a typo than anything else. We could alternatively decide that the special reserved version name is '0', so that bootstrap script names look like "foo-0-1.0.sql". But if you don't want to have any built-in assumptions about what version names mean, you might not like that idea. Third, I'm also not thrilled with the syntax "ALTER EXTENSION foo UPGRADE". UPGRADE isn't an existing keyword (note that VERSION is). And I don't see any strong reason to assume that the version change is an "upgrade". Authors might well choose to support sidegrades or downgrades, especially with experimental modules. I suggest either ALTER EXTENSION foo UPDATE [ TO 'version' ] ALTER EXTENSION foo VERSION [ 'version' ] the main excuse for the latter being that it's closer to the comparable syntax in CREATE EXTENSION. OK, that's enough bikeshedding for today ... Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Debian readline/libedit breakage
On 02/10/2011 06:36 PM, Stephen Frost wrote: * Daniel Farina (drfar...@acm.org) wrote: I have to admit, this change in debian packaging -- which I have noticed, and not a little -- makes my hands angry. I considered looking into the problem, but were I doing it, I would have considered teaching psql to support NSS or GnuTLS as totally viable alternatives to this problem, as to keep readline. Supporting GnuTLS would be really nice.. That's how we addressed the same issue w/ OpenLDAP (I was involved in that as a Debian co-maintainer). GnuTLS has limitations too, but in the end, I find those more palatable (and the GnuTLS maintainer is certainly willing to work on improving it) than dropping readline. :/ Strikes me as a lot of work to buy nothing much. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Debian readline/libedit breakage
* Daniel Farina (drfar...@acm.org) wrote: > I have to admit, this change in debian packaging -- which I have > noticed, and not a little -- makes my hands angry. I considered > looking into the problem, but were I doing it, I would have considered > teaching psql to support NSS or GnuTLS as totally viable alternatives > to this problem, as to keep readline. Supporting GnuTLS would be really nice.. That's how we addressed the same issue w/ OpenLDAP (I was involved in that as a Debian co-maintainer). GnuTLS has limitations too, but in the end, I find those more palatable (and the GnuTLS maintainer is certainly willing to work on improving it) than dropping readline. :/ THanks, Stephen signature.asc Description: Digital signature
[HACKERS] Careful PL/Perl Release Not Required
Hackers, With regard to this (very welcome) commit: > commit 50d89d422f9c68a52a6964e5468e8eb4f90b1d95 > Author: Andrew Dunstan > Date: Sun Feb 6 17:29:26 2011 -0500 > > Force strings passed to and from plperl to be in UTF8 encoding. > > String are converted to UTF8 on the way into perl and to the > database encoding on the way back. This avoids a number of > observed anomalies, and ensures Perl a consistent view of the > world. > > Some minor code cleanups are also accomplished. > > Alex Hunsaker, reviewed by Andy Colson. I just want to emphasize that this needs to be highlighted as a compatibility change in the release notes. As an example, I currently have this code in PGXN to process a TEXT param to a function: my $dist_meta = JSON::XS->new->utf8->decode(shift); After I upgrade to 9.0, I will have to change that to: my $dist_meta = JSON::XS->new->utf8(0)->decode(shift); The upshot is that in those cases where the raw bytes are what's actually wanted, users will have to modify their functions to turn off the utf8 flag. This probably won't be that common, but Oleg, for example, will need to convert his fixed function from: > CREATE OR REPLACE FUNCTION url_decode(Vkw varchar) RETURNS varchar AS $$ > use strict; > use URI::Escape; > utf8::decode($_[0]); > return uri_unescape($_[0]); $$ LANGUAGE plperlu; To: > CREATE OR REPLACE FUNCTION url_decode(Vkw varchar) RETURNS varchar AS $$ > use strict; > use URI::Escape; > return uri_unescape($_[0]); $$ LANGUAGE plperlu; So this needs to be highlighted in the release notes: If a PL/Perl function is currently relying on a parameter passed in bytes, it will need to be modified to deal with utf8 strings, instead. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sorting. When?
2011/2/10 mac_man2...@yahoo.it : > Which operations invoke the sorting algorithms implemented in the sorting > module (tuplesort.c) ? > Of course, one of those operations invoking sorting is the ORDER BY clause > and the DISTINCT too. > > Moreover, the Merge Join should be implemented invoking sorting. > > Is there any other operation invoking sorting? AFAIK, all set operators except for UNION ALL. (I am probably missing a whole boatload of other things.) Nicolas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Debian readline/libedit breakage
Excerpts from Joshua D. Drake's message of jue feb 10 19:34:31 -0300 2011: > Hello, > > Per: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109 O, the joy of having people mess up with legal stuff that nobody cares about creating endless work for everyone. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Debian readline/libedit breakage
Andrew Dunstan writes: > I'll be happy if you do, but why haven't I haven't noticed, say, RedHat > taking this line? Less narrow-minded interpretation of GPL requirements, perhaps. (And yes, we have real lawyers on staff considering these issues.) libedit is a long way from being ready to replace readline, much as one could wish it otherwise. If Debian want to shoot themselves in the foot like that, we can't stop them, but neither should we be devoting our project resources to fixing libedit. (I have seen some noise recently on the Fedora lists about putting work into libedit, so maybe something good will come of that. I'm just not ready to define it as my/our problem.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Dimitri Fontaine writes: > Tom Lane writes: >> I think we should embed the version number in the script file name, > What I don't like in that is that this restrict what the version strings > can look like. In debian for example it's pretty common to use the ~ > separator, because 1.0~alpha1 < 1.0~beta < 1.0 with their sorting rules. > And this trick won't work on windows filenames, AFAIK. That's one > reason why I've wanted to stay away from having the version number > strings encoded into the filename in the first place. Well, yeah, but if you accept the principle that there should be a separate script file for each version and update combination, you're pretty much going to have to embed the version strings into the filenames to keep your sanity. My feeling about this is that we should recommend that version identifiers be limited to ASCII letters, digits, dots, and underscore, but assume that extension authors are adults and can grasp the risks of using other characters. We should not be in the business of trying to force authors to write portable code whether they want to or not. > But if you get to sorting rules of version strings, you have to define > them properly and impose them to users. I think we've now converged on the agreement that we don't need to use anything but equality checks. So it doesn't matter how the author thinks the strings sort --- the upgrade scripts he provides define what can follow what, and that's all we need to know. > That way a SQL query can check if there's a new version available on > your system. That's useful in some places to use as a monitoring alert > coupled with nagios. The sysadmin team does the apt-get install part of > the job and then the DBA team is paged to go upgrade the extensions in > the databases, or shut the alarm somehow. Well, you could look to see if there is a script that can update your current version to something else. The existing pg_available_extensions view needs to be rethought a bit, probably, but I'm not sure how. >> So, concrete proposal is to enforce the "extension-version.sql" and >> "extension-oldversion-newversion.sql" naming rules for scripts, which >> means getting rid of the script name parameter in control files. > Well, just for the record, we could extend the script property to be a > key value thing that pairs a version string with an upgrade script > name. Yeah, but that doesn't get you away from having to name the script files somehow, and it isn't going to be pleasant for anybody to use a naming convention that isn't basically embedding the version numbers. We could argue about details like whether dash is the best separator, but that's pretty far down the list of important things. >> "version" parameter should be renamed to something like "current_version" >> or "default_version". We also have to be wary of whether any other > I can already hear people wanting version aliases instead. We could > support e.g. 4 or 5 aliases like 'stable', 'support', 'alpha', 'beta' > and maybe 'experimental'. Then rather than defining "current_version" > authors would define any set of those keywords here, and CREATE > EXTENSION and ALTER EXTENSION would by default only care for > resp. 'stable' and 'support'. Hmm. That might be worth doing, but let's leave it for later when we find out how much demand there really is. It does strike me that what we ought to call the default-version parameter is just "default", since that would fit in reasonably well with such an extension later. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Debian readline/libedit breakage
On 02/10/2011 05:34 PM, Joshua D. Drake wrote: Hello, Per: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109 It seems we may have a problem to consider. As far as I know, we are the only major platform that supports libedit but our default is readline. Unfortunately readline is not compatible with OpenSSL (apparently?) licensing. This seems that it may be a problem for us considering the pre-package builds we do. What does everyone think? Should we work on getting libedit up to snuff? I'll be happy if you do, but why haven't I haven't noticed, say, RedHat taking this line? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Debian readline/libedit breakage
On Thu, Feb 10, 2011 at 2:34 PM, Joshua D. Drake wrote: > Hello, > > Per: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109 > > It seems we may have a problem to consider. As far as I know, we are the > only major platform that supports libedit but our default is readline. > Unfortunately readline is not compatible with OpenSSL (apparently?) > licensing. > > This seems that it may be a problem for us considering the pre-package > builds we do. > > What does everyone think? Should we work on getting libedit up to snuff? I have to admit, this change in debian packaging -- which I have noticed, and not a little -- makes my hands angry. I considered looking into the problem, but were I doing it, I would have considered teaching psql to support NSS or GnuTLS as totally viable alternatives to this problem, as to keep readline. -- fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
>> I don't see how that affects my point? You can spell "1.0" as "0.1" >> and "1.1" as "0.2" if you like that kind of numbering, but I don't >> see that that has any real impact. At the end of the day an author is >> going to crank out a series of releases, and if he cares about people >> using those releases for production, he's going to have to provide at >> least a upgrade script to move an existing database from release N to >> release N+1. > > Yeah, but given a rapidly-developing extension, that could create a lot of > extra work. I don't know that there's much of a way around that, other than > concatenating files to build migration scripts from parts (perhaps via `Make` > as dim suggested). But it can get complicated pretty fast. My desire here is > to keep the barrier to creating PostgreSQL extensions as low as is reasonably > possible. I assume this has already been discussed and rejected (or it wouldn't still be an issue), but what's wrong with the equivalent of \i in the successive .sql upgrade files? Or is the server running the scripts itself and no equivalent include feature exists in raw sql? Regards, David -- David Christensen End Point Corporation da...@endpoint.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SPI_exec doesn't return proc context (on 9.1)
Alvaro Herrera writes: > I wrote such a patch some time ago but never applied it ... > http://archives.postgresql.org/message-id/20091116162531.ga3...@alvh.no-ip.org > I still wonder if it's useful enough to be applied. Would it solve your > use case? Needs to be fixed to behave sanely for the typbyval case (as implemented, it's depending on the caller to check that). Otherwise, seems pretty reasonable. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Sorting. When?
Hi. Which operations invoke the sorting algorithms implemented in the sorting module (tuplesort.c) ? Of course, one of those operations invoking sorting is the ORDER BY clause and the DISTINCT too. Moreover, the Merge Join should be implemented invoking sorting. Is there any other operation invoking sorting? Thanks. Regards. Fava -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
David Christensen writes: > I assume this has already been discussed and rejected (or it wouldn't still > be an issue), but what's wrong with the equivalent of \i in the successive > .sql upgrade files? Or is the server running the scripts itself and no > equivalent include feature exists in raw sql? The latter. It wouldn't be that hard to invent something that would pull in another file, but there are some issues concerning how you figure out where to look for the file. In any case, if we go down that path, we're still putting the burden on the extension author to maintain a pile of little bitty script files -- a task that will get quite un-fun once you have dozens of active versions. Automatically applying the files in sequence should be a lot more convenient and less bug-prone. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Debian readline/libedit breakage
Hello, Per: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109 It seems we may have a problem to consider. As far as I know, we are the only major platform that supports libedit but our default is readline. Unfortunately readline is not compatible with OpenSSL (apparently?) licensing. This seems that it may be a problem for us considering the pre-package builds we do. What does everyone think? Should we work on getting libedit up to snuff? JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SPI_exec doesn't return proc context (on 9.1)
Excerpts from Pavel Stehule's message of sáb ene 29 16:56:40 -0300 2011: > 2011/1/29 Tom Lane : > > The less crocky way to do that is to use SPI_palloc() for something that > > should be allocated in the outer context. > > I understand. Is there some way, where I can use a cstring_to_text > function? There isn't simple way to get a saveCtx. > > some like SPI_copyDatum ... ? I wrote such a patch some time ago but never applied it ... http://archives.postgresql.org/message-id/20091116162531.ga3...@alvh.no-ip.org I still wonder if it's useful enough to be applied. Would it solve your use case? -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove more SGML tabs.
On Thu, Feb 10, 2011 at 11:29:43AM -0300, Alvaro Herrera wrote: > Excerpts from Peter Eisentraut's message of jue feb 10 07:58:16 -0300 2011: > > > One thing I was thinking of was that we could add a global make > > maintainer-check target (a name I picked up from other projects) > > which would run various source code sanity checks. Besides the > > SGML tabs issue, my favourite would be duplicate_oids. Maybe if > > we could find a third use case, we'd have a quorum for > > implementing this. > > The c++ headers thing? Or the compiles-in-isolation test for > headers? Both sound reasonable, given the number of times they come up and the ease of checking them mechanically. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove more SGML tabs.
On Thu, Feb 10, 2011 at 09:16:09PM +0200, Peter Eisentraut wrote: > On tor, 2011-02-10 at 10:40 -0800, David Fetter wrote: > > I think all such checks belong in .git/hooks/pre-commit, and need > > to be as cross-platform as needed for committers. Would a > > *n*x-based version do for a start? > > I think as a matter of principle, the only things that belongs into > git hooks are things that relate to the repository itself, such as > commit metadata (author checks, commit message spell checks, etc.). > Anything that relates to the source belongs into the source. So on that principle, should "make check" do these checks? Just to put another viewpoint out there, the reason that such hooks exist is precisely to enable people to do deeper checks than commit metadata. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Dimitri Fontaine writes: > "David E. Wheeler" writes: >> Yes, I think that this is a great solution. I only have to create on >> upgrade script for each release, and I don't have to worry about >> concatenating anything or be required to change my versioning >> algorithm. > You still have to make sure that the C code remains compatible with any > intermediate release, for the whole life of your extension. But I agree > that it's way better than what we had before. What you have to do is make sure the C code remains compatible with any version you are shipping an install or upgrade script for. Once you decide that versions before, say, 2.0 are dead as doornails, you remove all the older scripts, and you can delete the .so infrastructure for them too. For example, suppose I have foobar 1.5 installed on my system. If you are shipping a package that includes foobar-1.5-2.0.sql, I should reasonably expect that I can install that package first and upgrade the extension afterwards. If you aren't shipping any script that claims it can upgrade from 1.5, you don't need to provide .so compatibility either --- it's clear that I have to upgrade first and install your newer package after. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 9:38 PM, Tom Lane wrote: >> Well, the difference is that loose objects are just on my system, >> whereas extensions are supposed to work on anybody's system. I'm not >> clear that it's possible to write an extension that depends on a >> relocatable extension in a sensible way. If it is, objection >> withdrawn. > > I don't deny that there are risks here. But I think the value of being > able to move an extension when it is safe outweighs the difficulty that > sometimes it isn't safe. I think we can leave making it safer as a > topic for future investigation. Personally, I'ld rather be able to install the *same* extension/version in different schemas at the same time then move an extension from 1 schema to another, although I have no problems with extensions moving out under a function's foot (just like loose objects). a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 1:38 PM, Tom Lane wrote: > I don't deny that there are risks here. But I think the value of being > able to move an extension when it is safe outweighs the difficulty that > sometimes it isn't safe. I think we can leave making it safer as a > topic for future investigation. > > Dimitri did suggest treating an extension as nonrelocatable if there is > any other extension installed that depends on it. But that seems like > more of a kluge than a nice solution, primarily because it does nothing > for the loose-object risks. I'd rather just document that moving an > extension post-installation might break things, and leave it at that for > now. +1 David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > On Thu, Feb 10, 2011 at 3:52 PM, Tom Lane wrote: >> Again, it's not really any different from the case where the dependent >> objects are "loose" rather than members of an extension. > Well, the difference is that loose objects are just on my system, > whereas extensions are supposed to work on anybody's system. I'm not > clear that it's possible to write an extension that depends on a > relocatable extension in a sensible way. If it is, objection > withdrawn. I don't deny that there are risks here. But I think the value of being able to move an extension when it is safe outweighs the difficulty that sometimes it isn't safe. I think we can leave making it safer as a topic for future investigation. Dimitri did suggest treating an extension as nonrelocatable if there is any other extension installed that depends on it. But that seems like more of a kluge than a nice solution, primarily because it does nothing for the loose-object risks. I'd rather just document that moving an extension post-installation might break things, and leave it at that for now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
"David E. Wheeler" writes: > Yes, I think that this is a great solution. I only have to create on > upgrade script for each release, and I don't have to worry about > concatenating anything or be required to change my versioning > algorithm. You still have to make sure that the C code remains compatible with any intermediate release, for the whole life of your extension. But I agree that it's way better than what we had before. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > Well, the difference is that loose objects are just on my system, > whereas extensions are supposed to work on anybody's system. I'm not > clear that it's possible to write an extension that depends on a > relocatable extension in a sensible way. If it is, objection > withdrawn. I proposed that in this case, we bypass the relocatable property and just have the system work out that reverse dependencies make all those extensions not relocatable. Tom said that he does not see the point in trying to limit this foot gun power. -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] arrays as pl/perl input arguments [PATCH]
On Feb 10, 2011, at 9:44 PM, Andrew Dunstan wrote: > > > On 02/10/2011 08:15 AM, Alexey Klyukin wrote: >> On Feb 9, 2011, at 9:28 PM, Alex Hunsaker wrote: >> >>> On Wed, Feb 9, 2011 at 08:24, Alexey Klyukin >>> wrote: What was actually broken in encode_array_literal support of composite types (it converted perl hashes to the literal composite-type constants, expanding nested arrays along the way) ? I think it would be a useful extension of the existing encode_array_literal. >>> Yeah, It does not work because it did not take into account the order >>> of composite columns. It always put them alphabetically by column >>> name. To do it properly we would need to pass in a typid or a column >>> order or something. Ideally we could expose the new >>> plperl_array_to_datum() to plperl functions in some manner. >> Damn, right. Each perl hash corresponds to multiple composite types, >> different >> by the order of the type elements. Passing the typid sounds like a fair >> requirement (and if it's missing we could assume that the order of columns in >> composites doesn't matter to the caller). >> >> Let me try implementing that as an XS interface to plperl_array_to_datum. > > > Are you intending this as a completion of the current patch or as 9.2 work? > If the former you need to send it in real fast. I'd like to extend the current patch, going to post the update by tomorrow. /A -- Alexey Klyukin The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python custom exceptions for SPI
On 11-02-10 03:13 PM, Jan Urbański wrote: On 10/02/11 20:24, Peter Eisentraut wrote: Here is the rest of my review. Submission Review --- Patch applies cleanly. Documentation is still outstanding but Jan has promised it soon. Usability Review --- We don't have this for plpython, that we have a similar idea with plpgsql. I think this feature is useful and worth having. The CamelCase naming of the exceptions is consistent with how the built-in python exceptions are named (camel case). Feature Test --- I did basic testing of the feature (catching a few exception types thrown from both direct SQL and prepared statements) and the feature worked as expected. Performance Impact The impact of mapping error codes to exception types shouldn't come into play unless an SPI error is returned and with the hash it should still be minimal. Code Review - Ideally char * members of ExceptionMap would be const, but since many versions of python take a non-const value to PyErr_NewException that won't work :( After you search the for an exception in the hash you have: /* We really should find it, but just in case have a fallback */ Assert(entry != NULL); exc = entry ? entry->exc : PLy_exc_spi_error; I'm not sure the assert is needed here. Just falling back to the exception type seems reasonable and more desirable than an assert if showhow a new exception gets missed from the list. I don't feel that strongly on this. line 3575: PLy_elog(ERROR, "Failed to add the spiexceptions module"); "Failed" should be "failed" Other than that the patch looks fine to me. Updated again. Why do the error messages print spiexceptions.SyntaxError instead of plpy.spiexceptions.SyntaxError? Is this intentional or just the way it comes out of Python? That's how traceback.format_exception() works IIRC, which is what the Python interpreter uses and what PL/Python mimicks in PLy_traceback. Please add some documentation. Not a list of all exceptions, but at least a paragraph that various kinds of specific exceptions may be generated, what package and module they are in, and how they relate. Sure, Steve already asked for docs in another thread, and I'm writing them. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 1:22 PM, Tom Lane wrote: > Well, okay, let's go with that plan then. If we don't need to assume > anything more than equality of version names being meaningful, I think > chaining update scripts automatically should solve most of the > complaints here. People who really want to maintain shortcut scripts > still could, but I think it'd be an unusual case. Yes, I think that this is a great solution. I only have to create on upgrade script for each release, and I don't have to worry about concatenating anything or be required to change my versioning algorithm. +1 Finally, a solution! Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Revised patches to add table function support to PL/Tcl (TODO item)
On 02/08/2011 08:37 PM, Andrew Dunstan wrote: On 02/07/2011 11:30 PM, Robert Haas wrote: On Tue, Dec 28, 2010 at 9:23 PM, Karl Lehenbauer wrote: On Dec 28, 2010, at 7:29 PM, Tom Lane wrote: This patch appears to be changing a whole lot of stuff that in fact pg_indent has never changed, so there's something wrong with the way you are doing it. It looks like a bad typedef list from here. You were right, Tom. The problem was that typedefs "pltcl_interp_desc", "pltcl_proc_key", and "pltcl_proc_ptr" weren't in src/tools/pgindent/typedefs.list. After adding them (and building and installing the netbsd-based, patched indent), pgindent only changes a handful of lines. pltcl-karl-try3-1-of-3-pgindent.patch patches typedefs.list with the three missing typedefs and pltcl.c with the small changes made by pgindent (it shifted some embedded comments left within their lines, mainly). As before, but "try3" now, pltcl-karl-try3-2-of-3-objects.patch converts pltcl.c to use the "Tcl objects" C API. And as before, but "try3" now, pltcl-karl-try3-3-of-3-setof.patch adds returning record and SETOF record. This patch did not get reviewed, because the person who originally planned to review it had a hardware failure that prevented him from doing so. Can anyone pick this up? I will have a look at it. As promised I have had a look. The first point is that it doesn't have any documentation at all. The second is that it doesn't appear from a my admittedly short look to support nested composites, or perhaps more importantly composites with array fields. I think if we're going to add support for composites to pltcl, we should make sure we support these from the start rather than store up for ourselves the sorts of trouble that we're now grappling with in plperl-land. We shouldn't start to make pltcl users pass back composed array or record literals, if possible. As for the API changes, I'd like to have that piece reviewed by someone more familiar with the Tcl API than I am. I'm not sure who if anyone we have that has that familiarity, now Jan is no longer active. I know this has been on the table for six weeks, and an earlier review might have given Karl more chance to remedy these matters in time. I'm sorry about that, it's a pity the original reviewer ran into issues. But for now I'm inclined to mark this as "Returned with Feedbnack". cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 3:52 PM, Tom Lane wrote: >> The real issue is what happens when you want to install >> extension A, which depends on extensions B, C, and D, and B, C, and D >> are all in non-standard locations. Does that have any chance of >> working under the system we're proposing? > > Again, it's not really any different from the case where the dependent > objects are "loose" rather than members of an extension. It's pretty > much up to the user to be aware of the consequences. If we had a way to > mark individual functions as safe or unsafe for renames to happen, it'd > be reasonable to extend that notion to whole extensions. But we don't > have that and I don't think it's appropriate to hold extensions to a > higher standard than we do loose objects --- especially when it takes > superuser privileges to break things by moving an extension but not to > break them by moving loose objects. Well, the difference is that loose objects are just on my system, whereas extensions are supposed to work on anybody's system. I'm not clear that it's possible to write an extension that depends on a relocatable extension in a sensible way. If it is, objection withdrawn. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > On Thu, Feb 10, 2011 at 3:58 PM, Tom Lane wrote: >> Hmm. The problem with that is that once there are large numbers of >> intermediate versions, the number of potential paths grows >> exponentially. > It's certainly not exponential i.e. O(2^n) or something of that form. > Even a naive application of Dijkstra's algorithm is only going to be > O(n^2) in the number of versions, which is likely tolerable even if > upgrades are supported for dozens of old versions. Well, okay, let's go with that plan then. If we don't need to assume anything more than equality of version names being meaningful, I think chaining update scripts automatically should solve most of the complaints here. People who really want to maintain shortcut scripts still could, but I think it'd be an unusual case. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 4:14 PM, Dimitri Fontaine wrote: > Robert Haas writes: >> No, you have to get *those other module authors* to make *their* >> extensions not relocatable so that you can depend on them. > > Just tell me exactly in which world an extension's author is setting up > the dependencies in the 'required' property and yet fails to realise > that those dependencies mean his extension is not relocatable? And > will refuse to fix the problem when bugs are filled? No, the problem is this. I write an extension called foo. By default, it installs in schema foo. You write an extension called bar. By default, it installs in schema bar. It also depends on foo. Now Alice wants to install foo and bar. But she already has a schema called foo, so she installs the extension foo in foo2. Now she tries to install bar, but it doesn't work, because it is looking for objects in schema foo, and on this system they are in foo2. There's no way for you, as the author of bar, to fix this problem, other than to persuade me, as the author of foo, that I should make my extension not relocatable. I might not want to do that. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Tom Lane writes: > I don't think it's appropriate to hold extensions to a > higher standard than we do loose objects --- especially when it takes > superuser privileges to break things by moving an extension but not to > break them by moving loose objects. FWIW, +1. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 3:58 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Feb 10, 2011 at 3:02 PM, Tom Lane wrote: >>> The design as I sketched it didn't need to make any assumptions at all >>> about the meaning of the version identifiers. But if you were willing >>> to assume that the identifiers are comparable/sortable by some rule, > >> You don't need them to be sortable. You just need them to be >> comparable, and equality seems like a plenty good enough comparison >> rule. You can compute the shortest chain of upgrade scripts that can >> take you from the current version to the target version. > > Hmm. The problem with that is that once there are large numbers of > intermediate versions, the number of potential paths grows > exponentially. I was envisioning an algorithm like this: > > 1. Scan directory for upgrade scripts with oldversion = version we > have, and take the one with largest newversion <= version we want. > > 2. Apply this script (or more likely, just remember it until we've > verified there is a chain leading to version we want). > > 3. If now the version is not what we want, return to step 1. > > I don't see an equally efficient method if we only have equality. It's certainly not exponential i.e. O(2^n) or something of that form. Even a naive application of Dijkstra's algorithm is only going to be O(n^2) in the number of versions, which is likely tolerable even if upgrades are supported for dozens of old versions. It might break down if there are hundreds of old versions, but that doesn't seem likely to be a real problem in practice. But if you're concerned about it, you can replace the linked list that the naive algorithm uses with a binary heap or (if you really want to go nuts) a fibonacci heap. The latter approach has a runtime of O(n + m lg m), where n is the number of versions and m is the number of upgrade scripts. You need one heck of a lot of backward compatibility before that algorithm breaks a sweat. Even the binary heap is only O((n + m) lg m), which pretty darn fast. Personally, I think we'll be lucky if people support ten back revs, let alone three hundred, but it's a simple matter of programming - and an afternoon with an introductory algorithms textbook - to make it as efficient as we could ever want it to be. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > No, you have to get *those other module authors* to make *their* > extensions not relocatable so that you can depend on them. Just tell me exactly in which world an extension's author is setting up the dependencies in the 'required' property and yet fails to realise that those dependencies mean his extension is not relocatable? And will refuse to fix the problem when bugs are filled? I'm not following your reasonning… -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Hi, Tom Lane writes: > I spent some time reviewing this thread. I think the major point that's Thanks for doing that, we badly needed someone without an horse in this race to do that and finish the design. > So I believe that it'd be a good idea if it were possible for an extension > author to distribute a package that implements, say, versions 1.0, 1.1, > and 2.0 of hstore. Not all will choose to do the work needed for that, of > course, and that's fine. But the extension mechanism ought to permit it. Agreed. I've been weary of being told I'm trying to put too much into the first PostgreSQL release with extensions, but I'm happy to see you want to take it this far. So well, here follows some ideas I've been trying hard not to push too soon :) > To do this, we need to remove the concept that the control file specifies > "the" version of an extension; rather the version is associated with the > SQL script file. I think we should embed the version number in the script > file name, and require one to be present (no more omitted version > numbers). So you would distribute, say, > hstore-1.0.sql > hstore-1.1.sql > hstore-2.0.sql > representing the scripts needed to install these three versions from What I don't like in that is that this restrict what the version strings can look like. In debian for example it's pretty common to use the ~ separator, because 1.0~alpha1 < 1.0~beta < 1.0 with their sorting rules. And this trick won't work on windows filenames, AFAIK. That's one reason why I've wanted to stay away from having the version number strings encoded into the filename in the first place. But if you get to sorting rules of version strings, you have to define them properly and impose them to users. That's both a good thing and a bad thing, but IMO requires that we provide a proper data type for that. So my opinion here is that we should not only go with your design here with the version string in the filename, but also imposes how to spell out version strings in a way that we know will work for PostgreSQL on every supported system. > scratch. CREATE EXTENSION would have an option to select which > version to install. If the option is omitted, there are at least two > things we could do: > 1. Choose the newest available version. > 2. Let the control file specify which version is the default. > I think I prefer #2 because it avoids needing a rule for comparing > version identifiers, and it caters to the possibility that the "newest" > version isn't yet mature enough to be a good default. I like this idea. +1 for having the default version to install in the control file. See below for some more details about that, though. > As for upgrades, let's just expect upgrade scripts to be named > extension-oldversion-newversion.sql. ALTER EXTENSION UPGRADE knows the > relevant oldversion from pg_extension, and newversion can be handled the > same way as in CREATE, ie, either the user says which version to update to > or we use the default version from the control file. Again, I like the idea and how simple it make things look, but I think if we should then bite the bullet and restrict what a version string is expected to be and offer a data type with proper sorting while at it. And of course use that as the pg_extension.extversion column type. That way a SQL query can check if there's a new version available on your system. That's useful in some places to use as a monitoring alert coupled with nagios. The sysadmin team does the apt-get install part of the job and then the DBA team is paged to go upgrade the extensions in the databases, or shut the alarm somehow. > I don't seriously expect most extension authors to bother preparing > upgrade scripts for any cases except adjacent pairs of versions. > That means that if a user comes along and wants to upgrade across several > versions of the extension, he'll have to do it in several steps: > ALTER EXTENSION hstore UPGRADE TO '1.1'; > ALTER EXTENSION hstore UPGRADE TO '2.0'; > ALTER EXTENSION hstore UPGRADE TO '2.1'; > I don't see that as being a major problem --- how often will people have > the need to do that, anyway? Authors who feel that it is a big deal can > expend the work to provide shortcut scripts. I do not see adequate return > on investment from the regexp-matching complications in the currently > submitted patch. The regexp matching reason to live is so that we don't have to know anything about version strings at all. If you're saying that a version string can not contain a dash and must be a valid filesystem name (often enough, for all systems supported by PostgreSQL), and you're now saying that ALTER EXTENSION UPGRADE could automate multi-steps upgrade, then I think we have to provide the "version" (or "pgversion") data type and all that jazz. If we get to somehow, even lightly, depend on some rules, better offer them in code and documentation rather than have them implicit.
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > On Thu, Feb 10, 2011 at 3:02 PM, Tom Lane wrote: >> The design as I sketched it didn't need to make any assumptions at all >> about the meaning of the version identifiers. But if you were willing >> to assume that the identifiers are comparable/sortable by some rule, > You don't need them to be sortable. You just need them to be > comparable, and equality seems like a plenty good enough comparison > rule. You can compute the shortest chain of upgrade scripts that can > take you from the current version to the target version. Hmm. The problem with that is that once there are large numbers of intermediate versions, the number of potential paths grows exponentially. I was envisioning an algorithm like this: 1. Scan directory for upgrade scripts with oldversion = version we have, and take the one with largest newversion <= version we want. 2. Apply this script (or more likely, just remember it until we've verified there is a chain leading to version we want). 3. If now the version is not what we want, return to step 1. I don't see an equally efficient method if we only have equality. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 3:46 PM, David E. Wheeler wrote: > On Feb 10, 2011, at 12:07 PM, Robert Haas wrote: > >> You don't need them to be sortable. You just need them to be >> comparable, and equality seems like a plenty good enough comparison >> rule. You can compute the shortest chain of upgrade scripts that can >> take you from the current version to the target version. > > You have to be able to apply them in order. Unless I'm missing something, > that means you need to be able to sort them. Not at all. Say the currently installed version of the "dungeon" extension is "kobold" and you want to upgrade to "bugbear". You have the following scripts: dungeon-goblin-orc.sql dungeon-hobgoblin-bugbear.sql dungeon-kobold-goblin.sql dungeon-orc-hobgoblin.sql Now, it's pretty clear that the only way to get to bugbear is to come from hobgoblin, and the only way to get to hobgoblin is to come from orc. orc can be reached only from goblin, which can be reached only from kobold. So it's 100% clear that you have to apply the scripts in the following order: dungeon-kobold-goblin.sql dungeon-goblin-orc.sql dungeon-orc-hobgoblin.sql dungeon-hobgoblin-bugbear.sql Note that this even works if the versions aren't totally ordered. For example, suppose you release version 0.1 of a module and later you release a 1.0, which unfortunately is incompatible: there's no upgrade path from 0.1 to 1.0. In time, 1.0 is superseded by 1.1. And then you make some improvements to the old 0.1 code base and release that as 0.2. Finally, you come up with an idea for unifying the two and release a 1.2 version, which supports upgrades from all the previous versions. You just ship: foo-0.1-0.2.sql foo-0.2-1.2.sql foo-1.0-1.1.sql foo-1.1-1.2.sql If the user asks to upgrade to version 1.2, we'll observe that you can get to 1.2 from 1.1 or from 0.2. Not knowing what the version numbers mean, we'll look a bit further and see that you can get from 1.0 to 1.1 or from 0.1 to 0.2. Thus you can get to 1.2 like this: 0.1 -> 0.2 -> 1.2 0.2 -> 1.2 1.0 -> 1.1 -> 1.2 1.1 -> 1.2 -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > On Thu, Feb 10, 2011 at 2:42 PM, Tom Lane wrote: >> Now, if you want to argue that moving an extension after the fact (ALTER >> EXTENSION SET SCHEMA) is so dangerous as to be useless, I wouldn't >> argue very hard. Do you want to propose ripping that out? But >> relocating at first install doesn't seem horrible. > I'm not very concerned about letting people set the schema after the > fact. If we think it's OK for them to whack the location around at > first install, I don't know why we shouldn't also let them whack it > around later. The argument was that whether it's safe to move it during initial install is strictly a property of the extension's own internals. Once it's been in the database for awhile, moving it safely depends not only on the extension's internals but also on whether you have created any *other* objects that depend on where the extension is; for example, functions that have its schema name embedded in a SET search_path property or even hardwired in their code. However, this risk isn't really any different from when you do ALTER foo SET SCHEMA on a "loose" object, so on reflection it's not clear to me that we should refuse this case when we allow the latter. We're merely allowing people to shoot themselves in the foot with a machine-gun instead of a revolver, by providing a command that encapsulates a whole lot of SET SCHEMA commands in one action. > The real issue is what happens when you want to install > extension A, which depends on extensions B, C, and D, and B, C, and D > are all in non-standard locations. Does that have any chance of > working under the system we're proposing? Again, it's not really any different from the case where the dependent objects are "loose" rather than members of an extension. It's pretty much up to the user to be aware of the consequences. If we had a way to mark individual functions as safe or unsafe for renames to happen, it'd be reasonable to extend that notion to whole extensions. But we don't have that and I don't think it's appropriate to hold extensions to a higher standard than we do loose objects --- especially when it takes superuser privileges to break things by moving an extension but not to break them by moving loose objects. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 12:07 PM, Robert Haas wrote: > You don't need them to be sortable. You just need them to be > comparable, and equality seems like a plenty good enough comparison > rule. You can compute the shortest chain of upgrade scripts that can > take you from the current version to the target version. You have to be able to apply them in order. Unless I'm missing something, that means you need to be able to sort them. > But I'd be happy to leave that for 9.2. Yeah, if necessary. The only downside to that is, if we do indeed need them to be sortable, then we'd have to mandate a versioning format. And if there were existing extensions before 9.2, that might mess with them. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 12:02 PM, Tom Lane wrote: > Oh, I see, you're just saying that it's not unlikely somebody could find > himself with dozens of minor releases all being supported. Yeah, he'd > then really need to provide shortcut upgrade scripts, and > building/maintaining those would be a pain. Yes, exactly. > The design as I sketched it didn't need to make any assumptions at all > about the meaning of the version identifiers. But if you were willing > to assume that the identifiers are comparable/sortable by some rule, > then it wouldn't be that hard for ALTER EXTENSION UPGRADE to figure out > how to chain a series of upgrade scripts together to get from A to B, > and then there would be no need for manual maintenance of shortcut > scripts. IIRC the main objection to doing it that way was that the > underlying .so has to be compatible (at least to the extent of allowing > CREATE OR REPLACE FUNCTION) with all the intermediate versions --- but > if you believe the use-case I'm arguing for, that would be wanted > anyway, because all the intermediate versions would be considered > potentially useful stopping points. And that was essentially my original proposal. > I'm not philosophically opposed to requiring the version numbers to be > sortable, I just didn't want to introduce the concept if we didn't have > to. But maybe automatic application of a series of upgrade scripts is > enough reason. I always thought it was. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 3:33 PM, Dimitri Fontaine wrote: > Robert Haas writes: >>> Now, if you want to argue that moving an extension after the fact (ALTER >>> EXTENSION SET SCHEMA) is so dangerous as to be useless, I wouldn't >>> argue very hard. Do you want to propose ripping that out? But >>> relocating at first install doesn't seem horrible. > > Either an extension is relocatable or you have to deal with what Josh > Berkus the search_path hell. Lots of databases are using a host of > schema for their own objects already, and will want to have extensions > either all in the same place or scattered around each in its own schema. > > I don't think we are in a position to impose a choice to our users here. Well, for that matter, the user could want to install the same SQL objects in more than one schema, in effect installing the same extension twice. >> I'm not very concerned about letting people set the schema after the >> fact. If we think it's OK for them to whack the location around at >> first install, I don't know why we shouldn't also let them whack it >> around later. The question I have is whether it's really reasonable >> to let extension-owned objects be moved around at all. It'll probably >> work fine as long as there are no other extensions depending on the >> one that's getting moved, but it doesn't pay to design for the trivial > > If your extension depends on some others and your scripts are not > prepared to deal with those being moved around, you just setup your > extension as not relocatable. That's it. No, you have to get *those other module authors* to make *their* extensions not relocatable so that you can depend on them. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: >> Now, if you want to argue that moving an extension after the fact (ALTER >> EXTENSION SET SCHEMA) is so dangerous as to be useless, I wouldn't >> argue very hard. Do you want to propose ripping that out? But >> relocating at first install doesn't seem horrible. Either an extension is relocatable or you have to deal with what Josh Berkus the search_path hell. Lots of databases are using a host of schema for their own objects already, and will want to have extensions either all in the same place or scattered around each in its own schema. I don't think we are in a position to impose a choice to our users here. > I'm not very concerned about letting people set the schema after the > fact. If we think it's OK for them to whack the location around at > first install, I don't know why we shouldn't also let them whack it > around later. The question I have is whether it's really reasonable > to let extension-owned objects be moved around at all. It'll probably > work fine as long as there are no other extensions depending on the > one that's getting moved, but it doesn't pay to design for the trivial If your extension depends on some others and your scripts are not prepared to deal with those being moved around, you just setup your extension as not relocatable. That's it. > case. The real issue is what happens when you want to install > extension A, which depends on extensions B, C, and D, and B, C, and D > are all in non-standard locations. Does that have any chance of > working under the system we're proposing? Yes. It all depends on what's in the extension and what exactly the dependency is. You have more problem when calling another extension's function relying on the search_path that you have when using another extension's data type. But it boils down to which way the dependency is setup. And if moving objects breaks the install, you move them back then fill a bug, and the extension's author changes relocatable to false in the next version, or fix the bug in another way. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python custom exceptions for SPI
On 10/02/11 20:24, Peter Eisentraut wrote: > On sön, 2011-02-06 at 20:44 +0100, Jan Urbański wrote: >> On 27/01/11 23:24, Jan Urbański wrote: >>> On 11/01/11 12:20, Jan Urbański wrote: On 11/01/11 01:27, Tom Lane wrote: > Hannu Krosing writes: >> On 10.1.2011 17:20, Jan Urbański wrote: >>> I changed that patch to use Perl instead of sed to generate the >>> exceptions, which should be a more portable. >>> >>> Updated as an incremental patch on to of the recently sent version of >>> explicit-subxacts. >> >> Updated again. > > Why do the error messages print spiexceptions.SyntaxError instead of > plpy.spiexceptions.SyntaxError? Is this intentional or just the way it > comes out of Python? That's how traceback.format_exception() works IIRC, which is what the Python interpreter uses and what PL/Python mimicks in PLy_traceback. > Please add some documentation. Not a list of all exceptions, but at > least a paragraph that various kinds of specific exceptions may be > generated, what package and module they are in, and how they relate. Sure, Steve already asked for docs in another thread, and I'm writing them. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 3:02 PM, Tom Lane wrote: > "David E. Wheeler" writes: >> On Feb 10, 2011, at 11:31 AM, Tom Lane wrote: >>> I don't see how that affects my point? You can spell "1.0" as "0.1" >>> and "1.1" as "0.2" if you like that kind of numbering, but I don't >>> see that that has any real impact. At the end of the day an author is >>> going to crank out a series of releases, and if he cares about people >>> using those releases for production, he's going to have to provide at >>> least a upgrade script to move an existing database from release N to >>> release N+1. > >> Yeah, but given a rapidly-developing extension, that could create a lot of >> extra work. I don't know that there's much of a way around that, other than >> concatenating files to build migration scripts from parts (perhaps via >> `Make` as dim suggested). But it can get complicated pretty fast. My desire >> here is to keep the barrier to creating PostgreSQL extensions as low as is >> reasonably possible. > > Oh, I see, you're just saying that it's not unlikely somebody could find > himself with dozens of minor releases all being supported. Yeah, he'd > then really need to provide shortcut upgrade scripts, and > building/maintaining those would be a pain. > > The design as I sketched it didn't need to make any assumptions at all > about the meaning of the version identifiers. But if you were willing > to assume that the identifiers are comparable/sortable by some rule, > then it wouldn't be that hard for ALTER EXTENSION UPGRADE to figure out > how to chain a series of upgrade scripts together to get from A to B, > and then there would be no need for manual maintenance of shortcut > scripts. IIRC the main objection to doing it that way was that the > underlying .so has to be compatible (at least to the extent of allowing > CREATE OR REPLACE FUNCTION) with all the intermediate versions --- but > if you believe the use-case I'm arguing for, that would be wanted > anyway, because all the intermediate versions would be considered > potentially useful stopping points. > > I'm not philosophically opposed to requiring the version numbers to be > sortable, I just didn't want to introduce the concept if we didn't have > to. But maybe automatic application of a series of upgrade scripts is > enough reason. You don't need them to be sortable. You just need them to be comparable, and equality seems like a plenty good enough comparison rule. You can compute the shortest chain of upgrade scripts that can take you from the current version to the target version. But I'd be happy to leave that for 9.2. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 2:42 PM, Tom Lane wrote: > General opinion around Red Hat is relocatable RPMs don't work. But > pushing a set of functions from one schema to another is a very much > narrower problem than what an RPM has to deal with, so I'm not convinced > that the analogy holds. > > Now, if you want to argue that moving an extension after the fact (ALTER > EXTENSION SET SCHEMA) is so dangerous as to be useless, I wouldn't > argue very hard. Do you want to propose ripping that out? But > relocating at first install doesn't seem horrible. I'm not very concerned about letting people set the schema after the fact. If we think it's OK for them to whack the location around at first install, I don't know why we shouldn't also let them whack it around later. The question I have is whether it's really reasonable to let extension-owned objects be moved around at all. It'll probably work fine as long as there are no other extensions depending on the one that's getting moved, but it doesn't pay to design for the trivial case. The real issue is what happens when you want to install extension A, which depends on extensions B, C, and D, and B, C, and D are all in non-standard locations. Does that have any chance of working under the system we're proposing? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
"David E. Wheeler" writes: > On Feb 10, 2011, at 11:31 AM, Tom Lane wrote: >> I don't see how that affects my point? You can spell "1.0" as "0.1" >> and "1.1" as "0.2" if you like that kind of numbering, but I don't >> see that that has any real impact. At the end of the day an author is >> going to crank out a series of releases, and if he cares about people >> using those releases for production, he's going to have to provide at >> least a upgrade script to move an existing database from release N to >> release N+1. > Yeah, but given a rapidly-developing extension, that could create a lot of > extra work. I don't know that there's much of a way around that, other than > concatenating files to build migration scripts from parts (perhaps via `Make` > as dim suggested). But it can get complicated pretty fast. My desire here is > to keep the barrier to creating PostgreSQL extensions as low as is reasonably > possible. Oh, I see, you're just saying that it's not unlikely somebody could find himself with dozens of minor releases all being supported. Yeah, he'd then really need to provide shortcut upgrade scripts, and building/maintaining those would be a pain. The design as I sketched it didn't need to make any assumptions at all about the meaning of the version identifiers. But if you were willing to assume that the identifiers are comparable/sortable by some rule, then it wouldn't be that hard for ALTER EXTENSION UPGRADE to figure out how to chain a series of upgrade scripts together to get from A to B, and then there would be no need for manual maintenance of shortcut scripts. IIRC the main objection to doing it that way was that the underlying .so has to be compatible (at least to the extent of allowing CREATE OR REPLACE FUNCTION) with all the intermediate versions --- but if you believe the use-case I'm arguing for, that would be wanted anyway, because all the intermediate versions would be considered potentially useful stopping points. I'm not philosophically opposed to requiring the version numbers to be sortable, I just didn't want to introduce the concept if we didn't have to. But maybe automatic application of a series of upgrade scripts is enough reason. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
On Feb 10, 2011, at 11:21 AM, Tom Lane wrote: >> It will certainly mess up syntax highlighting and matching bracket detection >> in pretty much all text editors... > > Yeah. It's a cute-looking notation but surely it will cause many more > problems than it's worth. I agree with Robert's suggestion of plain > functions named like range_co() etc. I could see myself using ranges in ways similar to using arrays. It should would be nice to have the equivalent of ARRAY[] and/or ARRAY() to use for ranges… Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
Robert Haas writes: > On Thu, Feb 10, 2011 at 1:26 PM, Tom Lane wrote: >> requires, relocatable and schema: These are problematic, because it's not >> out of the question that someone might want to change these properties >> from one version to another. But as things are currently set up, we must >> know these things before we start to run the extension script, because >> they are needed to set up the search_path correctly. > My biggest concern with this extensions work is that these variables > are poorly designed. Yeah, I didn't especially like relocatable/schema either. I thought for awhile about redefining relocatable as a three-way switch, corresponding to the three use cases (relocatable after the fact, relocatable only at initial install, no relocation) but didn't pull the trigger. It is advantageous to have an explicit notion of a particular schema containing the extension's exported stuff, so that we can add that schema into the search path for dependent extensions. That means that you can't easily remove the explicit schema value for the third case, so it's not that easy to make it look cleaner. > The extension mechanism is basically the > equivalent of RPM for inside the database. And while in theory there > is such a thing as a relocatable RPM, I don't know that I've ever used > it, at least not successfully. General opinion around Red Hat is relocatable RPMs don't work. But pushing a set of functions from one schema to another is a very much narrower problem than what an RPM has to deal with, so I'm not convinced that the analogy holds. Now, if you want to argue that moving an extension after the fact (ALTER EXTENSION SET SCHEMA) is so dangerous as to be useless, I wouldn't argue very hard. Do you want to propose ripping that out? But relocating at first install doesn't seem horrible. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] arrays as pl/perl input arguments [PATCH]
On 02/10/2011 08:15 AM, Alexey Klyukin wrote: On Feb 9, 2011, at 9:28 PM, Alex Hunsaker wrote: On Wed, Feb 9, 2011 at 08:24, Alexey Klyukin wrote: What was actually broken in encode_array_literal support of composite types (it converted perl hashes to the literal composite-type constants, expanding nested arrays along the way) ? I think it would be a useful extension of the existing encode_array_literal. Yeah, It does not work because it did not take into account the order of composite columns. It always put them alphabetically by column name. To do it properly we would need to pass in a typid or a column order or something. Ideally we could expose the new plperl_array_to_datum() to plperl functions in some manner. Damn, right. Each perl hash corresponds to multiple composite types, different by the order of the type elements. Passing the typid sounds like a fair requirement (and if it's missing we could assume that the order of columns in composites doesn't matter to the caller). Let me try implementing that as an XS interface to plperl_array_to_datum. Are you intending this as a completion of the current patch or as 9.2 work? If the former you need to send it in real fast. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
On Feb 10, 2011, at 10:41 AM, Jeff Davis wrote: > This might solve the constructor problem nicely if we could do things > like: > RANGE[10,20) > But I have a feeling that will either cause a bizarre problem with the > grammar, or someone will think it's not very SQL-like. I like it a lot better than the funkily-named functions you posted yesterday. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 11:31 AM, Tom Lane wrote: > I'm not really addressing that in this proposal. You could imagine > supporting all the extension versions in one .so, or you could have one > per version (meaning the upgrade scripts would have to CREATE OR REPLACE > all the C functions to re-point them at a different .so), or mixed > cases. Right now the PGXS infrastructure would favor the first because > it has only limited ability to build multiple .so's in one directory; > but we could think about improving that if there's demand. > > Note that you can version a function even within a single .so, for > example if hstore 1.0 defines foo() one way and hstore 1.1 defines > it another, you could make the latter point to the C function name > foo_1_1 while C function foo continues to provide the old behavior. > You have to at least provide a stub foo (that could just throw error > if called) for as long as you want to support upgrading from 1.0. Good enough for me. > I don't see how that affects my point? You can spell "1.0" as "0.1" > and "1.1" as "0.2" if you like that kind of numbering, but I don't > see that that has any real impact. At the end of the day an author is > going to crank out a series of releases, and if he cares about people > using those releases for production, he's going to have to provide at > least a upgrade script to move an existing database from release N to > release N+1. Yeah, but given a rapidly-developing extension, that could create a lot of extra work. I don't know that there's much of a way around that, other than concatenating files to build migration scripts from parts (perhaps via `Make` as dim suggested). But it can get complicated pretty fast. My desire here is to keep the barrier to creating PostgreSQL extensions as low as is reasonably possible. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding new variables into GUC
Josh Berkus writes: >> You basically need the variable, the entry in the appropriate array in >> guc.c, and some documentation (at least if you'd like anyone else to >> ever use the code). Try looking at some past patches that added GUCs >> similar to yours. > For completeness, it would also be good to add rows to the pg_settings > system catalog, but that's not necessary for testing. Huh? pg_settings is a dynamically generated view. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
"David E. Wheeler" writes: > +1. I assume there will be some way to build versioned shared object > libraries too, then? I'm not really addressing that in this proposal. You could imagine supporting all the extension versions in one .so, or you could have one per version (meaning the upgrade scripts would have to CREATE OR REPLACE all the C functions to re-point them at a different .so), or mixed cases. Right now the PGXS infrastructure would favor the first because it has only limited ability to build multiple .so's in one directory; but we could think about improving that if there's demand. Note that you can version a function even within a single .so, for example if hstore 1.0 defines foo() one way and hstore 1.1 defines it another, you could make the latter point to the C function name foo_1_1 while C function foo continues to provide the old behavior. You have to at least provide a stub foo (that could just throw error if called) for as long as you want to support upgrading from 1.0. >> In this scheme, all the extension scripts are independent. We spent quite >> a lot of time arguing about ways to avoid duplication of code between >> scripts, but frankly I'm not convinced that that's worth troubling over. >> As far as the initial-install scripts go, once you've released 1.0 it's >> unlikely you'll ever change it again, so the fact that you copied and >> pasted it as a starting point for 1.1 isn't really a maintenance burden. > I disagree with this. A lot of dynamic language libraries never get to > 1.0, and even if they do can go through periods of extensive development > with major changes from version to version. I don't see how that affects my point? You can spell "1.0" as "0.1" and "1.1" as "0.2" if you like that kind of numbering, but I don't see that that has any real impact. At the end of the day an author is going to crank out a series of releases, and if he cares about people using those releases for production, he's going to have to provide at least a upgrade script to move an existing database from release N to release N+1. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python custom exceptions for SPI
On sön, 2011-02-06 at 20:44 +0100, Jan Urbański wrote: > On 27/01/11 23:24, Jan Urbański wrote: > > On 11/01/11 12:20, Jan Urbański wrote: > >> On 11/01/11 01:27, Tom Lane wrote: > >>> Hannu Krosing writes: > On 10.1.2011 17:20, Jan Urbański wrote: > > I changed that patch to use Perl instead of sed to generate the > > exceptions, which should be a more portable. > > > > Updated as an incremental patch on to of the recently sent version of > > explicit-subxacts. > > Updated again. Why do the error messages print spiexceptions.SyntaxError instead of plpy.spiexceptions.SyntaxError? Is this intentional or just the way it comes out of Python? Please add some documentation. Not a list of all exceptions, but at least a paragraph that various kinds of specific exceptions may be generated, what package and module they are in, and how they relate. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
Florian Pflug writes: >> This might solve the constructor problem nicely if we could do things >> like: >> RANGE[10,20) >> But I have a feeling that will either cause a bizarre problem with the >> grammar, or someone will think it's not very SQL-like. > It will certainly mess up syntax highlighting and matching bracket detection > in pretty much all text editors... Yeah. It's a cute-looking notation but surely it will cause many more problems than it's worth. I agree with Robert's suggestion of plain functions named like range_co() etc. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove more SGML tabs.
On tor, 2011-02-10 at 10:40 -0800, David Fetter wrote: > I think all such checks belong in .git/hooks/pre-commit, and need to > be as cross-platform as needed for committers. Would a *n*x-based > version do for a start? I think as a matter of principle, the only things that belongs into git hooks are things that relate to the repository itself, such as commit metadata (author checks, commit message spell checks, etc.). Anything that relates to the source belongs into the source. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
> This might solve the constructor problem nicely if we could do things > like: > RANGE[10,20) > But I have a feeling that will either cause a bizarre problem with the > grammar, or someone will think it's not very SQL-like. It will certainly mess up syntax highlighting and matching bracket detection in pretty much all text editors... best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On 08.02.2011 20:53, Robert Haas wrote: That having been said, there is at least one part of this patch which looks to be in pretty good shape and seems independently useful regardless of what happens to the rest of it, and that is the code that sends replies from the standby back to the primary. This allows pg_stat_replication to display the write/flush/apply log positions on the standby next to the sent position on the primary, which as far as I am concerned is pure gold. Simon had this set up to happen only when synchronous replication or XID feedback in use, but I think people are going to want it even with plain old asynchronous replication, because it provides a FAR easier way to monitor standby lag than anything we have today. I've extracted this portion of the patch, cleaned it up a bit, written docs, and attached it here. Thanks! I wasn't too sure how to control the timing of the replies. It's worth noting that you have to send them pretty frequently for the distinction between xlog written and xlog flushed to have any value. What I've done here is made it so that every time we read all available data on the socket, we send a reply. After flushing, we send another reply. And then just for the heck of it we send a reply at least every 10 seconds (configurable), which causes the last-known-apply position to eventually get updated on the master. This means the apply position can lag reality by a bit. Seems reasonable. As the patch stands, however, the standby doesn't send any status updates if its busy receiving, writing, and flushing the incoming WAL. That would happen if you have a fast network, and slow disk, and the standby is catching up, e.g after restoring a base backup. I added a XLogWalRcvSendReply() call into XLogWalRcvFlush() so that it also sends a status update every time the WAL is flushed. If the walreceiver is busy receiving and flushing, that would happen once per WAL segment, which seems sensible. The comment above StandbyReplyMessage said that its message type is 'r'. However, no message type was actually sent for the replies. A message type byte seems like a good idea, for the sake of extensibility, so I made the code match that comment. I also added documentation of this new message type in the manual section about the streaming replication protocol. I committed the patch with those changes, and some minor comment tweaks and other kibitzing. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding new variables into GUC
2011/2/10 Josh Berkus : > >> You basically need the variable, the entry in the appropriate array in >> guc.c, and some documentation (at least if you'd like anyone else to >> ever use the code). Try looking at some past patches that added GUCs >> similar to yours. > > For completeness, it would also be good to add rows to the pg_settings > system catalog, but that's not necessary for testing. pg_settings doesn't need to be separately updated. It's just a view. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Thu, Feb 10, 2011 at 1:26 PM, Tom Lane wrote: > requires, relocatable and schema: These are problematic, because it's not > out of the question that someone might want to change these properties > from one version to another. But as things are currently set up, we must > know these things before we start to run the extension script, because > they are needed to set up the search_path correctly. My biggest concern with this extensions work is that these variables are poorly designed. The extension mechanism is basically the equivalent of RPM for inside the database. And while in theory there is such a thing as a relocatable RPM, I don't know that I've ever used it, at least not successfully. I'm worried this is going to be a pretty serious rough edge that's difficult to file down later. Forcing everything into a single schema (like pg_extension) seems a bit too draconian, but this idea that you can install things wherever you like and somehow it's gonna just work seems pretty optimistic. However, that's a side point. The overall design you propose seems reasonable to me. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding new variables into GUC
> You basically need the variable, the entry in the appropriate array in > guc.c, and some documentation (at least if you'd like anyone else to > ever use the code). Try looking at some past patches that added GUCs > similar to yours. For completeness, it would also be good to add rows to the pg_settings system catalog, but that's not necessary for testing. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
On Feb 10, 2011, at 10:26 AM, Tom Lane wrote: > 1. Choose the newest available version. > 2. Let the control file specify which version is the default. > I think I prefer #2 because it avoids needing a rule for comparing > version identifiers, and it caters to the possibility that the "newest" > version isn't yet mature enough to be a good default. +1. I assume there will be some way to build versioned shared object libraries too, then? > In this scheme, all the extension scripts are independent. We spent quite > a lot of time arguing about ways to avoid duplication of code between > scripts, but frankly I'm not convinced that that's worth troubling over. > As far as the initial-install scripts go, once you've released 1.0 it's > unlikely you'll ever change it again, so the fact that you copied and > pasted it as a starting point for 1.1 isn't really a maintenance burden. I disagree with this. A lot of dynamic language libraries never get to 1.0, and even if they do can go through periods of extensive development with major changes from version to version. Just have a look at the pgTAP changes file for an example: https://github.com/theory/pgtap/blob/master/Changes I already do a *lot* of work in the Makefile to patch things so that it works all the way back to 8.0. And I'm adding stuff now to generate other files that will contain a subset of the pgTAP functionality. I don't think I'd ever write upgrade scripts for pgTAP, but I've worked with a lot of Perl modules that have followed similar aggressive development, and can imagine times when I'd need to write upgrade scripts for aggressively-developed PostgreSQL extensions. And I quail at the idea. Lord help me if I'd need to also write create patches for my upgrade scripts to support older versions of PostgreSQL. > Version upgrade scripts won't share any code at all, unless the author is > trying to provide shortcut scripts for multi-version jumps, and as I said, > I doubt that many will bother. Also, it'll be some time before there's > much need for multi-version update scripts anyway, so I am not feeling > that it is necessary to solve that now. We could later on add some kind > of script inclusion capability to allow authors to avoid code duplication > in multi-version update scripts, but it's just not urgent. Okay, that would be a big help. And I'm fine with it being something to "maybe be added later." We'll see then what cow paths develop, and demands for pasture fences to be cut down. Or something. > So, concrete proposal is to enforce the "extension-version.sql" and > "extension-oldversion-newversion.sql" naming rules for scripts, which > means getting rid of the script name parameter in control files. > (Instead, we could have a directory parameter that tells which directory > holds all the install and upgrade scripts for the extension.) +1 I like this idea. I'm already putting all my scripts into an sql/ directory for PGXN distributions: https://github.com/theory/pg-semver > encoding: I don't see any big problem with insisting that all scripts for > a given extension be in the same encoding. +1. Also, can't one set client_encoding in the scripts anyway? > requires, relocatable and schema: These are problematic, because it's not > out of the question that someone might want to change these properties > from one version to another. But as things are currently set up, we must > know these things before we start to run the extension script, because > they are needed to set up the search_path correctly. > > Perhaps for now it's sufficient to say that these properties can't change > across versions. Alternatively, we could allow there to be a secondary > version-specific control file that can override the main control file. > IOW, we'd read "extension.control" to get the directory and > default_version values, then determine the version we are installing or > upgrading to, then see if there's an "extension-version.control" file > in the extension's directory, and if so read that and let it replace > the remaining parameters' values. +1. I'll need to play around with some of this stuff to see how it affects PGXN distributions. My main concern will be allowing an "extension distribution" to somehow work both on 9.1 with EXTENSIONs and in < 9.0 as PGXS-installed modules currently work, without too much pain to the developer to support previous versions of PostgreSQL. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
On Thu, Feb 10, 2011 at 1:41 PM, Jeff Davis wrote: > This might solve the constructor problem nicely if we could do things > like: > RANGE[10,20) > But I have a feeling that will either cause a bizarre problem with the > grammar, or someone will think it's not very SQL-like. I think won't cause any problem at all if RANGE is fully reserved, but like you say we probably don't want to do that unless it's absolutely necessary, and if you don't actually need to be able to type in foo RANGE JOIN bar then it probably isn't. I think your proposed naming schema for constructors is pretty reasonable, except I might use "o" for open and "c" for closed rather than "i" and "_", i.e. range_oo(), range_oc(), range_co(), range_cc(). If that'll get us by without fully reserving RANGE then I'd certainly be in favor of doing it that way. I was just saying - if we were inevitably going to have to reserve RANGE, then we could try to squeeze a little more out of it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
On Thu, 2011-02-10 at 13:07 -0500, Robert Haas wrote: > According to our documentation[1], RANGE is reserved in SQL:2008 and > SQL:2003, which makes it more imaginable to reserve it than it would > be otherwise. Oh, interesting. > I believe that in a previous email you mentioned that > you were hoping to implement RANGE JOIN, and I will just note that the > restrictions of the grammar require that any keyword that immediately > follows the previous expression and precedes JOIN must be fully > reserved. I'm not sure if you meant that a range join would literally > use the syntax RANGE JOIN, but if so then you're going to have to > argue for fully reserving RANGE anyway, in which case there'd be no > special reason not to allow RANGE [1,10) to mean just that. On the > other hand, if a RANGE JOIN just means a regular join on some funky > operator, and there's no other reason to reserve range, I wouldn't do > it just to get a nicer syntax here. It's mostly just a regular join on a funky operator. We may want that operator to allow a new plan (range merge join); but I think we can determine that it's a range join from the use of the operator. I'll have to look into that more. > Have you done investigation of what RANGE is used to mean in the SQL > spec? Is what you're implementing (a) spec, (b) similar idea, but not > the spec, or (c) something completely different? I'm guessing (c) but > I have no idea what the spec is using it for. (c) was my intention. I did take a brief look at the spec a while back, but I'll take a more detailed look. I think it only has to do with window specifications. This might solve the constructor problem nicely if we could do things like: RANGE[10,20) But I have a feeling that will either cause a bizarre problem with the grammar, or someone will think it's not very SQL-like. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Remove more SGML tabs.
On Thu, Feb 10, 2011 at 12:58:16PM +0200, Peter Eisentraut wrote: > On ons, 2011-02-09 at 08:00 -0800, David Fetter wrote: > > On Wed, Feb 09, 2011 at 01:17:06PM +, Bruce Momjian wrote: > > > Remove more SGML tabs. > > > > Perhaps we should see about putting something in .git/hooks/pre-commit > > so people can focus on more substantive matters. > > > > Is there some kind of cross-platform way to do this? I'm thinking > > that given the fact that our build system already requires Perl, there > > should be, but I'm not quite sure how this would be accomplished. > > There is make check target in doc/src/sgml/ that is supposed to catch > this. But it's probably hard to remember to run that. It's "check-tabs", and if you hadn't mentioned it, I'd never have seen it. > One thing I was thinking of was that we could add a global make > maintainer-check target (a name I picked up from other projects) which > would run various source code sanity checks. Besides the SGML tabs > issue, my favourite would be duplicate_oids. Maybe if we could find a > third use case, we'd have a quorum for implementing this. I think all such checks belong in .git/hooks/pre-commit, and need to be as cross-platform as needed for committers. Would a *n*x-based version do for a start? Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types (catversion.h)
On Thu, Feb 10, 2011 at 1:23 PM, Heikki Linnakangas wrote: > On 10.02.2011 20:01, Peter Eisentraut wrote: >> >> On tor, 2011-02-10 at 09:28 -0800, Jeff Davis wrote: >>> >>> I originally put it there so that I wouldn't mix up data directories >>> with a patch I'm reviewing, but I agree that it seems easier this way. >> >> FWIW, I disagree with Tom and do recommend putting the catversion change >> in the patch. > > I'm very bad at remembering to bump it, so I also won't mind patch authors > doing it. > > The ideal reminder would be some special comment you could put on the > catversion line that would cause "git push" to fail if it's still there when > I try to push the commit to the repository. There doesn't seem to be a > "pre-push" hook in git, although some googling suggests that it would be > quite easy to write a small wrapper shell script to check that. I'm > seriously considering to do that, given that I more often forget to bump > catversion than not. And I share Tom's preference, which is to not include it, because I usually apply patches using patch, and when diff hunks fail it's a nuisance for me. So basically, do whatever you want, someone won't like it no matter what. :-) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER EXTENSION UPGRADE, v3
I spent some time reviewing this thread. I think the major point that's not received adequate discussion is this: the design assumes that there's just one "current version" of any extension, and that's not good enough. David Fetter was trying to make that point upthread but didn't seem to convince people. I'm convinced though. I think that one of the major selling points of extensions could be having a controlled way of exposing different versions of an API and letting users select which one is in use in each database. Look at how much effort we spend maintaining back branches of the core code for people who don't want to, eg, update their apps to avoid pre-8.3-style implicit casting. (Yeah, I know that on-disk compatibility is another major reason for staying on a back branch, but API changes are definitely part of it.) So I believe that it'd be a good idea if it were possible for an extension author to distribute a package that implements, say, versions 1.0, 1.1, and 2.0 of hstore. Not all will choose to do the work needed for that, of course, and that's fine. But the extension mechanism ought to permit it. Over time we might get to a point where somebody could be running the latest version of the core database (with all the bug fixes and other goodness of that) but his application compatibility problems are solved by running back-rev versions of certain extensions. To do this, we need to remove the concept that the control file specifies "the" version of an extension; rather the version is associated with the SQL script file. I think we should embed the version number in the script file name, and require one to be present (no more omitted version numbers). So you would distribute, say, hstore-1.0.sql hstore-1.1.sql hstore-2.0.sql representing the scripts needed to install these three versions from scratch. CREATE EXTENSION would have an option to select which version to install. If the option is omitted, there are at least two things we could do: 1. Choose the newest available version. 2. Let the control file specify which version is the default. I think I prefer #2 because it avoids needing a rule for comparing version identifiers, and it caters to the possibility that the "newest" version isn't yet mature enough to be a good default. As for upgrades, let's just expect upgrade scripts to be named extension-oldversion-newversion.sql. ALTER EXTENSION UPGRADE knows the relevant oldversion from pg_extension, and newversion can be handled the same way as in CREATE, ie, either the user says which version to update to or we use the default version from the control file. I don't seriously expect most extension authors to bother preparing upgrade scripts for any cases except adjacent pairs of versions. That means that if a user comes along and wants to upgrade across several versions of the extension, he'll have to do it in several steps: ALTER EXTENSION hstore UPGRADE TO '1.1'; ALTER EXTENSION hstore UPGRADE TO '2.0'; ALTER EXTENSION hstore UPGRADE TO '2.1'; I don't see that as being a major problem --- how often will people have the need to do that, anyway? Authors who feel that it is a big deal can expend the work to provide shortcut scripts. I do not see adequate return on investment from the regexp-matching complications in the currently submitted patch. In this scheme, all the extension scripts are independent. We spent quite a lot of time arguing about ways to avoid duplication of code between scripts, but frankly I'm not convinced that that's worth troubling over. As far as the initial-install scripts go, once you've released 1.0 it's unlikely you'll ever change it again, so the fact that you copied and pasted it as a starting point for 1.1 isn't really a maintenance burden. Version upgrade scripts won't share any code at all, unless the author is trying to provide shortcut scripts for multi-version jumps, and as I said, I doubt that many will bother. Also, it'll be some time before there's much need for multi-version update scripts anyway, so I am not feeling that it is necessary to solve that now. We could later on add some kind of script inclusion capability to allow authors to avoid code duplication in multi-version update scripts, but it's just not urgent. So, concrete proposal is to enforce the "extension-version.sql" and "extension-oldversion-newversion.sql" naming rules for scripts, which means getting rid of the script name parameter in control files. (Instead, we could have a directory parameter that tells which directory holds all the install and upgrade scripts for the extension.) Also, the "version" parameter should be renamed to something like "current_version" or "default_version". We also have to be wary of whether any other control-file parameters specify something that might be version-specific. Looking at the current list: comment: probably OK to consider this as a default for all versions. We al
Re: [HACKERS] postponing some large patches to 9.2
On Thu, 2011-02-10 at 09:46 -0500, Robert Haas wrote: > On Tue, Feb 8, 2011 at 7:58 PM, Jeff Davis wrote: > > On the flip side, if we don't provide review to WIP patches during the > > 3rd commitfest, how do we expect to get anything close to committable on > > the 1st commitfest of the next cycle? > > I'm not sure exactly what you're going for here, because I don't think > I've ever proposed any special treatment of patches in the third > CommitFest, I actually meant 4th (this one). I forgot that the July one was actually a part of the 9.1 cycle. > But if > you were to say that WIP patches *in general* get a lot less review > than non-WIP patches, I would agree with you. > > To some extent, I think that's inevitable. It's not fun to review WIP > patches. Agreed, but it doesn't really apply to this situation. There was still a week left, and the reviewer was still reviewing. So I found it jarring when you said that it had received enough review, and bounced it. In my opinion, if we're going to entertain WIP patches during a commitfest, we shouldn't bounce them early for being WIP. We can bounce them for other causes, like "waiting on author" or "we couldn't find a reviewer" or "we're out of time". > I've found that it's > nearly always better to post specific questions that you want to know > the answer to, rather than a patch where people have to guess what > parts you want feedback on. Well, I've certainly posted some specific questions. I don't expect to get an answer to all of them right away, and certainly many have been answered -- but I didn't just throw the code out and wait. For instance: http://archives.postgresql.org/message-id/1297230650.27157.398.camel@jdavis Anyway, I don't think any of this affected the patch, I was just surprised. I'll leave it at that, because I'm sure you're busy wrapping up this commitfest. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types (catversion.h)
On 10.02.2011 20:01, Peter Eisentraut wrote: On tor, 2011-02-10 at 09:28 -0800, Jeff Davis wrote: I originally put it there so that I wouldn't mix up data directories with a patch I'm reviewing, but I agree that it seems easier this way. FWIW, I disagree with Tom and do recommend putting the catversion change in the patch. I'm very bad at remembering to bump it, so I also won't mind patch authors doing it. The ideal reminder would be some special comment you could put on the catversion line that would cause "git push" to fail if it's still there when I try to push the commit to the repository. There doesn't seem to be a "pre-push" hook in git, although some googling suggests that it would be quite easy to write a small wrapper shell script to check that. I'm seriously considering to do that, given that I more often forget to bump catversion than not. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types (catversion.h)
On Thu, 2011-02-10 at 15:38 +0100, Erik Rijkers wrote: > I've removed the change to catversion.h (18 lines, starting at 4985) from the > patch file; then it > applies cleanly. I should mention that the last patch changed the representation to be more compact. So, if you have any existing test data it will need to be reloaded to work with the latest. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Type constructors
On Wed, Feb 9, 2011 at 2:09 AM, Jeff Davis wrote: > That's how arrays do it: there's a special Expr node that represents an > array expression. Maybe the same thing could be used for range types, > but I fear that there may be some grammar conflicts. I doubt we'd want > to fully reserve the keyword "range". According to our documentation[1], RANGE is reserved in SQL:2008 and SQL:2003, which makes it more imaginable to reserve it than it would be otherwise. I believe that in a previous email you mentioned that you were hoping to implement RANGE JOIN, and I will just note that the restrictions of the grammar require that any keyword that immediately follows the previous expression and precedes JOIN must be fully reserved. I'm not sure if you meant that a range join would literally use the syntax RANGE JOIN, but if so then you're going to have to argue for fully reserving RANGE anyway, in which case there'd be no special reason not to allow RANGE [1,10) to mean just that. On the other hand, if a RANGE JOIN just means a regular join on some funky operator, and there's no other reason to reserve range, I wouldn't do it just to get a nicer syntax here. Have you done investigation of what RANGE is used to mean in the SQL spec? Is what you're implementing (a) spec, (b) similar idea, but not the spec, or (c) something completely different? I'm guessing (c) but I have no idea what the spec is using it for. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company [1] http://www.postgresql.org/docs/current/static/sql-keywords-appendix.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types (catversion.h)
On tor, 2011-02-10 at 09:28 -0800, Jeff Davis wrote: > I originally put it there so that I wouldn't mix up data directories > with a patch I'm reviewing, but I agree that it seems easier this way. FWIW, I disagree with Tom and do recommend putting the catversion change in the patch. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types (catversion.h)
On Thu, 2011-02-10 at 12:04 -0500, Tom Lane wrote: > "Erik Rijkers" writes: > > On Wed, February 9, 2011 09:35, Jeff Davis wrote: > >> Updated patch. > > > I just wanted to mention that this latest patch doesn't quite apply as-is, > > because of catversion changes. > > Just a note: standard practice is for submitted patches to *not* touch > catversion.h. The committer will add that change before committing. > Otherwise, it's just guaranteed to cause merge problems such as this > one. (It's not unreasonable to mention the need for a catversion bump > in the description of the patch, if you think the committer might not > realize it.) OK, I'll remove that then. I originally put it there so that I wouldn't mix up data directories with a patch I'm reviewing, but I agree that it seems easier this way. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types (catversion.h)
"Erik Rijkers" writes: > On Wed, February 9, 2011 09:35, Jeff Davis wrote: >> Updated patch. > I just wanted to mention that this latest patch doesn't quite apply as-is, > because of catversion changes. Just a note: standard practice is for submitted patches to *not* touch catversion.h. The committer will add that change before committing. Otherwise, it's just guaranteed to cause merge problems such as this one. (It's not unreasonable to mention the need for a catversion bump in the description of the patch, if you think the committer might not realize it.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extensions versus pg_upgrade
Tom Lane writes: > That would be rejected because you're not allowed to drop an individual > member object of an extension. (And no, I don't want to have a kluge in > dependency.c that makes that test work differently when > creating_extension.) Fair enough, all the more as soon as we have ALTER EXTENSION DROP :) Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding new variables into GUC
=?ISO-8859-7?B?yNzt7/Ig0OHw4fDd9PHv9Q==?= writes: > I am an MSc student in the department of Informatics and > Telecommunications of the University of Athens and as part of my > thesis I am examining a new path/plan cost model for DB optimizers. I > have successfully changed the optimizer of PostgreSQL in order to > implement this model, but I have stumbled upon a very little detail: > until now I use some hardcoded values in my code which I would like to > make accessible through GUC. After much googling the only relative > pages I have found are about configuring existing PostgreSQL variables > and src/backend/utils/misc/README does not mention anything about > adding new vars. Can anybody please provide some help? You basically need the variable, the entry in the appropriate array in guc.c, and some documentation (at least if you'd like anyone else to ever use the code). Try looking at some past patches that added GUCs similar to yours. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extensions versus pg_upgrade
Dimitri Fontaine writes: > Tom Lane writes: >> Actually, it occurs to me that the need for ALTER EXTENSION DROP could >> be upon us sooner than we think. The cases where an extension upgrade >> script would need that are >> (1) you want to remove some deprecated piece of the extension's API; >> (2) you want to remove some no-longer-needed internal function. >> Without ALTER EXTENSION DROP it's flat out impossible to do either. > What if you just DROP FUNCTION in the upgrade script? That would be rejected because you're not allowed to drop an individual member object of an extension. (And no, I don't want to have a kluge in dependency.c that makes that test work differently when creating_extension.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extensions versus pg_upgrade
Tom Lane writes: > Actually, it occurs to me that the need for ALTER EXTENSION DROP could > be upon us sooner than we think. The cases where an extension upgrade > script would need that are > (1) you want to remove some deprecated piece of the extension's API; > (2) you want to remove some no-longer-needed internal function. > Without ALTER EXTENSION DROP it's flat out impossible to do either. What if you just DROP FUNCTION in the upgrade script? That said if the function is used in some expression index or worse, triggers, you certainly want to give users the opportunity to delay the step where the function is no more part of the extension from the step where you get rid of it. But if the function is implemented in C and the newer shared object has removed it… > So I'm thinking it'd be smart to expend the small amount of additional > effort needed to support DROP right off the bat. I think that > AlterExtensionAddStmt could be extended with an add/drop boolean for > a net addition of only a few dozen lines of code, most of that being a > suitable search-and-delete function in pg_depend.c. Given your phrasing about the size of this project, I can't see any downside here. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extensions versus pg_upgrade
On Thu, Feb 10, 2011 at 10:41 AM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Feb 8, 2011 at 9:48 PM, Tom Lane wrote: >>> In contrast, ALTER EXTENSION ADD doesn't presuppose that you couldn't >>> add the object to multiple extensions; and it has a natural inverse, >>> ALTER EXTENSION DROP. I am not necessarily suggesting that we will ever >>> allow either of those things, but I do suggest that we should pick a >>> syntax that doesn't look like it's being forced to conform if we ever >>> want to do it. The DROP case at least seems like it might be wanted >>> in the relatively near future. > >> Yep. > > Actually, it occurs to me that the need for ALTER EXTENSION DROP could > be upon us sooner than we think. The cases where an extension upgrade > script would need that are > (1) you want to remove some deprecated piece of the extension's API; > (2) you want to remove some no-longer-needed internal function. > Without ALTER EXTENSION DROP it's flat out impossible to do either. > > Deprecated API is not exactly far to seek in our contrib modules, > either --- the example that just reminded me of this is hstore's => > operator, which we're already going so far as to print warnings about. > We're not going to get to remove that until at least one release after > we support ALTER EXTENSION DROP. > > So I'm thinking it'd be smart to expend the small amount of additional > effort needed to support DROP right off the bat. I think that > AlterExtensionAddStmt could be extended with an add/drop boolean for > a net addition of only a few dozen lines of code, most of that being a > suitable search-and-delete function in pg_depend.c. > > Any objections? No, I was pretty much just waiting for you to arrive at the same conclusion I'd already reached. :-) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extensions versus pg_upgrade
Robert Haas writes: > On Tue, Feb 8, 2011 at 9:48 PM, Tom Lane wrote: >> In contrast, ALTER EXTENSION ADD doesn't presuppose that you couldn't >> add the object to multiple extensions; and it has a natural inverse, >> ALTER EXTENSION DROP. I am not necessarily suggesting that we will ever >> allow either of those things, but I do suggest that we should pick a >> syntax that doesn't look like it's being forced to conform if we ever >> want to do it. The DROP case at least seems like it might be wanted >> in the relatively near future. > Yep. Actually, it occurs to me that the need for ALTER EXTENSION DROP could be upon us sooner than we think. The cases where an extension upgrade script would need that are (1) you want to remove some deprecated piece of the extension's API; (2) you want to remove some no-longer-needed internal function. Without ALTER EXTENSION DROP it's flat out impossible to do either. Deprecated API is not exactly far to seek in our contrib modules, either --- the example that just reminded me of this is hstore's => operator, which we're already going so far as to print warnings about. We're not going to get to remove that until at least one release after we support ALTER EXTENSION DROP. So I'm thinking it'd be smart to expend the small amount of additional effort needed to support DROP right off the bat. I think that AlterExtensionAddStmt could be extended with an add/drop boolean for a net addition of only a few dozen lines of code, most of that being a suitable search-and-delete function in pg_depend.c. Any objections? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
