[HACKERS] Introduction
Dear pgsql-hackers, My name is Sivasankar Ramasubramanian (you can call me Shiv). I am one of the students taking part in this years Google Summer of Code program. I am Indian but I study and live in Singapore (at the National University of Singapore). On the program I hope to learn as much about professional software engineering principles as PostgreSQL. My project is aimed towards extending and hopefully improving upon pgtune. If any of you have some ideas or thoughts to share. I am all ears!! Regards, Shiv
Re: [HACKERS] GSoC 2011: Fast GiST index build
On Tue, Apr 26, 2011 at 1:10 PM, Alexander Korotkov aekorot...@gmail.comwrote: Since algorithm is focused to reduce I/O, we should expect best acceleration in the case when index doesn't fitting to memory. Size of buffers is comparable to size of whole index. It means that if we can hold buffers in memory then we mostly can hold whole index in memory. That's why I think we should have simple on-disk buffers management for first representative benchmark. Since we need to free all buffers after index built, I believe that buffers should be stored separately. If not, index become bloat immediatly after creation. We probably need to create a temporary relation to store buffers in it. If my thought is right, then is there any example of using temporary relation? With best regards, Alexander Korotkov.
Re: [HACKERS] Proposal - asynchronous functions
On 04/26/2011 11:17 PM, Robert Haas wrote: IIRC, we kind of got stuck on the prerequisite wamalloc patch, and that sunk the whole thing. :-( Right, that prerequisite was the largest stumbling block. As I certainly mentioned back then, it should be possible to get rid of the imessages dependency (and thus wamalloc). So whoever really wants to implement asynchronous functions (or autonomous transactions) is more than welcome to try that. Please keep in mind that you'd need an alternative communication path. Not only for the bgworker infrastructure itself, but for communication between the requesting backend and the bgworker (except for fire-and-forget jobs like autovacuum, of course. OTOH even those could benefit from communicating back their state to the coordinator.. eh.. autovacuum launcher). Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011: Fast GiST index build
On 27.04.2011 09:51, Alexander Korotkov wrote: On Tue, Apr 26, 2011 at 1:10 PM, Alexander Korotkovaekorot...@gmail.comwrote: Since algorithm is focused to reduce I/O, we should expect best acceleration in the case when index doesn't fitting to memory. Size of buffers is comparable to size of whole index. It means that if we can hold buffers in memory then we mostly can hold whole index in memory. That's why I think we should have simple on-disk buffers management for first representative benchmark. Since we need to free all buffers after index built, I believe that buffers should be stored separately. If not, index become bloat immediatly after creation. We probably need to create a temporary relation to store buffers in it. If my thought is right, then is there any example of using temporary relation? A temporary relation is a bit heavy-weight for this, a temporary file should be enough. See src/backend/storage/file/buffile.c, BufFileCreateTemp() function in particular. Or perhaps a tuplestore suits better, see src/backend/utils/sort/tuplestore.c, that's simpler to use if you're storing tuples. tuplestore only supports storing heap tuples at the moment, but it could easily be extended to store index tuples, like tuplesort.c does. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Memory leak in FDW
On 26.04.2011 21:30, Tom Lane wrote:
Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes:
The trivial fix is to reset the per-tuple memory context between
iterations.
Have you tested this with SRFs?
ForeignNext seems like quite the wrong place for resetting
exprcontext in any case ...
ExecScan would be more appropriate I guess (attached).
This means the context will be reset between each tuple even for nodes
like seqscan that don't use the per-tuple context for anything.
AllocSetReset exits quickly if there's nothing to do, but it takes a
couple of function calls to get there. I wouldn't normally worry about
that, but this is a very hot path for simple queries.
I tested this with:
CREATE TABLE foo AS SELECT generate_series(1,1000);
I ran SELECT COUNT(*) FROM foo many times with \timing on, and took
the smallest time with and without the patch. I got:
1230 ms with the patch
1186 ms without the patch
This was quite repeatable, it's consistently faster without the patch.
That's a bigger difference than I expected. Any random code change can
swing results on micro-benchmarks like this by 1-2%, but that's over 3%.
Do we care?
I might be getting a bit carried away with this, but we could buy that
back by moving the isReset flag from AllocSetContext to
MemoryContextData. That way MemoryContextReset can exit more quickly if
there's nothing to do, patch attached.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
diff --git a/src/backend/executor/execScan.c b/src/backend/executor/execScan.c
index 5089616..e900588 100644
--- a/src/backend/executor/execScan.c
+++ b/src/backend/executor/execScan.c
@@ -120,13 +120,17 @@ ExecScan(ScanState *node,
*/
qual = node-ps.qual;
projInfo = node-ps.ps_ProjInfo;
+ econtext = node-ps.ps_ExprContext;
/*
* If we have neither a qual to check nor a projection to do, just skip
* all the overhead and return the raw scan tuple.
*/
if (!qual !projInfo)
+ {
+ ResetExprContext(econtext);
return ExecScanFetch(node, accessMtd, recheckMtd);
+ }
/*
* Check to see if we're still projecting out tuples from a previous scan
@@ -148,7 +152,6 @@ ExecScan(ScanState *node,
* storage allocated in the previous tuple cycle. Note this can't happen
* until we're done projecting out tuples from a scan tuple.
*/
- econtext = node-ps.ps_ExprContext;
ResetExprContext(econtext);
/*
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index e95dcb6..42f76b7 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -128,7 +128,7 @@ typedef void *AllocPointer;
* different from the aset being physically empty (empty blocks list) because
* we may still have a keeper block. It's also different from the set being
* logically empty, because we don't attempt to detect pfree'ing the last
- * active chunk.
+ * active chunk. XXX isReset is not here anymore, move comment
*/
typedef struct AllocSetContext
{
@@ -136,7 +136,6 @@ typedef struct AllocSetContext
/* Info about storage allocated in this context: */
AllocBlock blocks; /* head of list of blocks in this set */
AllocChunk freelist[ALLOCSET_NUM_FREELISTS]; /* free chunk lists */
- bool isReset; /* T = no space alloced since last reset */
/* Allocation parameters for this context: */
Size initBlockSize; /* initial block size */
Size maxBlockSize; /* maximum block size */
@@ -418,8 +417,6 @@ AllocSetContextCreate(MemoryContext parent,
context-keeper = block;
}
- context-isReset = true;
-
return (MemoryContext) context;
}
@@ -463,10 +460,6 @@ AllocSetReset(MemoryContext context)
AssertArg(AllocSetIsValid(set));
- /* Nothing to do if no pallocs since startup or last reset */
- if (set-isReset)
- return;
-
#ifdef MEMORY_CONTEXT_CHECKING
/* Check for corruption and leaks before freeing */
AllocSetCheck(context);
@@ -510,8 +503,6 @@ AllocSetReset(MemoryContext context)
/* Reset block size allocation sequence, too */
set-nextBlockSize = set-initBlockSize;
-
- set-isReset = true;
}
/*
@@ -620,8 +611,6 @@ AllocSetAlloc(MemoryContext context, Size size)
set-blocks = block;
}
- set-isReset = false;
-
AllocAllocInfo(set, chunk);
return AllocChunkGetPointer(chunk);
}
@@ -653,9 +642,6 @@ AllocSetAlloc(MemoryContext context, Size size)
randomize_mem((char *) AllocChunkGetPointer(chunk), size);
#endif
- /* isReset must be false already */
- Assert(!set-isReset);
-
AllocAllocInfo(set, chunk);
return AllocChunkGetPointer(chunk);
}
@@ -813,8 +799,6 @@ AllocSetAlloc(MemoryContext context, Size size)
randomize_mem((char *) AllocChunkGetPointer(chunk), size);
#endif
- set-isReset = false;
-
AllocAllocInfo(set, chunk);
return AllocChunkGetPointer(chunk);
}
@@ -913,9 +897,6 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
set-header.name, chunk);
#endif
- /* isReset must be false already */
-
[HACKERS] Predicate locking
I'm currently need predicate locking in the project, so there are two ways to get it by now: implement it by creating special database records to lock with SELECT FOR UPDATE or wait while they will be implemented in Postgres core. Is there something like predicate locking on the TODO list currently? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Predicate locking
2011/4/27 Vlad Arkhipov arhi...@dc.baikal.ru: I'm currently need predicate locking in the project, so there are two ways to get it by now: implement it by creating special database records to lock with SELECT FOR UPDATE or wait while they will be implemented in Postgres core. Is there something like predicate locking on the TODO list currently? I assume you want (real, as opposed to what is in 9.1 now) SERIALIZABLE transactions, in which case you could check: URL:http://wiki.postgresql.org/wiki/Serializable Nicolas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Predicate locking
27.04.2011 17:45, Nicolas Barbier: 2011/4/27 Vlad Arkhipovarhi...@dc.baikal.ru: I'm currently need predicate locking in the project, so there are two ways to get it by now: implement it by creating special database records to lock with SELECT FOR UPDATE or wait while they will be implemented in Postgres core. Is there something like predicate locking on the TODO list currently? I assume you want (real, as opposed to what is in 9.1 now) SERIALIZABLE transactions, in which case you could check: URL:http://wiki.postgresql.org/wiki/Serializable Nicolas Not sure about the whole transaction, I think it degrades the performance too much as transactions access many tables. Just wanted SELECT FOR UPDATE to prevent inserting records into a table with the specified condition. It seems to be very typical situation when you have a table like CREATE TABLE timetable (start_ts TIMESTAMP, end_ts TIMESTAMP) and before insertion in this table want to guarantee that there is no overlapped time intervals there. So, first you need to lock the range in the table, then to check if there are any records in this range. In my case this table is the only for which I need such kind of locking. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Predicate locking
On 27.04.2011 12:24, Vlad Arkhipov wrote: 27.04.2011 17:45, Nicolas Barbier: 2011/4/27 Vlad Arkhipovarhi...@dc.baikal.ru: I'm currently need predicate locking in the project, so there are two ways to get it by now: implement it by creating special database records to lock with SELECT FOR UPDATE or wait while they will be implemented in Postgres core. Is there something like predicate locking on the TODO list currently? I assume you want (real, as opposed to what is in 9.1 now) SERIALIZABLE transactions, in which case you could check: URL:http://wiki.postgresql.org/wiki/Serializable Nicolas Not sure about the whole transaction, I think it degrades the performance too much as transactions access many tables. Just wanted SELECT FOR UPDATE to prevent inserting records into a table with the specified condition. It seems to be very typical situation when you have a table like CREATE TABLE timetable (start_ts TIMESTAMP, end_ts TIMESTAMP) and before insertion in this table want to guarantee that there is no overlapped time intervals there. So, first you need to lock the range in the table, then to check if there are any records in this range. In my case this table is the only for which I need such kind of locking. You can do that with exclusion constraints: http://www.postgresql.org/docs/9.0/static/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSION) See also Depesz's blog post for a specific example on how to use it for time ranges: http://www.depesz.com/index.php/2010/01/03/waiting-for-8-5-exclusion-constraints/ And Jeff Davis's blog post that uses the period data type instead of the hack to represent time ranges as boxes: http://thoughts.j-davis.com/2009/11/08/temporal-keys-part-2/ -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] alpha5
On 26.04.2011 17:36, Alvaro Herrera wrote: Excerpts from Alvaro Herrera's message of sáb abr 09 01:55:45 -0300 2011: Excerpts from Peter Eisentraut's message of lun mar 28 17:00:01 -0300 2011: On mån, 2011-03-28 at 09:35 -0400, Robert Haas wrote: Actually those are all my fault. Sorry, I'm still learning the ropes. I didn't realize xref couldn't be used in the release notes; it looks like Bruce usedlink rather thanxref for the things I usedxref for. This is the sort of thing for which make maintainer-check would be very useful. And/or we could add the creation of these files to make doc or make world or something. I suggest having them be created in doc/src/sgml all target. This gets them in make docs and make world AFAICT. This is a trivial patch (attached). The only problem with it is that make all would fail if lynx is not installed. Does it need Internet connection? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] maximum digits for NUMERIC
On Wed, Apr 27, 2011 at 4:47 AM, Bruce Momjian br...@momjian.us wrote: Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of mar abr 26 12:58:19 -0300 2011: Wow, I am so glad someone documented this. I often do factorial(4000) which generates 12673 digits when teaching classes, and it bugged me that we documented the limit as 1000 digits. I keep wondering why you want to know factorial(4000) so often. It is just to impress folks, and it is impressive. An instant screenful of digits is pretty cool. If you are into impressing people with big numbers (or maybe doing something useful with them too) you may take a look at http://pgmp.projects.postgresql.org/ -- Daniele -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
On 04/27/2011 12:50 AM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: All or almost all the warnings seen on Windows/Mingw of the type warning: unknown conversion type character 'm' in format come from checking of three functions: errmsg, elog and errdetail. I therefore propose to disable the attribute checking of those three functions, on Windows only (since that's the only place I've seen the warnings). That's a much more conservative change than I made previously which would have turned off all format warnings on Mingw, and along with fixing the INT64_FORMAT (see email just sent) would fix the vast majority of compiler warnings, so we'd be almost clean again on MinGW. That seems to me to be throwing the baby out with the bathwater. If Windows could be assumed to be just like every other platform, we could maybe figure that being format-warning-free elsewhere was sufficient checking; but that assumption is obviously wrong. We're not doing anything about the warnings, and I'm not sure there's anything we can do other than suppress them or live with them. The compiler is in fact quite correct, it doesn't know anything about %m, and if we were ever to use %m in a context where we actually expected it to output the contents of strerror(errno) the warning would be lost among a huge pile of these other warnings where its use is harmless because we expand it ourselves. That strikes me as a more potent danger. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] fixing INT64_FORMAT warnings on Mingw
On Wed, 2011-04-27 at 00:01 -0400, Andrew Dunstan wrote: Both %lld and %I64d can be used with mingw to print 64 bit integers. However, modern versions of gcc spit warnings with the former, and not the latter. However, since %lld works, it is chosen by our config setup since it comes first in the list of formats tried. Therefore, to keep the compiler happy I proposed to rearrange that so that %I64d is preferred if it's found to work. We should prefer the standard syntax (%lld) over nonstandard ways. There could just as well be platforms that accept %I64d but warn about it (as being nonstandard). If that doesn't work for a specific platform, either adjust the test so that it checks for warnings, or just manually override the result in pg_config_os.h. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fail to search in array, produced by subquery - is it a bug?
On Tue, Apr 26, 2011 at 10:29 PM, Dmitry Fefelov fo...@ac-sw.com wrote:
With Postgres 8.4 query like
SELECT *
FROM core.tag_links ctl
WHERE (ctl.tag_id = ANY (
SELECT array_agg(ct.id)
FROM core.tags ct
WHERE (LOWER(ct.tag) LIKE LOWER(('search tag')::text || '%') ESCAPE
E'\\')
));
well, if you *had* to use any you could rewrite that as:
SELECT *
FROM core.tag_links ctl
WHERE (ctl.tag_id = ANY ( array (
SELECT ct.id
FROM core.tags ct
WHERE (LOWER(ct.tag) LIKE LOWER(('search tag')::text || '%') ESCAPE
E'\\'))
));
but you're far better off still using 'where in/'where exists' for
this query. You could also expand an array with 'unnest' and use
'where in'.
according to the documentation, 'any' only takes array
expressions...this feels really awkward but i'm not sure if it's a
bug. the semantics of 'any' suck and I prefer not to use it :(.
this has come up a bunch of times in the archives (unfortunately,
searching for 'any' isn't pleasant) and I think there's an explanation
behind the current behavior. Couldn't find it though, so I'm not
sure.
merlin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] stored procedures - use cases?
On Tue, Apr 26, 2011 at 10:12 PM, Greg Stark gsst...@mit.edu wrote: On Tue, Apr 26, 2011 at 11:55 PM, Josh Berkus j...@agliodbs.com wrote: Here's where I wanted autonomous transactions just last week, and didn't have them so I had to use a python script outside the database: -- doing a CREATE INDEX CONCURRENTLY for 150 partitions on a partitioned table. -- doing a backfill operation for 10GB of computed data, taking 8 hours, where I don't want to hold a transaction open for 8 hours since this is a high-volume OLTP database. These don't seem like compelling use cases at all to me. You said you had to fall back to using a python script outside the database, but what disadvantage does that have? Why is moving your application logic into the database an improvement? Honestly in every case where I've had to move code that had been in a function to the application I've found there were tons of benefits. Everything from being able to better control the behaviour, to being able to parallelize the processing over multiple connections, being able to run parts of it at different times, being able to see the progress and control it from another session, being able to manage the code in version control, the list just goes on. Trying to move all the code into the database just makes life harder. my experience has been the opposite. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] operator classes for index?
Am 26.04.2011 17:37, schrieb Tom Lane: =?ISO-8859-1?Q?Yves_Wei=DFig?= weis...@rbg.informatik.tu-darmstadt.de writes: Am 26.04.2011 14:28, schrieb Robert Haas: On Tue, Apr 26, 2011 at 5:18 AM, Yves Weißig weis...@rbg.informatik.tu-darmstadt.de wrote: CREATE OPERATOR CLASS abstime_ops DEFAULT FOR TYPE abstime USING ebi FAMILY abstime_ops AS OPERATOR 1 = (abstime,abstime), FUNCTION 1 hashint4(abstime,abstime); it yields: ERROR: function hashint4(abstime, abstime) does not exist My copy of PostgreSQL has a hashint4(integer) function, but no hashint4(abstime, abstime) function. Yes, I know, maybe my question wasn't clear enough. Following statement: ... I get: hash;abstime_ops;hashint4;2227;702;702;1;hashint4;abstime;abstime as an entry and suppose that hashint4 also takes abstime How is it done? How is hashint4 used to hash a value of abstime? Cheating ;-). That entry is hard-wired in pg_amproc.h so it does not pass through the same kind of error checking that CREATE OPERATOR CLASS applies. It works, physically, because abstime and integer are binary compatible (both 4-byte int-aligned pass-by-value types), but the catalog entries are a bit inconsistent. If we wanted to make this look completely clean, we'd have to create an alias function that was declared to take abstime. For instance you could do it like this: create function hashabstime(abstime) returns int4 as 'hashint4' language internal strict immutable; and then say FUNCTION 1 hashabstime(abstime) in CREATE OPERATOR CLASS. You might find this extract from the opr_sanity regression test instructive: -- For hash we can also do a little better: the support routines must be -- of the form hash(lefttype) returns int4. There are several cases where -- we cheat and use a hash function that is physically compatible with the -- datatype even though there's no cast, so this check does find a small -- number of entries. SELECT p1.amprocfamily, p1.amprocnum, p2.proname, p3.opfname FROM pg_amproc AS p1, pg_proc AS p2, pg_opfamily AS p3 WHERE p3.opfmethod = (SELECT oid FROM pg_am WHERE amname = 'hash') AND p1.amprocfamily = p3.oid AND p1.amproc = p2.oid AND (amprocnum != 1 OR proretset OR prorettype != 'int4'::regtype OR pronargs != 1 OR NOT physically_coercible(amproclefttype, proargtypes[0]) OR amproclefttype != amprocrighttype) ORDER BY 1; amprocfamily | amprocnum |proname | opfname --+---++- 435 | 1 | hashint4 | date_ops 1999 | 1 | timestamp_hash | timestamptz_ops | 1 | hashchar | bool_ops 2223 | 1 | hashvarlena| bytea_ops 2225 | 1 | hashint4 | xid_ops 2226 | 1 | hashint4 | cid_ops (6 rows) regards, tom lane Thanks so much Tom, I was really loosing my mind on this one... now it works! Awesome. Yves -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new AM, catalog entries
Am 26.04.2011 17:07, schrieb Alvaro Herrera: Excerpts from Yves Weißig's message of mar abr 26 11:32:31 -0300 2011: I keep getting: ERROR: there is no built-in function named ebibuild This error message somehow leads me to fmgr.c where the contents of an array are inspected (in line 134). This array fmgr_builtins is built by fmgr_builtins Gen_fmgrtab.pl from pg_proc.h (when? during make?) Yes, during make. Do I have to edit pg_proc.h manually to add my methods for the new AM? Yes. Or maybe you can try using language C instead of internal (if you have it in a shared library). Thanks for the answer and a little bit more insight into pg! Yves -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Best way to construct Datum out of a string?
Hi, sadly, so far my search in the source code wasn't very successfull on this topic. So, how can I construct a Datum out of a string? Greetz, Yves -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Best way to construct Datum out of a string?
On 27.04.2011 17:06, Yves Weißig wrote: Hi, sadly, so far my search in the source code wasn't very successfull on this topic. So, how can I construct a Datum out of a string? What kind of a Datum do you want it to be? What data type? See CStringGetDatum, or perhaps CStringGetTextDatum(). Or perhaps you want to call the input function of some other datatype, with InputFunctionCall. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Best way to construct Datum out of a string?
On 04/27/2011 10:06 AM, Yves Weißig wrote: Hi, sadly, so far my search in the source code wasn't very successfull on this topic. So, how can I construct a Datum out of a string? CStringGetDatum() The code is replete with examples, cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] fixing INT64_FORMAT warnings on Mingw
On 04/27/2011 09:00 AM, Peter Eisentraut wrote: On Wed, 2011-04-27 at 00:01 -0400, Andrew Dunstan wrote: Both %lld and %I64d can be used with mingw to print 64 bit integers. However, modern versions of gcc spit warnings with the former, and not the latter. However, since %lld works, it is chosen by our config setup since it comes first in the list of formats tried. Therefore, to keep the compiler happy I proposed to rearrange that so that %I64d is preferred if it's found to work. We should prefer the standard syntax (%lld) over nonstandard ways. There could just as well be platforms that accept %I64d but warn about it (as being nonstandard). If that doesn't work for a specific platform, either adjust the test so that it checks for warnings, or just manually override the result in pg_config_os.h. I did it that way. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
Andrew Dunstan and...@dunslane.net writes: On 04/27/2011 12:50 AM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: All or almost all the warnings seen on Windows/Mingw of the type warning: unknown conversion type character 'm' in format come from checking of three functions: errmsg, elog and errdetail. I therefore propose to disable the attribute checking of those three functions, on Windows only (since that's the only place I've seen the warnings). That seems to me to be throwing the baby out with the bathwater. If Windows could be assumed to be just like every other platform, we could maybe figure that being format-warning-free elsewhere was sufficient checking; but that assumption is obviously wrong. We're not doing anything about the warnings, and I'm not sure there's anything we can do other than suppress them or live with them. The compiler is in fact quite correct, it doesn't know anything about %m, and if we were ever to use %m in a context where we actually expected it to output the contents of strerror(errno) the warning would be lost among a huge pile of these other warnings where its use is harmless because we expand it ourselves. That strikes me as a more potent danger. I don't buy that. The risk that gcc will let past a '%m' without complaint, in a function that doesn't actually support it, exists on most non-Linux platforms (ie pretty much anywhere you use gcc with non-GNU libc), and has existed from the beginning. Despite this, I cannot recall that we have ever had a bug of that ilk. But we have most certainly had bugs with incorrect/unportable matching of other format arguments. I think losing the ability to detect the latter in Windows-specific code is a terrible price to pay for silencing an easily-ignorable class of warnings. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
On 04/27/2011 10:29 AM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: On 04/27/2011 12:50 AM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: All or almost all the warnings seen on Windows/Mingw of the type warning: unknown conversion type character 'm' in format come from checking of three functions: errmsg, elog and errdetail. I therefore propose to disable the attribute checking of those three functions, on Windows only (since that's the only place I've seen the warnings). That seems to me to be throwing the baby out with the bathwater. If Windows could be assumed to be just like every other platform, we could maybe figure that being format-warning-free elsewhere was sufficient checking; but that assumption is obviously wrong. We're not doing anything about the warnings, and I'm not sure there's anything we can do other than suppress them or live with them. The compiler is in fact quite correct, it doesn't know anything about %m, and if we were ever to use %m in a context where we actually expected it to output the contents of strerror(errno) the warning would be lost among a huge pile of these other warnings where its use is harmless because we expand it ourselves. That strikes me as a more potent danger. I don't buy that. The risk that gcc will let past a '%m' without complaint, in a function that doesn't actually support it, exists on most non-Linux platforms (ie pretty much anywhere you use gcc with non-GNU libc), and has existed from the beginning. Despite this, I cannot recall that we have ever had a bug of that ilk. But we have most certainly had bugs with incorrect/unportable matching of other format arguments. I think losing the ability to detect the latter in Windows-specific code is a terrible price to pay for silencing an easily-ignorable class of warnings. What I'd like to know is why it doesn't complain elsewhere. The one non-Linux non-Windows machine I have is FBSD. Its gcc (4.2.1) doesn't expand %m but doesn't complain about it either. It does complain about other unknown formats. I wonder if they have patched gcc to silence the warnings? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
Andrew Dunstan and...@dunslane.net wrote: What I'd like to know is why it doesn't complain elsewhere. It appears to be allowed as an extension on some compliers. http://sourceware.org/ml/newlib/2006/msg00079.html -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
Andrew Dunstan and...@dunslane.net writes: What I'd like to know is why it doesn't complain elsewhere. That question is backwards ... The one non-Linux non-Windows machine I have is FBSD. Its gcc (4.2.1) doesn't expand %m but doesn't complain about it either. It's libc, not gcc, that's actually got the responsibility of processing format specifiers at runtime. gcc just assumes a particular behavior of libc. I quote from the gcc 4.4.5 manual, under -Wformat: The formats are checked against the format features supported by GNU libc version 2.2. These include all ISO C90 and C99 features, as well as features from the Single Unix Specification and some BSD and GNU extensions. Other library implementations may not support all these features; GCC does not support warning about features that go beyond a particular library's limitations. So the question to ask is not why gcc doesn't complain about %m elsewhere, but why it does complain in your Windows installation. I'm guessing that the mingw people hacked it. If you're lucky, they might have hacked in an extra switch to control the behavior --- I notice quite a few subsidiary switches that tweak -Wformat behavior in standard gcc 4.4.5. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] alpha5
Excerpts from Heikki Linnakangas's message of mié abr 27 07:47:32 -0300 2011: On 26.04.2011 17:36, Alvaro Herrera wrote: I suggest having them be created in doc/src/sgml all target. This gets them in make docs and make world AFAICT. This is a trivial patch (attached). The only problem with it is that make all would fail if lynx is not installed. Does it need Internet connection? No, it only dumps the local HTML file into plain text. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alignment padding bytes in arrays vs the planner
Greg Stark gsst...@mit.edu writes: On Wed, Apr 27, 2011 at 12:23 AM, Tom Lane t...@sss.pgh.pa.us wrote: Any ideas about better answers? Here's a crazy idea. We could use string equality of the out function's representation instead. If an output function doesn't consistently output the same data for things that are equal or different data for things that aren't equal then there's a bug in it already since it means the data type won't survive a pg_dump/reload. Hmm, cute idea, but existing output functions don't actually cooperate with this goal very well: * float4_out, float8_out, and most geometry types don't promise to produce invertible results unless extra_float_digits is set high enough. * timestamptz_out and friends react to both DateStyle and TimeZone settings, which means you lose in the other direction: identical values could print differently at different times. This is a killer for the planner's use since what it's doing is precisely comparing constants generated during index or rule creation to those appearing in queries. I think if we were going to go down this road, we'd have little choice but to invent the specific concept of a type-specific identity function, foo_identical(foo, foo) returns bool. This wouldn't present any pain for most datatype authors since omitting it would just license the system to assume bitwise equality is the right behavior. As far as the other issues I mentioned go: * We could dodge most of the performance hit if Const carried a flag indicating whether the datatype has an identity function or not. This would allow equal() to fall through without a table lookup in the large majority of cases. It wouldn't add any expense to Const construction since you already have to know or fetch other datatype properties like pass-by-value. * We could likely finesse the transactional-safety issue by allowing equal() to fall back on bitwise comparison if not IsTransactionState. I'm still not sure whether there actually are any cases where it has to work outside a transaction, and if there are, the dumb/conservative behavior is probably adequate. Still, it'd be a lot of work, and it doesn't offer any chance of a back-patchable fix. Looking at Noah's valgrind results, I'm inclined to just go seal off the known holes instead. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alignment padding bytes in arrays vs the planner
Noah Misch n...@leadboat.com writes: On Tue, Apr 26, 2011 at 07:23:12PM -0400, Tom Lane wrote: [input functions aren't the only problematic source of uninitialized datum bytes] FWIW, when I was running the test suite under valgrind, these were the functions that left uninitialized bytes in datums: array_recv, array_set, array_set_slice, array_map, construct_md_array, path_recv. If the test suite covers this well, we're not far off. (Actually, I only had the check in PageAddItem ... probably needed to be in one or two other places to catch as much as possible.) Hmm. Eyeballing arrayfuncs.c yesterday, I noted the following functions using palloc where palloc0 would be safer: array_recv array_get_slice array_set array_set_slice array_map construct_md_array construct_empty_array The last may not be an actual hazard since I think there are no pad bytes in its result, but on the other hand palloc0 is cheap insurance for it. I hadn't looked at the geometry functions but padding in paths isn't surprising at all. When dealing with very large arrays, there might be a case to be made for not using palloc0 but trying to zero just what needs zeroed. However that looks a bit complicated to get right, and it's not impossible that it could end up being slower, since it would add per-element processing to fill pad bytes instead of just skipping over them. (memset is pretty damn fast on most machines ...) For the moment I'm just going to do s/palloc/palloc0/ as a reliable and back-patchable fix --- possibly in future someone will care to look into whether the other way is a performance win. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
On 04/27/2011 11:02 AM, Tom Lane wrote: So the question to ask is not why gcc doesn't complain about %m elsewhere, but why it does complain in your Windows installation. I'm guessing that the mingw people hacked it. If you're lucky, they might have hacked in an extra switch to control the behavior --- I notice quite a few subsidiary switches that tweak -Wformat behavior in standard gcc 4.4.5. Hmm. The error disappears if I use -D__USE_MINGW_ANSI_STDIO=1 or -posix I don't know what other effects that might have, though. There's a description here: http://sourceforge.net/project/shownotes.php?release_id=24832 I'll experiment and see what happens. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
timeline garbage in pg_basebackup (was [HACKERS] gcc 4.6 warnings -Wunused-but-set-variable)
On tis, 2011-03-29 at 23:48 +0300, Peter Eisentraut wrote: The line I marked in pg_basebackup.c might be an actual problem: It goes through a whole lot to figure out the timeline and then doesn't do anything with it. This hasn't been addressed yet. It doesn't manifest itself as an actual problem, but it looks as though someone had intended something in that code and the code doesn't do that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] gcc 4.6 warnings -Wunused-but-set-variable
On tis, 2011-03-29 at 23:48 +0300, Peter Eisentraut wrote: As you might have heard, GCC 4.6 was released the other day. It generates a bunch of new warnings with the PostgreSQL source code, most of which belong to the new warning scenario -Wunused-but-set-variable, which is included in -Wall. In case someone else tries that, I have filed a bug with GCC regarding some of the other warnings: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48778 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: timeline garbage in pg_basebackup (was [HACKERS] gcc 4.6 warnings -Wunused-but-set-variable)
On Wed, Apr 27, 2011 at 18:55, Peter Eisentraut pete...@gmx.net wrote: On tis, 2011-03-29 at 23:48 +0300, Peter Eisentraut wrote: The line I marked in pg_basebackup.c might be an actual problem: It goes through a whole lot to figure out the timeline and then doesn't do anything with it. This hasn't been addressed yet. It doesn't manifest itself as an actual problem, but it looks as though someone had intended something in that code and the code doesn't do that. Do you have a ref to the actual problem? The subject change killed my threading, the email was trimmed to not include the actual problem, and it appears not to be listed on the open items list... ;) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI non-serializable UPDATE performance
On Mon, Apr 25, 2011 at 4:33 AM, Dan Ports d...@csail.mit.edu wrote: On Sat, Apr 23, 2011 at 08:54:31AM -0500, Kevin Grittner wrote: Even though this didn't show any difference in Dan's performance tests, it seems like reasonable insurance against creating a new bottleneck in very high concurrency situations. Dan, do you have a patch for this, or should I create one? Sure, patch is attached. Reading the code, IIUC, we check for RW conflicts after each write but only if the writer is running a serializable transaction. Am I correct in thinking that there is zero impact of SSI if nobody is running a serializable transaction? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] stored procedures - use cases?
These don't seem like compelling use cases at all to me. You said you had to fall back to using a python script outside the database, but what disadvantage does that have? Why is moving your application logic into the database an improvement? Since both were part of a code rollout, it complicated our deployment process considerably and took a deployment which could have been push-button automatic and forced us to do it by manually logging into the shell on the database server. Trying to move all the code into the database just makes life harder. I might make *your* life harder. It makes *mine* easier. If you pursue your argument a little further, Greg, why do we have functions at all? We could do it all in the application. Autonomous transactions have value on their own. But it's not so that you can run create index ocncurrently or vacuum or whatever. Why not? Why are you so intent on making my life harder? They're useful so that a single session can do things like log errors even when a transaction rolls back. That's *also* an excellent use case. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI non-serializable UPDATE performance
On Wed, Apr 27, 2011 at 06:26:52PM +0100, Simon Riggs wrote: Reading the code, IIUC, we check for RW conflicts after each write but only if the writer is running a serializable transaction. Am I correct in thinking that there is zero impact of SSI if nobody is running a serializable transaction? That is correct, now. (Well, other than having to check whether a serializable transaction is running, the cost of which is truly negligible.) Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI non-serializable UPDATE performance
Simon Riggs si...@2ndquadrant.com wrote: Reading the code, IIUC, we check for RW conflicts after each write but only if the writer is running a serializable transaction. Correct as far as that statement goes. There are cases where predicate lock maintenance is needed when dealing with locked resources; see below. Am I correct in thinking that there is zero impact of SSI if nobody is running a serializable transaction? Benchmarks until a recent change actually were coming out slightly faster with the SSI patch. Since there's not other reason that should have been true, that pretty much had to be the random alignment of code on CPU cache boundaries. A recent change canceled that improvement; until we reverted SSI entirely for a comparative benchmark point we briefly thought SSI caused an overall fraction of a percent regression. We've considered changing the call SSI method which does a quick return if not applicable technique to try to call SSI with a macro which skips the call if not applicable, but in the absence of any evidence of a performance hit with the simple, straightforward approach we have held off on that optimization attempt. There are cases where some minimal work is done in SSI for non-serializable transactions: (1) If a tuple which is predicate locked, or sits on a predicate- locked page, is updated, the predicate lock is duplicated for the new tuple. We have found patterns of updates involving four or more transactions where a non-serializable transaction can hide serialization anomalies among serializable transactions if we don't do this. Someone suggested that we could take out this call and just document that serializable transactions may not comply with the standard-defined behavior when there are concurrent non-serializable transactions. We were unable to show a measurable performance hit on this, although this was just with 32 clients hitting a 16 processor machine. There was at least a theoretical possibility that with higher levels of concurrency there could have been a new contention point for a LW lock here which could affect performance. We added a quick return which didn't need to check any locks at the front of this routine which is taken if there are no active serializable transactions on the cluster at the moment of update. (2) Page splits and page combines, even if they are from non-serializable transactions (like autovacuum) must take action if there are predicate locks on the pages involved. Again, there is a fast exit if no serializable transactions are active, and even before that check was added there was not a measurable impact on performance. If any of that isn't clear or leaves some concern, please let me know. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: timeline garbage in pg_basebackup (was [HACKERS] gcc 4.6 warnings -Wunused-but-set-variable)
On ons, 2011-04-27 at 19:17 +0200, Magnus Hagander wrote: On Wed, Apr 27, 2011 at 18:55, Peter Eisentraut pete...@gmx.net wrote: On tis, 2011-03-29 at 23:48 +0300, Peter Eisentraut wrote: The line I marked in pg_basebackup.c might be an actual problem: It goes through a whole lot to figure out the timeline and then doesn't do anything with it. This hasn't been addressed yet. It doesn't manifest itself as an actual problem, but it looks as though someone had intended something in that code and the code doesn't do that. Do you have a ref to the actual problem? The subject change killed my threading, the email was trimmed to not include the actual problem, and it appears not to be listed on the open items list... ;) In BaseBackup(), the variable timeline is assigned in a somewhat elaborate fashion, but then the result is not used for anything. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typed-tables patch broke pg_upgrade
Here is the patch to fix that, as discussed.
diff --git i/src/bin/pg_dump/pg_dump.c w/src/bin/pg_dump/pg_dump.c
index c2f6180..afc7fd7 100644
--- i/src/bin/pg_dump/pg_dump.c
+++ w/src/bin/pg_dump/pg_dump.c
@@ -12004,7 +12004,11 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
UNLOGGED : ,
reltypename,
fmtId(tbinfo-dobj.name));
- if (tbinfo-reloftype)
+ /*
+ * In case of a binary upgrade, we dump the table normally and attach
+ * it to the type afterward.
+ */
+ if (tbinfo-reloftype !binary_upgrade)
appendPQExpBuffer(q, OF %s, tbinfo-reloftype);
actual_atts = 0;
for (j = 0; j tbinfo-numatts; j++)
@@ -12032,7 +12036,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
bool has_notnull = (tbinfo-notnull[j]
(!tbinfo-inhNotNull[j] || binary_upgrade));
-if (tbinfo-reloftype !has_default !has_notnull)
+if (tbinfo-reloftype !has_default !has_notnull !binary_upgrade)
continue;
/* Format properly if not first attr */
@@ -12060,7 +12064,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
}
/* Attribute type */
-if (tbinfo-reloftype)
+if (tbinfo-reloftype !binary_upgrade)
{
appendPQExpBuffer(q, WITH OPTIONS);
}
@@ -12126,7 +12130,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
if (actual_atts)
appendPQExpBuffer(q, \n));
- else if (!tbinfo-reloftype)
+ else if (!(tbinfo-reloftype !binary_upgrade))
{
/*
* We must have a parenthesized attribute list, even though empty,
@@ -12192,6 +12196,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
* an INHERITS clause --- the latter would possibly mess up the column
* order. That also means we have to take care about setting
* attislocal correctly, plus fix up any inherited CHECK constraints.
+ * Analogously, we set up typed tables using ALTER TABLE / OF here.
*/
if (binary_upgrade tbinfo-relkind == RELKIND_RELATION)
{
@@ -12268,6 +12273,14 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
}
}
+ if (tbinfo-reloftype)
+ {
+appendPQExpBuffer(q, \n-- For binary upgrade, set up typed tables this way.\n);
+appendPQExpBuffer(q, ALTER TABLE ONLY %s OF %s;\n,
+ fmtId(tbinfo-dobj.name),
+ tbinfo-reloftype);
+ }
+
appendPQExpBuffer(q, \n-- For binary upgrade, set heap's relfrozenxid\n);
appendPQExpBuffer(q, UPDATE pg_catalog.pg_class\n
SET relfrozenxid = '%u'\n
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: timeline garbage in pg_basebackup (was [HACKERS] gcc 4.6 warnings -Wunused-but-set-variable)
On Wed, Apr 27, 2011 at 20:21, Peter Eisentraut pete...@gmx.net wrote: On ons, 2011-04-27 at 19:17 +0200, Magnus Hagander wrote: On Wed, Apr 27, 2011 at 18:55, Peter Eisentraut pete...@gmx.net wrote: On tis, 2011-03-29 at 23:48 +0300, Peter Eisentraut wrote: The line I marked in pg_basebackup.c might be an actual problem: It goes through a whole lot to figure out the timeline and then doesn't do anything with it. This hasn't been addressed yet. It doesn't manifest itself as an actual problem, but it looks as though someone had intended something in that code and the code doesn't do that. Do you have a ref to the actual problem? The subject change killed my threading, the email was trimmed to not include the actual problem, and it appears not to be listed on the open items list... ;) In BaseBackup(), the variable timeline is assigned in a somewhat elaborate fashion, but then the result is not used for anything. Ah, I see it. What happened there is I accidentally included it when I split my patches apart. It's required in the stream WAL in parallel to the base backup to decrease requirements on wal_keep_segmtents. But that patch was postponed since there were still bugs in it, and it wasn't entirely feature-complete, and we were pretty far past feature-freeze. So it's not needed in 9.1. I'll rip it out and move it over to the patch once it's ready to go for 9.2. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade automatic testing
On tor, 2011-04-07 at 23:02 +0300, Peter Eisentraut wrote:
Seeing that 9.1-to-9.1 pg_upgrade has apparently been broken for months,
it would probably be good to have some kind of automatic testing for it.
Attached is something I hacked together that at least exposes the
current problems, easily available by typing make check and waiting.
It does not yet fully implement the full testing procedure in the
TESTING file, in particular the diffing of the dumps (well, because you
can't get there yet).
Is that something that people are interested in refining?
(I think it would even be possible under this setup to create special
regression test cases that are only run under the pg_upgrade test run,
to exercise particularly tricky upgrade cases.)
Now that this is bound to be fixed, here is an updated script that runs
the entire test procedure including diffing the dumps at the end.
diff --git i/contrib/pg_upgrade/Makefile w/contrib/pg_upgrade/Makefile
index 8f3fd7c..9ec7bc0 100644
--- i/contrib/pg_upgrade/Makefile
+++ w/contrib/pg_upgrade/Makefile
@@ -21,3 +21,6 @@ top_builddir = ../..
include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
+
+check: test.sh
+ MAKE=$(MAKE) bindir=$(bindir) $(SHELL) $
diff --git i/contrib/pg_upgrade/test.sh w/contrib/pg_upgrade/test.sh
index e69de29..bfbbb1c 100644
--- i/contrib/pg_upgrade/test.sh
+++ w/contrib/pg_upgrade/test.sh
@@ -0,0 +1,43 @@
+set -eux
+
+: ${MAKE=make}
+temp_root=$PWD/tmp_check
+
+temp_install=$temp_root/install
+bindir=$temp_install/$bindir
+PATH=$bindir:$PATH
+export PATH
+
+PGDATA=$temp_root/data
+export PGDATA
+rm -rf $PGDATA $PGDATA.old
+PGPORT=65432
+export PGPORT
+
+logdir=$PWD/log
+rm -rf $logdir
+mkdir $logdir
+
+$MAKE -C ../.. install DESTDIR=$temp_install 21 | tee $logdir/install.log
+$MAKE -C ../pg_upgrade_support install DESTDIR=$temp_install 21 | tee -a $logdir/install.log
+$MAKE -C . install DESTDIR=$temp_install 21 | tee -a $logdir/install.log
+
+initdb 21 | tee $logdir/initdb1.log
+pg_ctl start -l $logdir/postmaster1.log -w
+$MAKE -C ../.. installcheck 21 | tee $logdir/installcheck.log
+pg_dumpall $temp_root/dump1.sql
+pg_ctl -m fast stop
+
+mv ${PGDATA} ${PGDATA}.old
+
+initdb 21 | tee $logdir/initdb2.log
+
+unset PGPORT
+
+pg_upgrade -d ${PGDATA}.old -D ${PGDATA} -b $bindir -B $bindir
+
+pg_ctl start -l $logdir/postmaster2.log -w
+pg_dumpall $temp_root/dump2.sql
+pg_ctl -m fast stop
+
+diff -q $temp_root/dump1.sql $temp_root/dump2.sql
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] alpha5
On tis, 2011-04-26 at 11:36 -0300, Alvaro Herrera wrote: This is a trivial patch (attached). The only problem with it is that make all would fail if lynx is not installed. Well, that will probably not be acceptable. I rather think the action item here is to set up a buildfarm-ish job that runs make distcheck, which is designed to, well, check if the dist-making works. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typed-tables patch broke pg_upgrade
Peter Eisentraut pete...@gmx.net writes: Here is the patch to fix that, as discussed. Looks sane --- I assume you tested it against the originally complained-of scenario? http://archives.postgresql.org/message-id/201103111328.p2bdsfd10...@momjian.us If so, please apply soon --- we need to wrap beta1 this evening. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] alpha5
On 04/27/2011 02:40 PM, Peter Eisentraut wrote: On tis, 2011-04-26 at 11:36 -0300, Alvaro Herrera wrote: This is a trivial patch (attached). The only problem with it is that make all would fail if lynx is not installed. Well, that will probably not be acceptable. I rather think the action item here is to set up a buildfarm-ish job that runs make distcheck, which is designed to, well, check if the dist-making works. I have just added some infrastructure support to the buildfarm for adding optional steps. This one would be *really* easy to add. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] PostgreSQL Core Team
I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] alpha5
Excerpts from Peter Eisentraut's message of mié abr 27 15:40:22 -0300 2011: On tis, 2011-04-26 at 11:36 -0300, Alvaro Herrera wrote: This is a trivial patch (attached). The only problem with it is that make all would fail if lynx is not installed. Well, that will probably not be acceptable. Agreed. I rather think the action item here is to set up a buildfarm-ish job that runs make distcheck, which is designed to, well, check if the dist-making works. Makes sense. (I wonder why don't we also install the HTML version of those files?) -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] PostgreSQL Core Team
On 27/04/2011 19:48, Dave Page wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Congratulations Magnus! So have you been anointed or afflicted? :-) Ray. -- Raymond O'Donnell :: Galway :: Ireland r...@iol.ie -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
Kudos! Cheers, David. On Wed, Apr 27, 2011 at 07:48:48PM +0100, Dave Page wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XML with invalid chars
On 04/26/2011 05:11 PM, Noah Misch wrote: On Mon, Apr 25, 2011 at 07:25:02PM -0400, Andrew Dunstan wrote: I came across this today, while helping a customer. The following will happily create a piece of XML with an embedded ^A: select xmlelement(name foo, null, E'abc\x01def'); Now, a ^A is totally forbidden in XML version 1.0, and allowed but only as #x01; or equivalent in XML version 1.1, and not as a 0x01 byte (seehttp://en.wikipedia.org/wiki/XML#Valid_characters) ISTM this is something we should definitely try to fix ASAP, even if we probably can't backpatch the fix. +1. Given that such a datum breaks dump+reload, it seems risky to do nothing at all in the back branches. Here's a patch along the lines suggested by Peter. I'm not sure what to do about the back branches and cases where data is already in databases. This is fairly ugly. Suggestions welcome. cheers andrew *** a/src/backend/utils/adt/xml.c --- b/src/backend/utils/adt/xml.c *** *** 1844,1850 escape_xml(const char *str) --- 1844,1865 case '\r': appendStringInfoString(buf, #x0d;); break; + case '\n': + case '\t': + appendStringInfoCharMacro(buf, *p); + break; default: + /* + * Any control char we haven't already explicitly handled + * (i.e. TAB, NL and CR)is an error. + * If we ever support XML 1.1 they will be allowed, + * but will have to be escaped. + */ + if (*p '\x20') + ereport(ERROR, + (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION), + errmsg(character out of range), + errdetail(XML does not support control characters.))); appendStringInfoCharMacro(buf, *p); break; } -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
On 27 April 2011 19:48, Dave Page dp...@postgresql.org wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Excellent! Magnus is a very valuable contributor to the PostgreSQL community and I think the community can only benefit from this addition to the core team. Congrats Magnus! -- Thom Brown Twitter: @darkixion IRC (freenode): dark_ixion Registered Linux user: #516935 EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
Excerpts from Dave Page's message of mié mar 02 16:38:00 -0300 2011: Should be. Moa is definitely Sun Studio: -bash-3.00$ /opt/sunstudio12.1/bin/cc -V cc: Sun C 5.10 SunOS_i386 2009/06/03 usage: cc [ options] files. Use 'cc -flags' for details And Huia is GCC: -bash-3.00$ /usr/sfw/bin/gcc --version gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802) BTW I just swapped the compiler details for those two animals in the buildfarm database. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On Wed, Apr 27, 2011 at 8:12 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Dave Page's message of mié mar 02 16:38:00 -0300 2011: Should be. Moa is definitely Sun Studio: -bash-3.00$ /opt/sunstudio12.1/bin/cc -V cc: Sun C 5.10 SunOS_i386 2009/06/03 usage: cc [ options] files. Use 'cc -flags' for details And Huia is GCC: -bash-3.00$ /usr/sfw/bin/gcc --version gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802) BTW I just swapped the compiler details for those two animals in the buildfarm database. I thought Andrew did that already? -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Typed-tables patch broke pg_upgrade
On Wed, Apr 27, 2011 at 09:30:41PM +0300, Peter Eisentraut wrote: Here is the patch to fix that, as discussed. Looks correct. Thanks. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On 04/27/2011 03:15 PM, Dave Page wrote: On Wed, Apr 27, 2011 at 8:12 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Dave Page's message of mié mar 02 16:38:00 -0300 2011: Should be. Moa is definitely Sun Studio: -bash-3.00$ /opt/sunstudio12.1/bin/cc -V cc: Sun C 5.10 SunOS_i386 2009/06/03 usage: cc [ options] files. Use 'cc -flags' for details And Huia is GCC: -bash-3.00$ /usr/sfw/bin/gcc --version gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802) BTW I just swapped the compiler details for those two animals in the buildfarm database. I thought Andrew did that already? I think I forgot. Thanks Alvaro. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
Excerpts from Dave Page's message of mié abr 27 16:15:33 -0300 2011: On Wed, Apr 27, 2011 at 8:12 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: BTW I just swapped the compiler details for those two animals in the buildfarm database. I thought Andrew did that already? He hadn't gotten around to actually doing it, apparently. Moa was still listed as GCC. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] PostgreSQL Core Team
On Apr 27, 2011, at 14:48, Dave Page wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Congratulations, Magnus! Michael Glaesemann grzm seespotcode net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
On Wed, Apr 27, 2011 at 1:48 PM, Dave Page dp...@postgresql.org wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Congratulations! -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] make world fails
I just did my usual:
make maintainer-clean \
./configure --prefix=/usr/local/pgsql-serializable \
--enable-debug \
--enable-cassert \
--enable-depend \
--with-libxml \
--with-python \
make world
Which ended badly with the attached.
I've been running this pretty much every day on one or two machines,
and this is new.
-Kevin
/usr/bin/perl -p -e 's/\[(amp|copy|egrave|gt|lt|mdash|nbsp|ouml|pi|quot|uuml)
*\]/\\1;/g;' \
-e '$_ .= qq{!DOCTYPE book PUBLIC -//OASIS//DTD DocBook
XML V4.2//EN http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd;\n} if
$. == 1;' \
postgres.xmltmp postgres.xml
rm postgres.xmltmp
xsltproc --stringparam pg.version '9.1devel' stylesheet-man.xsl postgres.xml
error : No such file or directory
warning: failed to load external entity
http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl;
compilation error: file stylesheet-man.xsl line 7 element import
xsl:import : unable to load
http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl
make[3]: *** [man-stamp] Error 5
make[3]: Leaving directory `/home/kgrittn/git/postgresql/kgrittn/doc/src/sgml'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/kgrittn/git/postgresql/kgrittn/doc/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/kgrittn/git/postgresql/kgrittn/doc'
make: *** [world-doc-recurse] Error 2
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] stored procedures - use cases?
On Wed, Apr 27, 2011 at 6:48 PM, Josh Berkus j...@agliodbs.com wrote:
If you pursue your argument a little further, Greg, why do we have
functions at all? We could do it all in the application.
Autonomous transactions have value on their own. But it's not so that
you can run create index ocncurrently or vacuum or whatever.
Why not? Why are you so intent on making my life harder?
Because we want to be able to manipulate data in queries in
data-type-specific ways. For example we want to do aggregations on the
result of a function or index scans across a user data type, etc. If
all the functions do is implement application logic then you end up
having half your application logic in the application and half in the
database and it's hard to keep them in sync.
To take the argument in the opposite extreme would you suggest we
should have html formatting functions in the database so that people
can have their entire web server just be print $dbh-('select
web_page(url)') ?
They're
useful so that a single session can do things like log errors even
when a transaction rolls back.
That's *also* an excellent use case.
What makes it an excellent use case is that it's basically impossible
to do without autonomous transactions. You can hack it with dblink but
it's much less clean and much higher overhead.
--
greg
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock
Composing my rather long-winded response to Simon got me thinking -- which just led me to realize there is probably a need to fix another thing related to SSI. For correct serializable behavior in the face of concurrent DDL execution, I think that a request for a heavyweight ACCESS EXCLUSIVE lock might need to block until all SIREAD locks on the relation have been released. Picture, for example, what might happen if one transaction acquires some predicate locks, then commits (releasing its heavyweight lock on the table), and before concurrent READ WRITE transactions complete there is a CLUSTER on the table. Or a DROP INDEX. :-( Both require an ACCESS EXCLUSIVE lock. Since an active transaction would already have an ACCESS SHARE lock when acquiring the SIREAD locks, this couldn't block in the other direction or with an active transaction. That means that it couldn't cause any deadlocks if we added blocking to the acquisition of an ACCESS EXCLUSIVE based on this. If we don't do this I don't think that there is a more serious impact than inaccurate conflict detection for serializable transactions which are active when these operations are performed. Well, that and the possibility of seeing SIRead locks in the pg_locks view for indexes or tables which no longer exist. So far I don't see any crash modes or effects on non-serializable transactions. If this change is too destabilizing for this point in the release we could document it as a limitation and fix it in 9.2. We're pretty aggressive about cleaning up SIREAD locks as soon as allowable after transaction completion, so this probably wouldn't happen very often. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Fix pg_size_pretty() to avoid overflow for inputs close to INT64
Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Tom Lane's message of lun abr 25 17:22:41 -0300 2011: Fix pg_size_pretty() to avoid overflow for inputs close to INT64_MAX. Apparently this change is causing Moa's SunStudio compiler to fail an assertion. [ scratches head... ] Hard to see why, there's nothing at all interesting in that code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
Kevin Grittner kevin.gritt...@wicourts.gov writes: I just did my usual: make maintainer-clean \ ./configure --prefix=/usr/local/pgsql-serializable \ --enable-debug \ --enable-cassert \ --enable-depend \ --with-libxml \ --with-python \ make world Which ended badly with the attached. Hmm, does it work any better if you revert http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=9412606265c2774712e3f805798896734b32c7fd ? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
On Wed, Apr 27, 2011 at 1:48 PM, Dave Page dp...@postgresql.org wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. yay magnus :-) merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock
On 27.04.2011 22:59, Kevin Grittner wrote: For correct serializable behavior in the face of concurrent DDL execution, I think that a request for a heavyweight ACCESS EXCLUSIVE lock might need to block until all SIREAD locks on the relation have been released. Picture, for example, what might happen if one transaction acquires some predicate locks, then commits (releasing its heavyweight lock on the table), and before concurrent READ WRITE transactions complete there is a CLUSTER on the table. Or a DROP INDEX. :-( Hmm, could we upgrade all predicate locks to relation-level predicate locks instead? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
Excerpts from Kevin Grittner's message of mié abr 27 16:39:01 -0300 2011: I just did my usual: make maintainer-clean \ ./configure --prefix=/usr/local/pgsql-serializable \ --enable-debug \ --enable-cassert \ --enable-depend \ --with-libxml \ --with-python \ make world Which ended badly with the attached. xsltproc --stringparam pg.version '9.1devel' stylesheet-man.xsl postgres.xml error : No such file or directory warning: failed to load external entity http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl; compilation error: file stylesheet-man.xsl line 7 element import xsl:import : unable to load http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl I think somebody mentioned long ago that the new manpage-generating toolchain can sometimes attempt to download XSL documents from that website, if not present in the machine. When you have a working connection and the site is up it works fine, but bombs out as soon as there's a network glitch etc. I think you need to install some Docbook XSL package or other. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: I just did my usual: make maintainer-clean \ ./configure --prefix=/usr/local/pgsql-serializable \ --enable-debug \ --enable-cassert \ --enable-depend \ --with-libxml \ --with-python \ make world Which ended badly with the attached. Hmm, does it work any better if you revert http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=9412606265c2774712e3f805798896734b32c7fd ? It worked with that reverted. I went back to the master branch and it worked there, too, on a retry. Could a transient failure to communicate with the referenced URL on the Internet: http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl have caused this failure? I don't know for sure that there was a failure, but that's what the message seemed to say. I *can* access that page with my browser at the moment. Is the build contingent on Internet access? Should it be? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Fix pg_size_pretty() to avoid overflow for inputs close to INT64
Excerpts from Tom Lane's message of mié abr 27 17:10:37 -0300 2011: Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Tom Lane's message of lun abr 25 17:22:41 -0300 2011: Fix pg_size_pretty() to avoid overflow for inputs close to INT64_MAX. Apparently this change is causing Moa's SunStudio compiler to fail an assertion. [ scratches head... ] Hard to see why, there's nothing at all interesting in that code. I agree, but it fails exactly in that code, and started to fail immediately after that patch. Maybe casting the 2 to int64 would fix it? -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Best way to construct Datum out of a string?
Am 27.04.2011 16:11, schrieb Heikki Linnakangas: On 27.04.2011 17:06, Yves Weißig wrote: Hi, sadly, so far my search in the source code wasn't very successfull on this topic. So, how can I construct a Datum out of a string? What kind of a Datum do you want it to be? What data type? See CStringGetDatum, or perhaps CStringGetTextDatum(). Or perhaps you want to call the input function of some other datatype, with InputFunctionCall. Ok, but how do I do that? Currently I am using: _ebi_mtab_insert(rel, CStringGetTextDatum(BVEC_NULL), bin_enc); This function does not mere than hashing the 2nd passed argument (with the internal hash functions of hash.c) but each time a different hash value is returned, so I am thinking I might pass a pointer and not the real Datum. I am highly irritated now... as for now I thought I understood Datum... -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
On Wed, Apr 27, 2011 at 9:29 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: I just did my usual: make maintainer-clean \ ./configure --prefix=/usr/local/pgsql-serializable \ --enable-debug \ --enable-cassert \ --enable-depend \ --with-libxml \ --with-python \ make world Which ended badly with the attached. Hmm, does it work any better if you revert http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=9412606265c2774712e3f805798896734b32c7fd ? It worked with that reverted. I went back to the master branch and it worked there, too, on a retry. Could a transient failure to communicate with the referenced URL on the Internet: http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl have caused this failure? I don't know for sure that there was a failure, but that's what the message seemed to say. I *can* access that page with my browser at the moment. Is the build contingent on Internet access? Should it be? I periodically see the installer builds fail at this step. It's really annoying. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Introduction
On Tue, Apr 26, 2011 at 11:20 PM, Shiv rama.the...@gmail.com wrote: Dear pgsql-hackers, My name is Sivasankar Ramasubramanian (you can call me Shiv). That's an awesome nickname. My project is aimed towards extending and hopefully improving upon pgtune. If any of you have some ideas or thoughts to share. I am all ears!! That'd be awesome. Do you not have some ideas in mind? It seems like in general it lacks a feedback mechanism to figure things out settings from workloads, instead relying on Greg Smith's sizable experience to do some arithmetic and get you off the ground in a number of common cases. -- fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] new AM, build returns error
Hi, my new AM starts working..., calling something like CREATE INDEX idx ON films USING ebi (did) returns now: ERROR: unexpected EBI relation size: 3, should be 4294967295 The second argument looks for me like max uint32... there might be somehting wrong here I think? Is there somehting wrong with: estimate_rel_size(heap, NULL, relpages, reltuples); The state is passed to the build callback again and again... and result-heap_tuples = reltuples; takes the estimated number of tuples. Where is the error? I know that this is a rather complicated question, due to no source code. Greetz, Yves -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
Excerpts from Alvaro Herrera's message of mié abr 27 17:28:32 -0300 2011: I think you need to install some Docbook XSL package or other. In my system (Debian) I have a catalog.xml file from the docbook-xsl package which has these two lines in it: rewriteURI uriStartString=http://docbook.sourceforge.net/release/xsl/current/; rewritePrefix=.// rewriteSystem systemIdStartString=http://docbook.sourceforge.net/release/xsl/current/; rewritePrefix=.// (where the ./ appears to refer to the /usr/share/xml/docbook/stylesheet/docbook-xsl directory) I take it that if I have a manpages/docbook.xsl in that path, it uses that instead of trying to fetch it from sourceforge. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 27.04.2011 22:59, Kevin Grittner wrote: For correct serializable behavior in the face of concurrent DDL execution, I think that a request for a heavyweight ACCESS EXCLUSIVE lock might need to block until all SIREAD locks on the relation have been released. Picture, for example, what might happen if one transaction acquires some predicate locks, then commits (releasing its heavyweight lock on the table), and before concurrent READ WRITE transactions complete there is a CLUSTER on the table. Or a DROP INDEX. :-( Hmm, could we upgrade all predicate locks to relation-level predicate locks instead? Tied to what backend? This only comes into play with transactions which have committed and are not yet able to release their predicate locks because of overlapping READ WRITE serializable transactions which are still active. Until that point the ACCESS SHARE lock (or stronger) is still there protecting against this. One way we could push this entirely into the heavyweight locking system would be for a committing transaction with predicate locks to somehow cause all overlapping read write serializable transactions which are still active to acquire an ACCESS SHARE lock on each relation for which the committing transaction has any predicate lock(s). (Unless of course they already hold some lock on a relevant relation.) This would need to be done before the committing transaction released its lock on the relation (which it must have acquired to read something which would cause it to have a predicate lock). Is that feasible? It seems like a lot of work, and I'm not sure how one transaction would convince another process's transaction to acquire the locks. As an alternative, perhaps we could find a way to leave the relation locks for a serializable transaction until it's safe to clean up the predicate locks? They could be downgraded to ACCESS SHARE if they are stronger. They would need to survive beyond not only the commit of the transaction, but also the termination of the connection. They would need to be immune to being chosen as deadlock victims. Any other ideas? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
2011/4/27 Dave Page dp...@postgresql.org: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Congratulations Magnus ! Regards, Dave. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Cédric Villemain 2ndQuadrant http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XML with invalid chars
On Wed, Apr 27, 2011 at 03:05:30PM -0400, Andrew Dunstan wrote: On 04/26/2011 05:11 PM, Noah Misch wrote: On Mon, Apr 25, 2011 at 07:25:02PM -0400, Andrew Dunstan wrote: I came across this today, while helping a customer. The following will happily create a piece of XML with an embedded ^A: select xmlelement(name foo, null, E'abc\x01def'); Now, a ^A is totally forbidden in XML version 1.0, and allowed but only as #x01; or equivalent in XML version 1.1, and not as a 0x01 byte (seehttp://en.wikipedia.org/wiki/XML#Valid_characters) ISTM this is something we should definitely try to fix ASAP, even if we probably can't backpatch the fix. +1. Given that such a datum breaks dump+reload, it seems risky to do nothing at all in the back branches. Here's a patch along the lines suggested by Peter. I'm not sure what to do about the back branches and cases where data is already in databases. This is fairly ugly. Suggestions welcome. We could provide a script in (or linked from) the release notes for testing the data in all your xml columns. To make things worse, the dump/reload problems seems to depend on your version of libxml2, or something. With git master, a CentOS 5 system with 2.6.26-2.1.2.8.el5_5.1 accepts the ^A byte, but an Ubuntu 8.04 LTS system with 2.6.31.dfsg-2ubuntu rejects it. Even with a patch like this, systems with a lenient libxml2 will be liable to store XML data that won't restore on a system with a strict libxml2. Perhaps we should emit a build-time warning if the local libxml2 is lenient? *** a/src/backend/utils/adt/xml.c --- b/src/backend/utils/adt/xml.c *** *** 1844,1850 escape_xml(const char *str) --- 1844,1865 case '\r': appendStringInfoString(buf, #x0d;); break; + case '\n': + case '\t': + appendStringInfoCharMacro(buf, *p); + break; default: + /* + * Any control char we haven't already explicitly handled + * (i.e. TAB, NL and CR)is an error. + * If we ever support XML 1.1 they will be allowed, + * but will have to be escaped. + */ + if (*p '\x20') This needs to be an unsigned comparison. On my system, char is signed, so SELECT xmlelement(name foo, null, E'\u0550') fails incorrectly. The XML character set forbids more than just control characters; see http://www.w3.org/TR/xml/#charsets. We also ought to reject, for example, SELECT xmlelement(name foo, null, E'\ufffe'). Injecting the check here aids xmlelement and xmlforest , but xmlcomment and xmlpi still let the invalid byte through. You can also still inject the byte into an attribute value via xmlelement. I wonder if it wouldn't make more sense to just pass any XML that we generate from scratch through libxml2. There are a lot of holes to plug, otherwise. + ereport(ERROR, + (errcode(ERRCODE_INVALID_TEXT_REPRESENTATION), + errmsg(character out of range), + errdetail(XML does not support control characters.))); appendStringInfoCharMacro(buf, *p); break; } -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
On ons, 2011-04-27 at 17:54 -0300, Alvaro Herrera wrote: Excerpts from Alvaro Herrera's message of mié abr 27 17:28:32 -0300 2011: I think you need to install some Docbook XSL package or other. In my system (Debian) I have a catalog.xml file from the docbook-xsl package which has these two lines in it: rewriteURI uriStartString=http://docbook.sourceforge.net/release/xsl/current/; rewritePrefix=.// rewriteSystem systemIdStartString=http://docbook.sourceforge.net/release/xsl/current/; rewritePrefix=.// (where the ./ appears to refer to the /usr/share/xml/docbook/stylesheet/docbook-xsl directory) I take it that if I have a manpages/docbook.xsl in that path, it uses that instead of trying to fetch it from sourceforge. Exactly. If you don't want to depend on net access, you can do something like make whatever XSLTPROCFLAGS=--nonet -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock
On Wed, Apr 27, 2011 at 02:59:19PM -0500, Kevin Grittner wrote: For correct serializable behavior in the face of concurrent DDL execution, I think that a request for a heavyweight ACCESS EXCLUSIVE lock might need to block until all SIREAD locks on the relation have been released. Picture, for example, what might happen if one I think you're correct about this, but also agree that it would be reasonable to document the limitation for now and punt on a fix until 9.2. On Wed, Apr 27, 2011 at 04:09:38PM -0500, Kevin Grittner wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hmm, could we upgrade all predicate locks to relation-level predicate locks instead? Tied to what backend? This only comes into play with transactions which have committed and are not yet able to release their predicate locks because of overlapping READ WRITE serializable transactions which are still active. Until that point the ACCESS SHARE lock (or stronger) is still there protecting against this. I think Heikki was suggesting to upgrade to relation-level SIREAD locks. This seems like it would work, but might cause a lot of aborts immediately afterwards. I do wonder if it might be necessary to upgrade index locks to relation-level locks on the heap relation, in cases of dropping the index. One way we could push this entirely into the heavyweight locking system would be for a committing transaction with predicate locks to somehow cause all overlapping read write serializable transactions which are still active to acquire an ACCESS SHARE lock on each relation for which the committing transaction has any predicate lock(s). I'm not entirely clear on how this would work, but it sounds like it could also have a significant performance cost. As an alternative, perhaps we could find a way to leave the relation locks for a serializable transaction until it's safe to clean up the predicate locks? They could be downgraded to ACCESS SHARE if they are stronger. They would need to survive beyond not only the commit of the transaction, but also the termination of the connection. They would need to be immune to being chosen as deadlock victims. This sounds like it would require major changes to the heavyweight lock manager. There's already a notion of keeping locks past backend termination for 2PC prepared transactions, but it would be hard to apply here. The most practical solutions I see here are to, when acquiring an AccessExclusive lock, either wait for any associated SIREAD locks to go away, or to promote them to relation level. Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade automatic testing
On Wed, Apr 27, 2011 at 09:32:16PM +0300, Peter Eisentraut wrote: On tor, 2011-04-07 at 23:02 +0300, Peter Eisentraut wrote: Seeing that 9.1-to-9.1 pg_upgrade has apparently been broken for months, it would probably be good to have some kind of automatic testing for it. Attached is something I hacked together that at least exposes the current problems, easily available by typing make check and waiting. It does not yet fully implement the full testing procedure in the TESTING file, in particular the diffing of the dumps (well, because you can't get there yet). Is that something that people are interested in refining? (I think it would even be possible under this setup to create special regression test cases that are only run under the pg_upgrade test run, to exercise particularly tricky upgrade cases.) Now that this is bound to be fixed, here is an updated script that runs the entire test procedure including diffing the dumps at the end. Enthusiastic +1 for this concept. There's at least one rough edge: it fails if you have another postmaster running on port 5432. Thanks, nm -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock
Dan Ports d...@csail.mit.edu wrote: On Wed, Apr 27, 2011 at 04:09:38PM -0500, Kevin Grittner wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hmm, could we upgrade all predicate locks to relation-level predicate locks instead? Tied to what backend? I think Heikki was suggesting to upgrade to relation-level SIREAD locks. Oh, yeah. That *is* what he said, isn't it. That's a simpler solution which works just fine. Please forget my over-excited flights of fancy on the topic -- I was so focused on what I thought was the solution I misread Heikki's much better idea as some variation on my theme. :-( -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] stored procedures - use cases?
Greg,
Because we want to be able to manipulate data in queries in
data-type-specific ways. For example we want to do aggregations on the
result of a function or index scans across a user data type, etc.
I don't see how this is different from wanting to capture error output,
which would face the same issues. You seem to be wanting to make a hard
feature easier by telling me that I don't actually want the things I
want. Wanna make it even easier? Then Stored Procedures are just
functions without a return value. That's a 40-line patch. Done!
If
all the functions do is implement application logic then you end up
having half your application logic in the application and half in the
database and it's hard to keep them in sync.
You build your applications your way, and I'll build mine my way. I'll
just ask you not to try to dictate to me how I should build
applications. Especially, since, based on the responses on this thread,
a LOT of people would like to have multitransaction control inside a
stored procedure script. I suspect that your experience of application
development has been rather narrow.
To take the argument in the opposite extreme would you suggest we
should have html formatting functions in the database so that people
can have their entire web server just be print $dbh-('select
web_page(url)') ?
Actually, you can already sort of do that using XSLT. So I don't
necessary think that's a prohibitive idea, depending on implementation.
After all, many of the new non-relational databases implement exactly this.
They're
useful so that a single session can do things like log errors even
when a transaction rolls back.
That's *also* an excellent use case.
What makes it an excellent use case is that it's basically impossible
to do without autonomous transactions. You can hack it with dblink but
it's much less clean and much higher overhead.
You could do it by using application code.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [ANNOUNCE] PostgreSQL Core Team
Congratulation! Regards, Nado Sent from my BlackBerry® powered by Sinyal Kuat INDOSAT -Original Message- From: Dave Page dp...@postgresql.org Sender: pgsql-announce-owner@postgresql.orgDate: Wed, 27 Apr 2011 19:48:48 To: pgsql-announcepgsql-annou...@postgresql.org Cc: PostgreSQL Hackerspgsql-hackers@postgresql.org; pgsql-gene...@postgresql.org Subject: [ANNOUNCE] PostgreSQL Core Team I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/ ---(end of broadcast)--- -To unsubscribe from this list, send an email to: pgsql-announce-unsubscr...@postgresql.org -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] stored procedures - use cases?
On Apr 27, 2011, at 3:28 PM, Josh Berkus wrote: Actually, you can already sort of do that using XSLT. So I don't necessary think that's a prohibitive idea, depending on implementation. After all, many of the new non-relational databases implement exactly this. The proposed JSON data type and construction functions (once there's agreement on an implementation) will allow this, too. Just serve JSON. Boom, instant REST server. David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
Congratulations. Jeremiah Peschka Microsoft SQL Server MVP MCITP: Database Developer, DBA On Apr 27, 2011 11:49 AM, Dave Page dp...@postgresql.org wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL Core Team
Congratulations!! (2011/04/28 3:48), Dave Page wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
On 04/27/2011 11:58 AM, Andrew Dunstan wrote: On 04/27/2011 11:02 AM, Tom Lane wrote: So the question to ask is not why gcc doesn't complain about %m elsewhere, but why it does complain in your Windows installation. I'm guessing that the mingw people hacked it. If you're lucky, they might have hacked in an extra switch to control the behavior --- I notice quite a few subsidiary switches that tweak -Wformat behavior in standard gcc 4.4.5. Hmm. The error disappears if I use -D__USE_MINGW_ANSI_STDIO=1 or -posix I don't know what other effects that might have, though. There's a description here: http://sourceforge.net/project/shownotes.php?release_id=24832 I'll experiment and see what happens. OK, having gone a long way down this hole, I think I have the answer. Using an attribute of 'gnu_printf' instead of just 'printf' on the elog.h functions clears all those warnings. The manual at http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html#Function-Attributes says: The parameter archetype determines how the format string is interpreted, and should be printf, scanf, strftime, gnu_printf, gnu_scanf, gnu_strftime or strfmon. (You can also use __printf__, __scanf__, __strftime__ or __strfmon__.) On MinGW targets, ms_printf, ms_scanf, and ms_strftime are also present. archtype values such as printf refer to the formats accepted by the system's C run-time library, while gnu_ values always refer to the formats accepted by the GNU C Library. I can confirm this works on my W64/Mingw machine. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [ANNOUNCE] PostgreSQL Core Team
Magnus, congratulations! In my short two years around PostgreSQL your name has been synonymous with super intelligence and a great attitude. All the best Sean On Apr 27, 2011, at 1:48 PM, Dave Page dp...@postgresql.org wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/ ---(end of broadcast)--- -To unsubscribe from this list, send an email to: pgsql-announce-unsubscr...@postgresql.org -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [ANNOUNCE] [HACKERS] PostgreSQL Core Team
My Congratulations too ;)!!! Enviado desde mi iPhone El 27/04/2011, a las 18:50, Hiroshi Saito hiro...@winpg.jp escribió: Congratulations!! (2011/04/28 3:48), Dave Page wrote: I'm pleased to announce that effective immediately, Magnus Hagander will be joining the PostgreSQL Core Team. Magnus has been a contributor to PostgreSQL for over 12 years, and played a major part in the development and ongoing maintenance of the native Windows port, quickly becoming a committer to help with his efforts. He's one of the project's webmasters and sysadmins and also contributes to related projects such as pgAdmin. In his spare time, he serves as President of the Board of PostgreSQL Europe. Regards, Dave. ---(end of broadcast)--- -To unsubscribe from this list, send an email to: pgsql-announce-unsubscr...@postgresql.org -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Introduction
Thank you for your compliment (about the name). Its quite sad though that the word Shiv does not exist in the Indian-English lexicon. So people just think of the God and not of a knife! I do have some ideas and heuristics in mind. I wanted to give out a small hello to the community for now. I will write up my ideas/approach properly, go through a round of feedback and then share it with the community of course. (Just that I have my final examinations going on now..so I need to wait for the weekend). I will be trying to talk to Greg as well apart from my mentors. Regards, Shiv On Thu, Apr 28, 2011 at 4:44 AM, Daniel Farina dan...@heroku.com wrote: On Tue, Apr 26, 2011 at 11:20 PM, Shiv rama.the...@gmail.com wrote: Dear pgsql-hackers, My name is Sivasankar Ramasubramanian (you can call me Shiv). That's an awesome nickname. My project is aimed towards extending and hopefully improving upon pgtune. If any of you have some ideas or thoughts to share. I am all ears!! That'd be awesome. Do you not have some ideas in mind? It seems like in general it lacks a feedback mechanism to figure things out settings from workloads, instead relying on Greg Smith's sizable experience to do some arithmetic and get you off the ground in a number of common cases. -- fdr
Re: [HACKERS] improvements to pgtune
Shiv wrote: On the program I hope to learn as much about professional software engineering principles as PostgreSQL. My project is aimed towards extending and hopefully improving upon pgtune. If any of you have some ideas or thoughts to share. I am all ears!! Well, first step on the software engineering side is to get a copy of the code in a form you can modify. I'd recommend grabbing it from https://github.com/gregs1104/pgtune ; while there is a copy of the program on git.postgresql.org, it's easier to work with the one on github instead. I can push updates over to the copy on postgresql.org easily enough, and that way you don't have to worry about getting an account on that server. There's a long list of suggested improvements to make at https://github.com/gregs1104/pgtune/blob/master/TODO Where I would recommend getting started is doing some of the small items on there, some of which I have already put comments into the code about but just not finished yet. Some examples: -Validate against min/max -Show original value in output -Limit shared memory use on Windows (see notes on shared_buffers at http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server for more information) -Look for postgresql.conf file using PGDATA environment variable -Look for settings files based on path of the pgtune executable -Save a settings reference files for newer versions of PostgreSQL (right now I only target 8.4) and allow passing in the version you're configuring. A common mistake made by GSOC students is to dive right in to trying to make big changes. You'll be more successful if you get practice at things like preparing and sharing patches on smaller changes first. At the next level, there are a few larger features that I would consider valuable that are not really addressed by the program yet: -Estimate how much shared memory is used by the combination of settings. See Table 17-2 at http://www.postgresql.org/docs/9.0/static/kernel-resources.html ; those numbers aren't perfect, and improving that table is its own useful project. But it gives an idea how they fit together. I have some notes at the end of the TODO file on how I think the information needed to produce this needs to be passed around the inside of pgtune. -Use that estimate to produce a sysctl.conf file for one platform; Linux is the easiest one to start with. I've attached a prototype showing how to do that, written in bash. -Write a Python-TK or web-based front-end for the program. Now that I know someone is going to work on this program again, I'll see what I can do to clean some parts of it up. There are a couple of things it's easier for me to just fix rather than to describe, like the way I really want to change how it adds comments to the settings it changes. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us #!/bin/bash # Output lines suitable for sysctl configuration based # on total amount of RAM on the system. The output # will allow up to 50% of physical memory to be allocated # into shared memory. # On Linux, you can use it as follows (as root): # # ./shmsetup /etc/sysctl.conf # sysctl -p # Early FreeBSD versions do not support the sysconf interface # used here. The exact version where this works hasn't # been confirmed yet. page_size=`getconf PAGE_SIZE` phys_pages=`getconf _PHYS_PAGES` if [ -z $page_size ]; then echo Error: cannot determine page size exit 1 fi if [ -z $phys_pages ]; then echo Error: cannot determine number of memory pages exit 2 fi shmall=`expr $phys_pages / 2` shmmax=`expr $shmall \* $page_size` echo \# Maximum shared segment size in bytes echo kernel.shmmax = $shmmax echo \# Maximum number of shared memory segments in pages echo kernel.shmall = $shmall -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] improvements to pgtune
Daniel Farina wrote: It seems like in general it lacks a feedback mechanism to figure things out settings from workloads, instead relying on Greg Smith's sizable experience to do some arithmetic and get you off the ground in a number of common cases. To credit appropriately, the model used right now actually originated with a Josh Berkus spreadsheet, from before I was doing this sort of work full-time. That's held up pretty well, but it doesn't fully reflect how I do things nowadays. The recent realization that pgtune is actually shipping as a package for Debian/Ubuntu now has made realize this is a much higher profile project now, one that I should revisit doing a better job on. Every time I've gotten pulled into discussions of setting parameters based on live monitoring, it's turned into a giant black hole--absorbs a lot of energy, nothing useful escapes from it. I credit completely ignoring that idea altogether, and using the simplest possible static settings instead, as one reason I managed to ship code here that people find useful. I'm not closed to the idea, just not optimistic it will lead anywhere useful. That makes it hard to work on when there are so many obvious things guaranteed to improve the program that could be done instead. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Predicate locking
27.04.2011 18:38, Heikki Linnakangas пишет: On 27.04.2011 12:24, Vlad Arkhipov wrote: 27.04.2011 17:45, Nicolas Barbier: 2011/4/27 Vlad Arkhipovarhi...@dc.baikal.ru: I'm currently need predicate locking in the project, so there are two ways to get it by now: implement it by creating special database records to lock with SELECT FOR UPDATE or wait while they will be implemented in Postgres core. Is there something like predicate locking on the TODO list currently? I assume you want (real, as opposed to what is in 9.1 now) SERIALIZABLE transactions, in which case you could check: URL:http://wiki.postgresql.org/wiki/Serializable Nicolas Not sure about the whole transaction, I think it degrades the performance too much as transactions access many tables. Just wanted SELECT FOR UPDATE to prevent inserting records into a table with the specified condition. It seems to be very typical situation when you have a table like CREATE TABLE timetable (start_ts TIMESTAMP, end_ts TIMESTAMP) and before insertion in this table want to guarantee that there is no overlapped time intervals there. So, first you need to lock the range in the table, then to check if there are any records in this range. In my case this table is the only for which I need such kind of locking. You can do that with exclusion constraints: http://www.postgresql.org/docs/9.0/static/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSION) See also Depesz's blog post for a specific example on how to use it for time ranges: http://www.depesz.com/index.php/2010/01/03/waiting-for-8-5-exclusion-constraints/ And Jeff Davis's blog post that uses the period data type instead of the hack to represent time ranges as boxes: http://thoughts.j-davis.com/2009/11/08/temporal-keys-part-2/ Exclusion constraints works only in simple cases. I need to check a great amount of business rules to assure that the insertion is possible. For example, for a table with columns (start_ts TIMESTAMP, end_ts TIMESTAMP, room BIGINT, visitor BIGINT, service BIGINT) it's not possible to have overlapped intervals for the same time and room, but different visitors. So, in terms of exclusion constraints I need something like: room WITH =, visitor WITH , (start_ts, end_ts) WITH which seems to be impossible. Predicate locking provides more flexible way to solve this problem. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XML with invalid chars
On 04/27/2011 05:30 PM, Noah Misch wrote: I'm not sure what to do about the back branches and cases where data is already in databases. This is fairly ugly. Suggestions welcome. We could provide a script in (or linked from) the release notes for testing the data in all your xml columns. Yeah, we'll have to do something like that. What a blasted mess, To make things worse, the dump/reload problems seems to depend on your version of libxml2, or something. With git master, a CentOS 5 system with 2.6.26-2.1.2.8.el5_5.1 accepts the ^A byte, but an Ubuntu 8.04 LTS system with 2.6.31.dfsg-2ubuntu rejects it. Even with a patch like this, systems with a lenient libxml2 will be liable to store XML data that won't restore on a system with a strict libxml2. Perhaps we should emit a build-time warning if the local libxml2 is lenient? No, I think we need to be strict ourselves. + if (*p '\x20') This needs to be an unsigned comparison. On my system, char is signed, so SELECT xmlelement(name foo, null, E'\u0550') fails incorrectly. Good point. Perhaps we'd be better off using iscntrl(*p). The XML character set forbids more than just control characters; see http://www.w3.org/TR/xml/#charsets. We also ought to reject, for example, SELECT xmlelement(name foo, null, E'\ufffe'). Injecting the check here aids xmlelement and xmlforest , but xmlcomment and xmlpi still let the invalid byte through. You can also still inject the byte into an attribute value via xmlelement. I wonder if it wouldn't make more sense to just pass any XML that we generate from scratch through libxml2. There are a lot of holes to plug, otherwise. Maybe there are, but I'd want lots of convincing that we should do that at this stage. Maybe for 9.2. I think we can plug the holes fairly simply for xmlpi and xmlcomment, and catch the attributes by moving this check up into map_sql_value_to_xml_value(). This is a significant data integrity bug, much along the same lines as the invalidly encoded data holes we plugged a release or two back. I'm amazed we haven't hit it till now, but we're sure to see more of it - XML use with Postgres is growing substantially, I believe. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XML with invalid chars
On Wed, Apr 27, 2011 at 11:22:37PM -0400, Andrew Dunstan wrote: On 04/27/2011 05:30 PM, Noah Misch wrote: To make things worse, the dump/reload problems seems to depend on your version of libxml2, or something. With git master, a CentOS 5 system with 2.6.26-2.1.2.8.el5_5.1 accepts the ^A byte, but an Ubuntu 8.04 LTS system with 2.6.31.dfsg-2ubuntu rejects it. Even with a patch like this, systems with a lenient libxml2 will be liable to store XML data that won't restore on a system with a strict libxml2. Perhaps we should emit a build-time warning if the local libxml2 is lenient? No, I think we need to be strict ourselves. Then I suppose we'd also scan for invalid characters in xml_parse()? Or, at least, do so when linked to a libxml2 that neglects to do so itself? Injecting the check here aids xmlelement and xmlforest , but xmlcomment and xmlpi still let the invalid byte through. You can also still inject the byte into an attribute value via xmlelement. I wonder if it wouldn't make more sense to just pass any XML that we generate from scratch through libxml2. There are a lot of holes to plug, otherwise. Maybe there are, but I'd want lots of convincing that we should do that at this stage. Maybe for 9.2. I think we can plug the holes fairly simply for xmlpi and xmlcomment, and catch the attributes by moving this check up into map_sql_value_to_xml_value(). I don't have much convincing to offer -- hunting down the holes seem fine, too. Thanks, nm -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Best way to construct Datum out of a string?
=?ISO-8859-15?Q?Yves_Wei=DFig?= weis...@rbg.informatik.tu-darmstadt.de writes: Am 27.04.2011 16:11, schrieb Heikki Linnakangas: What kind of a Datum do you want it to be? What data type? See CStringGetDatum, or perhaps CStringGetTextDatum(). Or perhaps you want to call the input function of some other datatype, with InputFunctionCall. Ok, but how do I do that? Currently I am using: _ebi_mtab_insert(rel, CStringGetTextDatum(BVEC_NULL), bin_enc); This function does not mere than hashing the 2nd passed argument (with the internal hash functions of hash.c) but each time a different hash value is returned, so I am thinking I might pass a pointer and not the real Datum. I am highly irritated now... as for now I thought I understood Datum... Well, it's hard to say for sure when you haven't shown us either what BVEC_NULL means or what _ebi_mtab_insert is doing with the value it gets ... but in fact a text Datum *is* a pointer, as is the Datum value for any other pass-by-reference type. Datum isn't magic, it's only a pointer-sized integer type. For anything bigger than that, the Datum value is a pointer to some data somewhere else. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make world fails
Peter Eisentraut pete...@gmx.net writes: On ons, 2011-04-27 at 17:54 -0300, Alvaro Herrera wrote: I take it that if I have a manpages/docbook.xsl in that path, it uses that instead of trying to fetch it from sourceforge. Exactly. If you don't want to depend on net access, you can do something like make whatever XSLTPROCFLAGS=--nonet Is there a way to say fetch all the documents I need for this build into my local cache? Then you could do that when your network was up, and not have to worry about failures in future. The set of URIs we reference doesn't change much. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
Andrew Dunstan and...@dunslane.net writes: OK, having gone a long way down this hole, I think I have the answer. Using an attribute of 'gnu_printf' instead of just 'printf' on the elog.h functions clears all those warnings. The manual at http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html#Function-Attributes says: The parameter archetype determines how the format string is interpreted, and should be printf, scanf, strftime, gnu_printf, gnu_scanf, gnu_strftime or strfmon. (You can also use __printf__, __scanf__, __strftime__ or __strfmon__.) On MinGW targets, ms_printf, ms_scanf, and ms_strftime are also present. archtype values such as printf refer to the formats accepted by the system's C run-time library, while gnu_ values always refer to the formats accepted by the GNU C Library. Hmm, interesting. I see gnu_printf also documented in the gcc 4.4.5 manual on my Fedora 13 machine (so it's not something specific to mingw) but it's not present on my ancient 2.95.3 gcc, so I'm not too sure when it was introduced. I'd suggest adjusting the elog.h declarations to use gnu_printf only on Windows, and printf elsewhere, for the moment. Maybe we can migrate towards using gnu_printf on other platforms later. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] improvements to pgtune
Every time I've gotten pulled into discussions of setting parameters based on live monitoring, it's turned into a giant black hole--absorbs a lot of energy, nothing useful escapes from it. I credit completely ignoring that idea altogether, and using the simplest possible static settings instead, as one reason I managed to ship code here that people find useful. I'm not closed to the idea, just not optimistic it will lead anywhere useful. That makes it hard to work on when there are so many obvious things guaranteed to improve the program that could be done instead. What would you list as the main things pgtune doesn't cover right now? I have my own list, but I suspect that yours is somewhat different. I do think that autotuning based on interrogating the database is possible. However, I think the way to make it not be a tar baby is to tackle it one setting at a time, and start with ones we have the most information for. One of the real challenges there is that some data can be gleaned from pg_* views, but a *lot* of useful performance data only shows up in the activity log, and then only if certain settings are enabled. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com San Francisco -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal - asynchronous functions
It sounds like there is interest in this feature, can it get added to the TODO list? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
On 04/28/2011 12:15 AM, Tom Lane wrote: I'd suggest adjusting the elog.h declarations to use gnu_printf only on Windows, and printf elsewhere, for the moment. Maybe we can migrate towards using gnu_printf on other platforms later. Yeah. In fact, if I adjust most of the format specs to gnu_printf I now get exactly five warnings on MinGW64 (down from about 600). All the 64 bit int format warnings and their cascaded effects dissolve, along with the %m warnings. What I'm thinking of doing is to set up something like: #define PG_PRINTF_CHECK __printf__ and on Windows redefine it to __gnu_printf__, and then set all the formats to use PG_PRINTF_CHECK. Sound OK? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
Andrew Dunstan and...@dunslane.net writes: What I'm thinking of doing is to set up something like: #define PG_PRINTF_CHECK __printf__ and on Windows redefine it to __gnu_printf__, and then set all the formats to use PG_PRINTF_CHECK. Sound OK? +1 ... those __attribute__ declarations are messy enough already without wrapping #ifdefs around them. (Don't want to find out what pgindent would do with that ...) Possibly PG_PRINTF_ATTRIBUTE would be a better name, but he who does the work gets to pick. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unknown conversion %m
Andrew Dunstan and...@dunslane.net writes: What I'm thinking of doing is to set up something like: #define PG_PRINTF_CHECK __printf__ BTW, gcc 2.95.3 documents printf, and not __printf__. Suggest not including the underscores, since that's apparently a johnny-come-lately spelling. It's not like any of this construct is even faintly portable to non-gcc compilers anyway ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
