Re: [HACKERS] InitPostgres and flatfiles question
Hi, Tom Lane wrote: It should happen automatically at commit of the CREATE DATABASE ... and you'd not be able to see the pg_database row before that anyway. So I'm not clear on what you're worried about. I've just found the stumbling block: the -c option of psql wraps all in a transaction, as man psql says: If the command string contains multiple SQL commands, they are processed in a single transaction, unless there are explicit BEGIN/COMMIT commands included in the string to divide it into multiple transactions. This is different from the behavior when the same string is fed to psql’s standard input. Thank you for clarification, I wouldn't have expected that (especially because CREATE DATABASE itself says, it cannot be run inside a transaction block... A transaction block (with BEGIN and COMMIT) seems to be more than just a transaction, right?) Regards Markus ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] 8.3 pending patch queue
On Mon, 2007-01-01 at 19:56 -0500, Bruce Momjian wrote: Andrew Dunstan wrote: Bruce Momjian wrote: I will start processing the patches held for 8.3 this week or next, now that the holiday break is over: http://momjian.postgresql.org/cgi-bin/pgpatches_hold Some of these look obsolete. Also, . the plperl out params patch needs substantial rework by its author, IMHO. . there is a new version of the enums patch that has been submitted. Yes, I will have to go through it, find the valuable ones, and get comments. Sounds good. I'm not clear about the difference between the unapplied patches list and the hold list. What is the significance of the two lists? There's a number of patches submitted to pgsql-patches that don't show up on either list. I haven't made a list of these, but they include major patches such as Grouped Item indexes and a number of others. Many of those are clearly marked as ready to apply/review/reject. Can I request that those be reviewed first? The unapplied patches list looks long and many things on it aren't even patches, AFAICS - presumably TODO items-in-waiting? Some minor points: [PATCHES] Incrementally Updated Backup, Simon Riggs has already been applied to 8.2 [PATCHES] WAL logging freezing, Heikki Linnakangas has already been agreed/applied to 8.2 -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Request for review: tsearch2 patch
I have tested with local-enabled environment and found a bug. Included is the new version of patches. Teodor, Oleg, what do you think about these patches? If ok, shall I commit to CVS head? -- Tatsuo Ishii SRA OSS, Inc. Japan Hi, Here are patches against tsearch2 with CVS head. Currently tsearch2 does not work with multibyte encoding which uses C locale. These patches are intended to solve the problem by using PostgreSQL in-house multibyte function instead of mbstowcs which does not work with C locale. Also iswalpha etc. will not be called in case of C locale since they are not working with it. Tested with the EUC_JP encoding (should be working with any multibye encodings). Existing single byte encodings should not be broken by the patches, I did not test though. -- Tatsuo Ishii SRA OSS, Inc. Japan Index: ts_locale.c === RCS file: /cvsroot/pgsql/contrib/tsearch2/ts_locale.c,v retrieving revision 1.7 diff -c -r1.7 ts_locale.c *** ts_locale.c 20 Nov 2006 14:03:30 - 1.7 --- ts_locale.c 4 Jan 2007 12:16:00 - *** *** 63,68 --- 63,101 return mbstowcs(to, from, len); } + + #else /* WIN32 */ + + size_t + char2wchar(wchar_t *to, const char *from, size_t len) + { + wchar_t *result; + size_t n; + + if (to == NULL) + return 0; + + if (lc_ctype_is_c()) + { + /* allocate neccesary memory for to including NULL terminate */ + result = (wchar_t *)palloc((len+1)*sizeof(wchar_t)); + + /* do the conversion */ + n = (size_t)pg_mb2wchar_with_len(from, (pg_wchar *)result, len); + if (n 0) + { + /* store the result */ + if (n len) + n = len; + memcpy(to, result, n*sizeof(wchar_t)); + pfree(result); + *(to + n) = '\0'; + } + return n; + } + return mbstowcs(to, from, len); + } + #endif /* WIN32 */ int *** *** 70,75 --- 103,113 { wchar_t character; + if (lc_ctype_is_c()) + { + return isalpha(TOUCHAR(ptr)); + } + char2wchar(character, ptr, 1); return iswalpha((wint_t) character); *** *** 80,85 --- 118,128 { wchar_t character; + if (lc_ctype_is_c()) + { + return isprint(TOUCHAR(ptr)); + } + char2wchar(character, ptr, 1); return iswprint((wint_t) character); *** *** 126,132 if ( wlen 0 ) ereport(ERROR, (errcode(ERRCODE_CHARACTER_NOT_IN_REPERTOIRE), !errmsg(transalation failed from server encoding to wchar_t))); Assert(wlen=len); wstr[wlen] = 0; --- 169,175 if ( wlen 0 ) ereport(ERROR, (errcode(ERRCODE_CHARACTER_NOT_IN_REPERTOIRE), !errmsg(translation failed from server encoding to wchar_t))); Assert(wlen=len); wstr[wlen] = 0; *** *** 152,158 if ( wlen 0 ) ereport(ERROR, (errcode(ERRCODE_CHARACTER_NOT_IN_REPERTOIRE), !errmsg(transalation failed from wchar_t to server encoding %d, errno))); Assert(wlen=len); out[wlen]='\0'; } --- 195,201 if ( wlen 0 ) ereport(ERROR, (errcode(ERRCODE_CHARACTER_NOT_IN_REPERTOIRE), !errmsg(translation failed from wchar_t to server encoding %d, errno))); Assert(wlen=len); out[wlen]='\0'; } Index: ts_locale.h === RCS file: /cvsroot/pgsql/contrib/tsearch2/ts_locale.h,v retrieving revision 1.7 diff -c -r1.7 ts_locale.h *** ts_locale.h 4 Oct 2006 00:29:47 - 1.7 --- ts_locale.h 4 Jan 2007 12:16:00 - *** *** 38,45 #else /* WIN32 */ /* correct mbstowcs */ - #define char2wchar mbstowcs #define wchar2char wcstombs #endif /* WIN32 */ #define t_isdigit(x) ( pg_mblen(x)==1 isdigit( TOUCHAR(x) ) ) --- 38,46 #else /* WIN32 */ /* correct mbstowcs */ #define wchar2char wcstombs + size_tchar2wchar(wchar_t *to, const char *from, size_t len); + #endif /* WIN32 */ #define
[HACKERS] Tabs or Spaces
Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. Trouble is, there are tabs throughout the code. What gives? Should I be replacing tabs as spaces, or leaving them alone? At all? Only in patches I submit? Everywhere? I have no opinion either way as to what should be done. Please advise. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 8.3 pending patch queue
Simon Riggs wrote: I'm not clear about the difference between the unapplied patches list and the hold list. What is the significance of the two lists? AIUI, the hold list is those patches providing new features that were held over between 8.2 feature freeze and 8.2 branch. Since they have been around for a while I think they have some claim to priority. The other list is just the normal running list of such patches that Bruce keeps. There's a number of patches submitted to pgsql-patches that don't show up on either list. That also happens. The only way I can see of ensuring it does not happen would be to auto-process all patch submissions. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 8.3 pending patch queue
On Thu, 2007-01-04 at 09:09 -0500, Andrew Dunstan wrote: Simon Riggs wrote: I'm not clear about the difference between the unapplied patches list and the hold list. What is the significance of the two lists? AIUI, the hold list is those patches providing new features that were held over between 8.2 feature freeze and 8.2 branch. Since they have been around for a while I think they have some claim to priority. The other list is just the normal running list of such patches that Bruce keeps. OK. Makes sense, thanks. There's a number of patches submitted to pgsql-patches that don't show up on either list. Hopefully the priority applies to all things that should be on the list. That also happens. The only way I can see of ensuring it does not happen would be to auto-process all patch submissions. Sounds a good idea. Patch farm anyone? Auto apply/make check? -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [PATCHES] Patch to log usage of temporary files
Bill Moran [EMAIL PROTECTED] writes: Andrew Dunstan [EMAIL PROTECTED] wrote: Might be more robust to say if (trace_temp_files = 0) I specified in the GUC config that minimum allowable value is -1. I'd still tend to go with Andrew's suggestion because it makes this particular bit of code self-defending against bad values. Yes, it's reasonably safe given that bit of coding way over yonder in guc.c, but there's no particularly good reason why this code has to depend on that to avoid doing something stupid. And it's easier to understand too --- you don't have to go looking in guc.c to convince yourself it's safe. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
Simon Riggs [EMAIL PROTECTED] writes: In this thread, I outlined an idea for reducing cost of WAL CRC checking http://archives.postgresql.org/pgsql-hackers/2006-10/msg01299.php wal_checksum = on (default) | off This still seems awfully dangerous to me. Recovery can occur with/without same setting of wal_checksum, to avoid complications from crashes immediately after turning GUC on. Surely not. Otherwise even the on setting is not really a defense. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Tabs or Spaces
Simon Riggs [EMAIL PROTECTED] writes: Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. No, it does the opposite (or I would hope so anyway). regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] InitPostgres and flatfiles question
Markus Schiltknecht [EMAIL PROTECTED] writes: I've just found the stumbling block: the -c option of psql wraps all in a transaction, as man psql says: ... Thank you for clarification, I wouldn't have expected that (especially because CREATE DATABASE itself says, it cannot be run inside a transaction block... A transaction block (with BEGIN and COMMIT) seems to be more than just a transaction, right?) Hm, that's an interesting point. psql's -c just shoves its whole argument string at the backend in one PQexec(), instead of dividing at semicolons as psql does with normal input. And so it winds up as a single transaction because postgres.c doesn't force a transaction commit until the end of the querystring. But that's not a transaction block in the normal sense and so it doesn't trigger the PreventTransactionChain defense in CREATE DATABASE and elsewhere. I wonder whether we ought to change that? The point of PreventTransactionChain is that we don't want the user rolling back the statement post-completion, but it seems that psql -c 'CREATE DATABASE foo; ABORT; BEGIN; ...' would bypass the check. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] Odd numeric-float4/8 casting behaviour
I noticed this odd discrepancy: postgres=# select -0.999::numeric(3,3)::float4 = -0.999::numeric(3,3); ?column? -- f (1 row) I believe this is happening because the numeric is being cast to float8 and then the float4-float8 cross-data-type operator is being used. It seems like it would be preferable to cast it to float4 and use the non-cross-data-type operator. They're both marked as implicit casts so I'm unclear what decides which gets used. Also, as a side note I was surprised to find the above being parsed as -(0.999::numeric(3,3)) rather than (-0.999)::numeric(3,3) is that expected? regression=# create or replace view x as select -0.999::numeric(3,3)::float4 = -0.999::numeric(3,3); CREATE VIEW regression=# \d x View public.x Column | Type | Modifiers --+-+--- ?column? | boolean | View definition: SELECT (- 0.999::numeric(3,3)::real) = (- 0.999::numeric(3,3))::double precision; regression=# create or replace view x as select (-0.999)::numeric(3,3)::float4 = (-0.999)::numeric(3,3); CREATE VIEW regression=# \d x View public.x Column | Type | Modifiers --+-+--- ?column? | boolean | View definition: SELECT -0.999::numeric(3,3)::real = -0.999::numeric(3,3)::double precision; -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] InitPostgres and flatfiles question
Am Donnerstag, 4. Januar 2007 16:36 schrieb Tom Lane: Markus Schiltknecht [EMAIL PROTECTED] writes: Hm, that's an interesting point. psql's -c just shoves its whole argument string at the backend in one PQexec(), instead of dividing at semicolons as psql does with normal input. And so it winds up as a single transaction because postgres.c doesn't force a transaction commit until the end of the querystring. But that's not a transaction block in the normal sense and so it doesn't trigger the PreventTransactionChain defense in CREATE DATABASE and elsewhere. I wonder whether we ought to change that? The point of PreventTransactionChain is that we don't want the user rolling back the statement post-completion, but it seems that psql -c 'CREATE DATABASE foo; ABORT; BEGIN; ...' would bypass the check. Maybe not directly related to that problem, but I had a problem with -c last month, when I noticed that this will not work: psql -c set client_encoding=iso-8859-1; select name from customer (UTF8 database, output is hmmm... broken german umlauts). Best regards Mario Weilguni ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Thu, 2007-01-04 at 10:00 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: In this thread, I outlined an idea for reducing cost of WAL CRC checking http://archives.postgresql.org/pgsql-hackers/2006-10/msg01299.php wal_checksum = on (default) | off This still seems awfully dangerous to me. Understood. Recovery can occur with/without same setting of wal_checksum, to avoid complications from crashes immediately after turning GUC on. Surely not. Otherwise even the on setting is not really a defense. Only when the CRC is exactly zero, which happens very very rarely. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2007-01-04 at 10:00 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Recovery can occur with/without same setting of wal_checksum, to avoid complications from crashes immediately after turning GUC on. Surely not. Otherwise even the on setting is not really a defense. Only when the CRC is exactly zero, which happens very very rarely. It works most of the time doesn't exactly satisfy me. What's the use-case for changing the variable on the fly anyway? Seems a better solution is just to lock down the setting at postmaster start. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Tabs or Spaces
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. No, it does the opposite (or I would hope so anyway). Sure looks that way in the code. In fact it detabifies and then retabifies, from what I can see. cheers andrew ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Odd numeric-float4/8 casting behaviour
Gregory Stark [EMAIL PROTECTED] writes: I believe this is happening because the numeric is being cast to float8 and then the float4-float8 cross-data-type operator is being used. It seems like it would be preferable to cast it to float4 and use the non-cross-data-type operator. They're both marked as implicit casts so I'm unclear what decides which gets used. Without having traced through the code, I think the fact that float8 is a preferred type is driving it. It's not clear whether we could change this without getting into a can't resolve ambiguous operator problem. Also, as a side note I was surprised to find the above being parsed as -(0.999::numeric(3,3)) rather than (-0.999)::numeric(3,3) is that expected? Yeah, :: binds VERY tightly. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Odd numeric-float4/8 casting behaviour
Tom Lane [EMAIL PROTECTED] writes: Gregory Stark [EMAIL PROTECTED] writes: I believe this is happening because the numeric is being cast to float8 and then the float4-float8 cross-data-type operator is being used. It seems like it would be preferable to cast it to float4 and use the non-cross-data-type operator. They're both marked as implicit casts so I'm unclear what decides which gets used. Without having traced through the code, I think the fact that float8 is a preferred type is driving it. It's not clear whether we could change this without getting into a can't resolve ambiguous operator problem. This is pre-operator-families, I thought preferred type was new with them. Perhaps we should have preferred operators rather than preferred types? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] InitPostgres and flatfiles question
Hi, Tom Lane wrote: Hm, that's an interesting point. psql's -c just shoves its whole argument string at the backend in one PQexec(), instead of dividing at semicolons as psql does with normal input. And so it winds up as a single transaction because postgres.c doesn't force a transaction commit until the end of the querystring. But that's not a transaction block in the normal sense and so it doesn't trigger the PreventTransactionChain defense in CREATE DATABASE and elsewhere. Is there a good reason to not let psql -c behave exactly like psql from STDIN? I found this exception to be quite confusing. Of course that could break compatibility with certain scripts, but can this be fixed without doing so? Regards Markus ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [PATCHES] Patch to log usage of temporary files
In response to Tom Lane [EMAIL PROTECTED]: Bill Moran [EMAIL PROTECTED] writes: Andrew Dunstan [EMAIL PROTECTED] wrote: Might be more robust to say if (trace_temp_files = 0) I specified in the GUC config that minimum allowable value is -1. I'd still tend to go with Andrew's suggestion because it makes this particular bit of code self-defending against bad values. Yes, it's reasonably safe given that bit of coding way over yonder in guc.c, but there's no particularly good reason why this code has to depend on that to avoid doing something stupid. And it's easier to understand too --- you don't have to go looking in guc.c to convince yourself it's safe. Ahh ... well, I've probably already argued about it more than it's worth. The patch is easy enough to adjust, find attached. -- Bill Moran Collaborative Fusion Inc. diff -c -r src.orig/backend/storage/file/fd.c src/backend/storage/file/fd.c *** src.orig/backend/storage/file/fd.c Thu Dec 7 15:44:42 2006 --- src/backend/storage/file/fd.c Wed Jan 3 15:05:54 2007 *** *** 50,55 --- 50,56 #include access/xact.h #include storage/fd.h #include storage/ipc.h + #include utils/guc.h /* *** *** 938,944 void FileClose(File file) { ! Vfd *vfdP; Assert(FileIsValid(file)); --- 939,946 void FileClose(File file) { ! Vfd *vfdP; ! struct stat filestats; Assert(FileIsValid(file)); *** *** 968,973 --- 970,992 { /* reset flag so that die() interrupt won't cause problems */ vfdP-fdstate = ~FD_TEMPORARY; + PG_TRACE1(temp__file__cleanup, vfdP-fileName); + if (trace_temp_files = 0) + { + if (stat(vfdP-fileName, filestats) == 0) + { + if (filestats.st_size = trace_temp_files) + { + ereport(LOG, + (errmsg(temp file: size %lu path \%s\, + filestats.st_size, vfdP-fileName))); + } + } + else + { + elog(LOG, Could not stat \%s\: %m, vfdP-fileName); + } + } if (unlink(vfdP-fileName)) elog(LOG, failed to unlink \%s\: %m, vfdP-fileName); diff -c -r src.orig/backend/utils/misc/guc.c src/backend/utils/misc/guc.c *** src.orig/backend/utils/misc/guc.c Wed Nov 29 09:50:07 2006 --- src/backend/utils/misc/guc.c Wed Jan 3 13:51:14 2007 *** *** 180,186 int log_min_messages = NOTICE; int client_min_messages = NOTICE; int log_min_duration_statement = -1; ! int num_temp_buffers = 1000; char *ConfigFileName; --- 180,187 int log_min_messages = NOTICE; int client_min_messages = NOTICE; int log_min_duration_statement = -1; ! int trace_temp_files = -1; ! int num_temp_buffers = 1000; char *ConfigFileName; *** *** 1471,1477 log_min_duration_statement, -1, -1, INT_MAX / 1000, NULL, NULL }, ! { {bgwriter_delay, PGC_SIGHUP, RESOURCES, gettext_noop(Background writer sleep time between rounds in milliseconds), --- 1472,1478 log_min_duration_statement, -1, -1, INT_MAX / 1000, NULL, NULL }, ! { {bgwriter_delay, PGC_SIGHUP, RESOURCES, gettext_noop(Background writer sleep time between rounds in milliseconds), *** *** 1657,1662 --- 1658,1673 }, server_version_num, PG_VERSION_NUM, PG_VERSION_NUM, PG_VERSION_NUM, NULL, NULL + }, + + { + {trace_temp_files, PGC_USERSET, LOGGING_WHAT, + gettext_noop(Log the use of temp files larger than this size.), + gettext_noop(Size and location of each temp file is reported.), + NULL + }, + trace_temp_files, + -1, -1, INT_MAX, NULL, NULL }, /* End-of-list marker */ diff -c -r src.orig/backend/utils/misc/postgresql.conf.sample src/backend/utils/misc/postgresql.conf.sample *** src.orig/backend/utils/misc/postgresql.conf.sample Mon Nov 20 20:23:37 2006 --- src/backend/utils/misc/postgresql.conf.sample Wed Jan 3 11:05:48 2007 *** *** 333,338 --- 333,341 #log_statement = 'none' # none, ddl, mod, all #log_hostname = off + #trace_temp_files = -1 # Log usage of temporary files larger than + # the specified size (in bytes). -1 disables. + # 0 effectively logs all temp file usage. #--- # RUNTIME STATISTICS diff -c -r src.orig/include/utils/guc.h src/include/utils/guc.h *** src.orig/include/utils/guc.h Thu Oct 19 14:32:47 2006 --- src/include/utils/guc.h Wed Jan 3 13:45:52 2007 *** *** 123,128 --- 123,129 extern int log_min_messages; extern int client_min_messages; extern int log_min_duration_statement; + extern int trace_temp_files; extern int num_temp_buffers; ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Odd numeric-float4/8 casting behaviour
Gregory Stark [EMAIL PROTECTED] writes: This is pre-operator-families, I thought preferred type was new with them. No, preferred types have been around for a very long time. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] InitPostgres and flatfiles question
Markus Schiltknecht [EMAIL PROTECTED] writes: Tom Lane wrote: Hm, that's an interesting point. psql's -c just shoves its whole argument string at the backend in one PQexec(), instead of dividing at semicolons as psql does with normal input. Is there a good reason to not let psql -c behave exactly like psql from STDIN? Backwards compatibility, mostly --- there seems to be a considerable risk of subtly breaking people's scripts if we change the transactional boundaries for psql -c commands. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Thu, 2007-01-04 at 11:09 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2007-01-04 at 10:00 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Recovery can occur with/without same setting of wal_checksum, to avoid complications from crashes immediately after turning GUC on. Surely not. Otherwise even the on setting is not really a defense. Only when the CRC is exactly zero, which happens very very rarely. It works most of the time doesn't exactly satisfy me. What's the use-case for changing the variable on the fly anyway? Seems a better solution is just to lock down the setting at postmaster start. That would prevent us from using the secondary checkpoint location, in the case of a crash effecting the primary checkpoint when it is a shutdown checkpoint where we changed the setting of wal_checksum. It seemed safer to allow a very rare error through to the next level of error checking rather than to close the door so tight that recovery would not be possible in a very rare case. If your're good with server start, so am I. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] 8.3 pending patch queue
On 1/4/07, Simon Riggs [EMAIL PROTECTED] wrote: On Thu, 2007-01-04 at 09:09 -0500, Andrew Dunstan wrote: That also happens. The only way I can see of ensuring it does not happen would be to auto-process all patch submissions. Sounds a good idea. Patch farm anyone? Auto apply/make check? I'm actually trying to simplify something I was working on at OSDL to do this. PLM at OSDL was a little to Linux focused. Will let everyone know when I have a working prototype. Mark ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
* Simon Riggs: Surely not. Otherwise even the on setting is not really a defense. Only when the CRC is exactly zero, which happens very very rarely. Have you tried switching to Adler32 instead of CRC32? -- Florian Weimer[EMAIL PROTECTED] BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Thu, 2007-01-04 at 17:58 +0100, Florian Weimer wrote: * Simon Riggs: Surely not. Otherwise even the on setting is not really a defense. Only when the CRC is exactly zero, which happens very very rarely. Have you tried switching to Adler32 instead of CRC32? No. Please explain further. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
Florian Weimer [EMAIL PROTECTED] writes: Have you tried switching to Adler32 instead of CRC32? Is anything known about the error detection capabilities of Adler32? There's a lot of math behind CRCs but AFAIR Adler's method is pretty much ad-hoc. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] ReadyForQuery()
In postgres.c main loop, I note that we issue ReadyForQuery() *after* updating pgstat and setting the ps display. Wouldn't it be better to issue ReadyForQuery() and then issue the stat stuff in the gap between processing? That way we would be less likely to care about pgstat and the ps, potentially getting rid of the GUCs for them completely? (I'm looking to place a PG_TRACE macro in there, so we can work out how long is spent waiting for client.) -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 8.3 pending patch queue
[EMAIL PROTECTED] wrote: On 1/4/07, Simon Riggs [EMAIL PROTECTED] wrote: On Thu, 2007-01-04 at 09:09 -0500, Andrew Dunstan wrote: That also happens. The only way I can see of ensuring it does not happen would be to auto-process all patch submissions. Sounds a good idea. Patch farm anyone? Auto apply/make check? I'm actually trying to simplify something I was working on at OSDL to do this. PLM at OSDL was a little to Linux focused. Will let everyone know when I have a working prototype. Feel free to discuss design/functionality any time. For example, a mechanism to feed patches to the buildfarm has previously been suggested. If this could be done in some automated, controlled and reasonably safe way it might be useful - it might afford reviewers a try before you buy option. Also, hooking this up with the stuff that Lukas Smith is doing might be good. cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] ReadyForQuery()
Simon Riggs [EMAIL PROTECTED] writes: Wouldn't it be better to issue ReadyForQuery() and then issue the stat stuff in the gap between processing? To me, ready for query means ready for query, not I think I might be ready soon. Otherwise you could argue for trying to move the message emission much further upstream than that. Another problem is that on a lot of kernels, control swaps to the client process the instant we issue the send(), and if the client is well-coded control will swap back when it send()s us the next query. If we rearrange things as you suggest then the state display will become quite misleading: it will claim we are still busy when actually the client has the result, and it will switch to idle *after* we've received a new command. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] 8.3 pending patch queue
On 1/4/07, Andrew Dunstan [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: On 1/4/07, Simon Riggs [EMAIL PROTECTED] wrote: On Thu, 2007-01-04 at 09:09 -0500, Andrew Dunstan wrote: That also happens. The only way I can see of ensuring it does not happen would be to auto-process all patch submissions. Sounds a good idea. Patch farm anyone? Auto apply/make check? I'm actually trying to simplify something I was working on at OSDL to do this. PLM at OSDL was a little to Linux focused. Will let everyone know when I have a working prototype. Feel free to discuss design/functionality any time. For example, a mechanism to feed patches to the buildfarm has previously been suggested. If this could be done in some automated, controlled and reasonably safe way it might be useful - it might afford reviewers a try before you buy option. Also, hooking this up with the stuff that Lukas Smith is doing might be good. I'll start another thread about what PLM is doing and what my initial ideas are. Do you have a pointer to what Lukas Smith is doing? Mark ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] ReadyForQuery()
On Thu, 2007-01-04 at 13:17 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Wouldn't it be better to issue ReadyForQuery() and then issue the stat stuff in the gap between processing? To me, ready for query means ready for query, not I think I might be ready soon. Otherwise you could argue for trying to move the message emission much further upstream than that. Another problem is that on a lot of kernels, control swaps to the client process the instant we issue the send(), and if the client is well-coded control will swap back when it send()s us the next query. If we rearrange things as you suggest then the state display will become quite misleading: it will claim we are still busy when actually the client has the result, and it will switch to idle *after* we've received a new command. Yeh, guessed there were some good arguments for the way it was. My thinking was, if the client is local and therefore likely to be fast, we could check for the reply before we signal stats. That way we could avoid posting idle altogether when we are very busy (and therefore would like the extra CPU/path length reduction). OTOH if the client is on the other end of a network, the duration is relatively lengthy and we'll easily be able to do things in the proposed order without a problem. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2007-01-04 at 11:09 -0500, Tom Lane wrote: It works most of the time doesn't exactly satisfy me. It seemed safer to allow a very rare error through to the next level of error checking rather than to close the door so tight that recovery would not be possible in a very rare case. If a DBA is turning checksums off at all, he's already bought into the assumption that he's prepared to recover from backups. What you don't seem to get here is that this feature is pretty darn questionable in the first place, and for it to have a side effect of poking a hole in the system's reliability even when it's off is more than enough to get it rejected outright. It's just a No Sale. I don't believe that the hole is real small, either, as overwrite-with-zeroes is not exactly an unheard-of failure mode for filesystems. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Thu, 2007-01-04 at 12:13 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2007-01-04 at 11:09 -0500, Tom Lane wrote: It works most of the time doesn't exactly satisfy me. It seemed safer to allow a very rare error through to the next level of error checking rather than to close the door so tight that recovery would not be possible in a very rare case. If a DBA is turning checksums off at all, he's already bought into the assumption that he's prepared to recover from backups. What you don't seem to get here is that this feature is pretty darn questionable in the first place, and for it to have a side effect of poking a hole in the system's reliability even when it's off is more than enough to get it rejected outright. It's just a No Sale. I get it, and I listened. I'm was/am happy to do it the way you suggested; I was merely explaining that I had considered the issue. New patch enclosed. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com Index: src/backend/access/transam/xlog.c === RCS file: /projects/cvsroot/pgsql/src/backend/access/transam/xlog.c,v retrieving revision 1.259 diff -c -r1.259 xlog.c *** src/backend/access/transam/xlog.c 8 Dec 2006 19:50:53 - 1.259 --- src/backend/access/transam/xlog.c 4 Jan 2007 17:34:13 - *** *** 137,142 --- 137,143 char *XLOG_sync_method = NULL; const char XLOG_sync_method_default[] = DEFAULT_SYNC_METHOD_STR; bool fullPageWrites = true; + boolcompute_wal_crc = true; #ifdef WAL_DEBUG bool XLOG_DEBUG = false; *** *** 607,613 */ doPageWrites = fullPageWrites || Insert-forcePageWrites; ! INIT_CRC32(rdata_crc); len = 0; for (rdt = rdata;;) { --- 608,614 */ doPageWrites = fullPageWrites || Insert-forcePageWrites; ! INIT_CRC32(rdata_crc); len = 0; for (rdt = rdata;;) { *** *** 615,621 { /* Simple data, just include it */ len += rdt-len; ! COMP_CRC32(rdata_crc, rdt-data, rdt-len); } else { --- 616,623 { /* Simple data, just include it */ len += rdt-len; ! if (compute_wal_crc) ! COMP_CRC32(rdata_crc, rdt-data, rdt-len); } else { *** *** 630,636 else if (rdt-data) { len += rdt-len; ! COMP_CRC32(rdata_crc, rdt-data, rdt-len); } break; } --- 632,639 else if (rdt-data) { len += rdt-len; ! if (compute_wal_crc) ! COMP_CRC32(rdata_crc, rdt-data, rdt-len); } break; } *** *** 647,653 else if (rdt-data) { len += rdt-len; ! COMP_CRC32(rdata_crc, rdt-data, rdt-len); } break; } --- 650,657 else if (rdt-data) { len += rdt-len; ! if (compute_wal_crc) ! COMP_CRC32(rdata_crc, rdt-data, rdt-len); } break; } *** *** 662,701 rdt = rdt-next; } ! /* ! * Now add the backup block headers and data into the CRC ! */ ! for (i = 0; i XLR_MAX_BKP_BLOCKS; i++) ! { ! if (dtbuf_bkp[i]) ! { ! BkpBlock *bkpb = (dtbuf_xlg[i]); ! char *page; ! COMP_CRC32(rdata_crc, ! (char *) bkpb, ! sizeof(BkpBlock)); ! page = (char *) BufferGetBlock(dtbuf[i]); ! if (bkpb-hole_length == 0) ! { ! COMP_CRC32(rdata_crc, ! page, ! BLCKSZ); ! } ! else ! { ! /* must skip the hole */ ! COMP_CRC32(rdata_crc, ! page, ! bkpb-hole_offset); ! COMP_CRC32(rdata_crc, ! page + (bkpb-hole_offset + bkpb-hole_length), ! BLCKSZ - (bkpb-hole_offset + bkpb-hole_length)); ! } ! } ! } ! ! /* * NOTE: We disallow len == 0 because it provides a useful bit of extra * error checking in ReadRecord. This means that all callers of * XLogInsert must supply at least some not-in-a-buffer data. However, we --- 666,708 rdt = rdt-next; } ! if (compute_wal_crc) ! { ! /* ! * Now add the backup block headers and data into the CRC ! */ ! for (i = 0; i XLR_MAX_BKP_BLOCKS; i++) ! { ! if (dtbuf_bkp[i]) ! { ! BkpBlock *bkpb = (dtbuf_xlg[i]); ! char *page; ! ! COMP_CRC32(rdata_crc, !(char *) bkpb, !sizeof(BkpBlock)); ! page = (char *) BufferGetBlock(dtbuf[i]); ! if (bkpb-hole_length == 0) ! { ! COMP_CRC32(rdata_crc, !page, !BLCKSZ); ! } ! else ! { ! /* must skip the hole */ !
Re: [HACKERS] [PATCHES] Small vcbuild patch
Magnus Hagander [EMAIL PROTECTED] writes: Oneliner that adds the capability to deal with defines that set string values - needs to be quoted in XML. Applied, thanks. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 8.3 pending patch queue
[EMAIL PROTECTED] wrote: On 1/4/07, Andrew Dunstan [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: On 1/4/07, Simon Riggs [EMAIL PROTECTED] wrote: On Thu, 2007-01-04 at 09:09 -0500, Andrew Dunstan wrote: That also happens. The only way I can see of ensuring it does not happen would be to auto-process all patch submissions. Sounds a good idea. Patch farm anyone? Auto apply/make check? I'm actually trying to simplify something I was working on at OSDL to do this. PLM at OSDL was a little to Linux focused. Will let everyone know when I have a working prototype. Feel free to discuss design/functionality any time. For example, a mechanism to feed patches to the buildfarm has previously been suggested. If this could be done in some automated, controlled and reasonably safe way it might be useful - it might afford reviewers a try before you buy option. Also, hooking this up with the stuff that Lukas Smith is doing might be good. I'll start another thread about what PLM is doing and what my initial ideas are. Do you have a pointer to what Lukas Smith is doing? some of the stuff in here: http://developer.postgresql.org/index.php/Main_Page cheers andrew ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Reverse-sort indexes and NULLS FIRST/LAST sorting
Martijn van Oosterhout kleptog@svana.org writes: On Mon, Jan 01, 2007 at 05:53:35PM -0500, Tom Lane wrote: One way we could handle this is to say that reverse-sort indexes are implemented by adding explicit catalog entries for reverse-sort opclasses, with no additions to the underlying btree index mechanisms. So you might make an index using a command like CREATE INDEX fooi ON foo (x, y reverse_int4_ops); Personally I favour this approach. It's also the approach similar to what I did with the COLLATE stuff. It's IMHO the cleanest because it encapsulates the order at the level where it's important. In particular, NULLS FIRST/LAST makes sense for btree, but no other index type, so storing the order seperatly is wasted space for any other index type. After further thought I'm leaning towards doing it the other way, ie, with explicit DESC and NULLS FIRST/LAST modifiers attached to index columns, rather than specialized opclasses. Comparing this to the concerns that have been mentioned: * Allows indexes to support mixed nulls-first-or-last orderings. Although I can't make a real strong argument why that's important, I have a feeling that we'll miss it if we can't handle it. * Solves the problem once, instead of requiring extra code in every datatype. Even though that code would largely be boilerplate copy-and-paste stuff, it's still tedious and easy to get wrong. * Possible slowdown of btree code: after looking a bit, I think this is a red herring. Most of the work would be done by flipping operator strategy numbers during scan setup. As best I can tell without actually coding it, the only addition required to the inner-loop comparison function will be one extra test-and-branch in the code paths where we've detected we are comparing a NULL to a non-NULL. That doesn't seem like a big problem. * Ugly wart added to pg_index: yeah, it is, although I have an idea that might make it more palatable. Rather than add a couple of boolean vectors (which is a datatype we don't have anyway), I'm thinking of adding an int2vector field named something like indoption which would store per-column flag bits with index-AM-specific meanings. DESC and NULLS FIRST/LAST would take up two of these bits for btree (and any other index AM that supports ordered scans), the rest are available for expansion. I do not know if there's any immediate use for such flag bits for GiST or GIN, but one obvious possible use for btree is to store collation info. (I suppose we can find a way to identify a collation with a dozen or less bits, so it should fit.) Another possible objection is that in the proposed CREATE INDEX syntax index-column-id [ opclass-name ] [ DESC ] [ NULLS {FIRST|LAST} ] DESC must be a fully reserved word else it can't be distinguished from an opclass name. But guess what, it already is. Comments? regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
[HACKERS] ideas for auto-processing patches
OSDL had a tool called PLM with a primary goal to test patches against the Linux kernel. It applied them and built them on multiple platforms. It's a pretty simple idea and here are some links to what it did; the systems appear to still be up for the moment so here are a couple of links to what it did. Summary of build results: http://plm.testing.osdl.org/patches/show/linux-2.6.20-rc3-git3 Summary of recent patches submitted into the system: http://plm.testing.osdl.org/patches/search_result It also provides an rss feed: http://plm.testing.osdl.org/rss There a couple of things initial things I wanted to change, which I think are improvements: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. 2. Apply and build patches against daily updates from the repositories, as opposed to only against a specified version of the source code. Thoughts? Regards, Mark ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Tabs or Spaces
Andrew Dunstan wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. No, it does the opposite (or I would hope so anyway). Sure looks that way in the code. In fact it detabifies and then retabifies, from what I can see. cheers andrew ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Tabs or Spaces
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. No, it does the opposite (or I would hope so anyway). Sorry, the mistake was in a reference to entab. It now reads: entab converts spaces to tabs, used by pgindent -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Tabs or Spaces
On Thu, 2007-01-04 at 16:01 -0500, Bruce Momjian wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. No, it does the opposite (or I would hope so anyway). Sorry, the mistake was in a reference to entab. It now reads: entab converts spaces to tabs, used by pgindent S'OK, just thought it was funny I'd been doing it backwards all this time: Actively been replacing tabs with spaces. :-) -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Rare corruption of pg_class index
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tom Lane replied: So as a general rule, the system tables should be considered a special case as far as transactional activity? To be more precise, you are saying that a system table must be locked in access exclusive mode before any change is made to guarantee no problems occur? No, I didn't say that --- I said that you need to lock the table whose schema you're trying to modify, to ensure that no one else is in the midst of accessing it using the old schema info. Sorry, I did mean the target table. So the oft-given advice of UPDATE pg_class SET relhasrules = false is actually completely unsafe unless the entire referenced table is completely locked, and unless you are using at least 8.2? I don't recall having ever given *that* advice to anyone. But yes, it's unsafe if there might be concurrent access to that table. The only context I've ever seen people use this sort of thing in is pg_restore --disable-triggers, and in that situation I think there's an implicit assumption that no one else is busy modifying the table you're restoring into. Not, not your advice, and perhaps not as common as SET reltriggers, but still invaluable for things like bulk loading. Thanks for the responses, I think I've finally got my head around the problem. At the very least, I've discovered another good reason to push production sites to use 8.2. - -- Greg Sabino Mullane [EMAIL PROTECTED] End Point Corporation PGP Key: 0x14964AC8 200701041708 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iD8DBQFFnXrivJuQZxSWSsgRAlSUAJ9xSg6NBO984pBT0Ea2fAnhFdfX/QCcDXKj 2j4m8MdNCnLX5iVXz4D8AAI= =NkGM -END PGP SIGNATURE- ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
* Tom Lane: Florian Weimer [EMAIL PROTECTED] writes: Have you tried switching to Adler32 instead of CRC32? Is anything known about the error detection capabilities of Adler32? There's a lot of math behind CRCs but AFAIR Adler's method is pretty much ad-hoc. Correct me if I'm wrong, but the main reason for the WAL CRC is to detect partial WAL writes (due to improper caching, for instance). This means that you're out of the realm of traditional CRC analysis anyway, because the things you are guarding against are neither burts errors nor n-bit errors (for small n). ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
Florian Weimer [EMAIL PROTECTED] writes: * Tom Lane: There's a lot of math behind CRCs but AFAIR Adler's method is pretty much ad-hoc. Correct me if I'm wrong, but the main reason for the WAL CRC is to detect partial WAL writes (due to improper caching, for instance). Well, that's *a* reason, but not the only one, and IMHO not one that gives any particular guidance on what kind of checksum to use. This means that you're out of the realm of traditional CRC analysis anyway, because the things you are guarding against are neither burts errors nor n-bit errors (for small n). I think short burst errors are fairly likely: the kind of scenario I'm worried about is a wild store corrupting a word of a WAL entry while it's waiting around to be written in the WAL buffers. So the CRC math does give me some comfort that that'll be detected. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
* Tom Lane: I think short burst errors are fairly likely: the kind of scenario I'm worried about is a wild store corrupting a word of a WAL entry while it's waiting around to be written in the WAL buffers. Ah, does this mean that each WAL entry gets its own checksum? In this case, Adler32 is indeed suboptimal because it doesn't use the full 32 bits for short inputs. It might still catch many wild stores, but the statistics are worse than for CRC32. (I had assumed that PostgreSQLs WAL checksumming was justified by the partial write issue. The wild store could easily occur with a heap page, too, and AFAIK, tuples, aren't checksummed. Which would be an interesting option, I guess.) ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] [PATCHES] wal_checksum = on (default) | off
Florian Weimer [EMAIL PROTECTED] writes: Ah, does this mean that each WAL entry gets its own checksum? Right. (I had assumed that PostgreSQLs WAL checksumming was justified by the partial write issue. The wild store could easily occur with a heap page, too, and AFAIK, tuples, aren't checksummed. Which would be an interesting option, I guess.) We've discussed it but there's never been a pressing reason to do it. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Problem with windows installer
Andreas 'ads' Scherbaum wrote: Hello all, a friend of mine ran into a problem installing PostgreSQL 8.0.9 on a Windows XP Pro machine. Before anyone is asking: it has to be a 8.0.x version and we even tried to install 8.2 and it did not work. Ok, the problem is: after installing all the files the installer wants to init the db and fails with the following error: Failed to create process for initdb: Anmeldung fehlgeschlagen: Dem Benutzer wurde der geforderte Anmeldetyp auf diesem Computer nicht erteilt. The german message means: Login failed: a requested login type was not given to the user on this computer. Since the message was returned in german and the installer was started in english, the message is generated by windows. See the FAQ at http://www.postgresql.org/docs/faqs.FAQ_windows.html#3.3 (question 3.3). //Magnus ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Problem with windows installer
Magnus Hagander wrote: Andreas 'ads' Scherbaum wrote: Hello all, a friend of mine ran into a problem installing PostgreSQL 8.0.9 on a Windows XP Pro machine. Before anyone is asking: it has to be a 8.0.x version and we even tried to install 8.2 and it did not work. Ok, the problem is: after installing all the files the installer wants to init the db and fails with the following error: Failed to create process for initdb: Anmeldung fehlgeschlagen: Dem Benutzer wurde der geforderte Anmeldetyp auf diesem Computer nicht erteilt. The german message means: Login failed: a requested login type was not given to the user on this computer. Since the message was returned in german and the installer was started in english, the message is generated by windows. See the FAQ at http://www.postgresql.org/docs/faqs.FAQ_windows.html#3.3 (question 3.3). I discussed this briefly with Robert on IM yesterday - he told me the account was installer created. Without a PC at the time I couldn't look into it further :-( Regards, Dave. ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[HACKERS] V4 protocol, extensible?
In another thread I was made aware that current protocol may not be easily extensible (if at all), and so the backend-data a certain driver needs may not be so easily added. http://www.postgresql.org/docs/current/static/protocol-message-formats.html So if there is not already a roadmap for this, one idea may be to use some kind of key-length-value type encoding, which seems to be one of the most accepted formats for extensible streams (though I am by no means an expert). Names could just be ints/oids and not chars to their real name. Just wanted to see if anyone had any thoughts on this, or if my presumption that metatdata in the current V3 cannot be easily added too, is wrong. Thank you, Ken ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] ideas for auto-processing patches
On Thu, 4 Jan 2007 [EMAIL PROTECTED] wrote: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. It seems to me that a better approach might be to mirror the CVS repo -- or at least make that an option -- and pull the sources locally. Having to pull down 100MB of data for every build might be onerous to some build farm members. Thanks, Gavin ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Problem with windows installer
On Thu, 04 Jan 2007 22:55:52 + Dave Page [EMAIL PROTECTED] wrote: Magnus Hagander wrote: See the FAQ at http://www.postgresql.org/docs/faqs.FAQ_windows.html#3.3 (question 3.3). I discussed this briefly with Robert on IM yesterday - he told me the account was installer created. Without a PC at the time I couldn't look into it further :-( We tried both, let the installer create the user (this works, he also added the service part) and create the user manually (and let the installer do the service part). In any case the installer stops at initdb. Bye -- Andreas 'ads' Scherbaum Failure is not an option. It comes bundled with your Microsoft product. (Ferenc Mantfeld) ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Tabs or Spaces
Simon Riggs wrote: On Thu, 2007-01-04 at 16:01 -0500, Bruce Momjian wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Minor request for clarification: I read in the Developer's FAQ that tabs should be interpreted as 4 spaces. ...and also that pgindent replaces tabs as spaces. No, it does the opposite (or I would hope so anyway). Sorry, the mistake was in a reference to entab. It now reads: entab converts spaces to tabs, used by pgindent S'OK, just thought it was funny I'd been doing it backwards all this time: Actively been replacing tabs with spaces. :-) That explains why your patches always seemed to be so weird to me spacing-wise :-) -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Tabs or Spaces
On Thu, 2007-01-04 at 21:41 -0300, Alvaro Herrera wrote: Simon Riggs wrote: S'OK, just thought it was funny I'd been doing it backwards all this time: Actively been replacing tabs with spaces. :-) That explains why your patches always seemed to be so weird to me LOL spacing-wise :-) Nobody ever said... laughs/rolls eyes/sighs -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] ideas for auto-processing patches
Gavin Sherry wrote: On Thu, 4 Jan 2007 [EMAIL PROTECTED] wrote: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. It seems to me that a better approach might be to mirror the CVS repo -- or at least make that an option -- and pull the sources locally. Having to pull down 100MB of data for every build might be onerous to some build farm members. Another idea is using the git-cvs interoperability system, as described here (albeit with SVN, but the idea is the same): http://tw.apinc.org/weblog/2007/01/03#subverting-git Now, if we were to use a distributed system like Monotone this sort of thing would be completely a non-issue ... -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] ideas for auto-processing patches
On Thu, 4 Jan 2007, Alvaro Herrera wrote: Gavin Sherry wrote: On Thu, 4 Jan 2007 [EMAIL PROTECTED] wrote: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. It seems to me that a better approach might be to mirror the CVS repo -- or at least make that an option -- and pull the sources locally. Having to pull down 100MB of data for every build might be onerous to some build farm members. Another idea is using the git-cvs interoperability system, as described here (albeit with SVN, but the idea is the same): http://tw.apinc.org/weblog/2007/01/03#subverting-git It seems like that will just add one more cog to the machinary for no extra benefit. Am I missing something? Now, if we were to use a distributed system like Monotone this sort of thing would be completely a non-issue ... Monotone is so 2006. The new new thing is mercurial! Gavin ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] ideas for auto-processing patches
Gavin Sherry wrote: On Thu, 4 Jan 2007 [EMAIL PROTECTED] wrote: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. It seems to me that a better approach might be to mirror the CVS repo -- or at least make that an option -- and pull the sources locally. Having to pull down 100MB of data for every build might be onerous to some build farm members. I am not clear about what is being proposed. Currently buildfarm syncs against (or pulls a fresh copy from, depending on configuration) either the main anoncvs repo or a mirror (which you can get using cvsup or rsync, among other mechanisms). I can imagine a mechanism in which we pull certain patches from a patch server (maybe using an RSS feed, or a SOAP call?) which could be applied before the run. I wouldn't want to couple things much more closely than that. The patches would need to be vetted first, or no sane buildfarm owner will want to use them. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] ideas for auto-processing patches
On Thu, 4 Jan 2007, Andrew Dunstan wrote: Gavin Sherry wrote: On Thu, 4 Jan 2007 [EMAIL PROTECTED] wrote: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. It seems to me that a better approach might be to mirror the CVS repo -- or at least make that an option -- and pull the sources locally. Having to pull down 100MB of data for every build might be onerous to some build farm members. I am not clear about what is being proposed. Currently buildfarm syncs against (or pulls a fresh copy from, depending on configuration) either the main anoncvs repo or a mirror (which you can get using cvsup or rsync, among other mechanisms). I can imagine a mechanism in which we pull certain patches from a patch server (maybe using an RSS feed, or a SOAP call?) which could be applied before the run. I wouldn't want to couple things much more closely than that. With PLM, you could test patches against various code branches. I'd guessed Mark would want to provide this capability. Pulling branches from anonvcvs regularly might be burdensome bandwidth-wise. So, like you say, a local mirror would be beneficial for patch testing. The patches would need to be vetted first, or no sane buildfarm owner will want to use them. It would be nice if there could be a class of trusted users whose patches would not have to be vetted. Thanks, Gavin ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] InitPostgres and flatfiles question
Tom Lane wrote: Markus Schiltknecht [EMAIL PROTECTED] writes: Tom Lane wrote: Hm, that's an interesting point. psql's -c just shoves its whole argument string at the backend in one PQexec(), instead of dividing at semicolons as psql does with normal input. Is there a good reason to not let psql -c behave exactly like psql from STDIN? Backwards compatibility, mostly --- there seems to be a considerable risk of subtly breaking people's scripts if we change the transactional boundaries for psql -c commands. True, but if we keep hitting people who don't expect this behavior, I wonder if we should just fix it and mention it in the release notes. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] ideas for auto-processing patches
On 1/4/07, Gavin Sherry [EMAIL PROTECTED] wrote: On Thu, 4 Jan 2007, Andrew Dunstan wrote: Gavin Sherry wrote: On Thu, 4 Jan 2007 [EMAIL PROTECTED] wrote: 1. Pull source directly from repositories (cvs, git, etc.) PLM doesn't really track actually scm repositories. It requires directories of source code to be traversed, which are set up by creating mirrors. It seems to me that a better approach might be to mirror the CVS repo -- or at least make that an option -- and pull the sources locally. Having to pull down 100MB of data for every build might be onerous to some build farm members. I am not clear about what is being proposed. Currently buildfarm syncs against (or pulls a fresh copy from, depending on configuration) either the main anoncvs repo or a mirror (which you can get using cvsup or rsync, among other mechanisms). I can imagine a mechanism in which we pull certain patches from a patch server (maybe using an RSS feed, or a SOAP call?) which could be applied before the run. I wouldn't want to couple things much more closely than that. With PLM, you could test patches against various code branches. I'd guessed Mark would want to provide this capability. Yeah, that pretty much covers it. Pulling branches from anonvcvs regularly might be burdensome bandwidth-wise. So, like you say, a local mirror would be beneficial for patch testing. Right some sort of local mirror would definitely speed things up. The patches would need to be vetted first, or no sane buildfarm owner will want to use them. It would be nice if there could be a class of trusted users whose patches would not have to be vetted. PLM's authentication is tied to OSDL's internal authentication system, but some I imagine setting up accounts and trusting specific users would be an easy first try. Regards, Mark ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] 8.3 pending patch queue
Andrew Dunstan wrote: Simon Riggs wrote: I'm not clear about the difference between the unapplied patches list and the hold list. What is the significance of the two lists? AIUI, the hold list is those patches providing new features that were held over between 8.2 feature freeze and 8.2 branch. Since they have been around for a while I think they have some claim to priority. The other list is just the normal running list of such patches that Bruce keeps. FYI, I haven't been applying patches as aggressively because we were kind of focused on 8.2.0 and the holidays. Now that that is over, there will be more activity. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] ideas for auto-processing patches
Gavin Sherry wrote: With PLM, you could test patches against various code branches. I'd guessed Mark would want to provide this capability. Pulling branches from anonvcvs regularly might be burdensome bandwidth-wise. So, like you say, a local mirror would be beneficial for patch testing. I think you're missing the point. Buildfarm members already typically have or can get very cheaply a copy of each branch they build (HEAD and/or REL*_*_STABLE). As long as the patch feed is kept to just patches which they can apply there should be no great bandwidth issues. The patches would need to be vetted first, or no sane buildfarm owner will want to use them. It would be nice if there could be a class of trusted users whose patches would not have to be vetted. Beyond committers? cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] InitPostgres and flatfiles question
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Markus Schiltknecht [EMAIL PROTECTED] writes: Is there a good reason to not let psql -c behave exactly like psql from STDIN? Backwards compatibility, mostly --- there seems to be a considerable risk of subtly breaking people's scripts if we change the transactional boundaries for psql -c commands. True, but if we keep hitting people who don't expect this behavior, I wonder if we should just fix it and mention it in the release notes. One other point is that if we change -c's behavior, there won't be *any* way to submit multiple queries in a single PQexec using plain psql --- it will require hacking up a special test program using libpq directly. Unless we have plans to obsolete multi-queries-per-PQexec altogether, this doesn't seem like a good idea. OTOH, you could argue that forbidding multiple queries in one PQexec isn't a bad idea; it would provide an additional defense against SQL-injection attacks. We did that already in the extended query protocol and I've not heard many complaints. I'd be willing to buy into doing both together, perhaps. regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] InitPostgres and flatfiles question
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Markus Schiltknecht [EMAIL PROTECTED] writes: Is there a good reason to not let psql -c behave exactly like psql from STDIN? Backwards compatibility, mostly --- there seems to be a considerable risk of subtly breaking people's scripts if we change the transactional boundaries for psql -c commands. True, but if we keep hitting people who don't expect this behavior, I wonder if we should just fix it and mention it in the release notes. One other point is that if we change -c's behavior, there won't be *any* way to submit multiple queries in a single PQexec using plain psql --- it will require hacking up a special test program using libpq directly. Unless we have plans to obsolete multi-queries-per-PQexec altogether, this doesn't seem like a good idea. What value is allowing multiple queies via PQexec() via psql, aside from avoiding BEGIN/END around your -c query string? OTOH, you could argue that forbidding multiple queries in one PQexec isn't a bad idea; it would provide an additional defense against SQL-injection attacks. We did that already in the extended query protocol and I've not heard many complaints. I'd be willing to buy into doing both together, perhaps. True. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] InitPostgres and flatfiles question
Bruce Momjian [EMAIL PROTECTED] writes: What value is allowing multiple queies via PQexec() The only argument I can think of is that it allows applications to be sloppy about parsing a SQL script into individual commands before they send it. (I think initdb may be guilty of exactly that BTW...) At the same time you could argue that such sloppiness is inherently a Bad Idea. regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] ideas for auto-processing patches
On Thu, 4 Jan 2007, Andrew Dunstan wrote: Gavin Sherry wrote: With PLM, you could test patches against various code branches. I'd guessed Mark would want to provide this capability. Pulling branches from anonvcvs regularly might be burdensome bandwidth-wise. So, like you say, a local mirror would be beneficial for patch testing. I think you're missing the point. Buildfarm members already typically have or can get very cheaply a copy of each branch they build (HEAD and/or REL*_*_STABLE). As long as the patch feed is kept to just patches which they can apply there should be no great bandwidth issues. Right... my comment was more for Mark. It would be nice if there could be a class of trusted users whose patches would not have to be vetted. Beyond committers? Hmmm... good question. I think so. I imagine the group would be small though. Thanks, Gavin ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] InitPostgres and flatfiles question
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: What value is allowing multiple queries via PQexec() The only argument I can think of is that it allows applications to be sloppy about parsing a SQL script into individual commands before they send it. (I think initdb may be guilty of exactly that BTW...) At the same time you could argue that such sloppiness is inherently a Bad Idea. I thought the idea was that psql was going to split multiple -c commands into separate PQexec() calls, so sloppy was OK, and if they want a single transaction, add BEGIN/END to the string. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] InitPostgres and flatfiles question
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: The only argument I can think of is that it allows applications to be sloppy about parsing a SQL script into individual commands before they send it. (I think initdb may be guilty of exactly that BTW...) At the same time you could argue that such sloppiness is inherently a Bad Idea. I thought the idea was that psql was going to split multiple -c commands into separate PQexec() calls, so sloppy was OK, and if they want a single transaction, add BEGIN/END to the string. No, psql isn't the point: we can certainly make its behavior match the backend's. What I'm wondering about is the effect on random PG-using applications: should we forbid them from sending multiple SQL commands per PQexec (or equivalent in other client library APIs)? Backwards compatibility says no, but you can make some decent arguments for forbidding it anyway. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] InitPostgres and flatfiles question
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: The only argument I can think of is that it allows applications to be sloppy about parsing a SQL script into individual commands before they send it. (I think initdb may be guilty of exactly that BTW...) At the same time you could argue that such sloppiness is inherently a Bad Idea. I thought the idea was that psql was going to split multiple -c commands into separate PQexec() calls, so sloppy was OK, and if they want a single transaction, add BEGIN/END to the string. No, psql isn't the point: we can certainly make its behavior match the backend's. What I'm wondering about is the effect on random PG-using applications: should we forbid them from sending multiple SQL commands per PQexec (or equivalent in other client library APIs)? Backwards compatibility says no, but you can make some decent arguments for forbidding it anyway. Yea, I was trying to separate the psql case from the PQexec() case. For psql, I think it is clear that -c _should_ act like a normal stdin query. That would eliminate confusion, and I don't see a large loss of functionality. The PQexec() case, the problem is we don't know who is using multi-statement PQexec() calls, and users can't always add BEGIN/END to fix them if they are embedded in applications. What we could do it do both and see what pushback we get during beta. We could always revert it before the final release. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Problem with windows installer
Hello all, a friend of mine ran into a problem installing PostgreSQL 8.0.9 on a Windows XP Pro machine. Before anyone is asking: it has to be a 8.0.x version and we even tried to install 8.2 and it did not work. Ok, the problem is: after installing all the files the installer wants to init the db and fails with the following error: Failed to create process for initdb: Anmeldung fehlgeschlagen: Dem Benutzer wurde der geforderte Anmeldetyp auf diesem Computer nicht erteilt. The german message means: Login failed: a requested login type was not given to the user on this computer. Since the message was returned in german and the installer was started in english, the message is generated by windows. See the FAQ at http://www.postgresql.org/docs/faqs.FAQ_windows.html#3.3 (question 3.3). I discussed this briefly with Robert on IM yesterday - he told me the account was installer created. Without a PC at the time I couldn't look into it further :-( The faq still applies as the most likely reason. It can certainly happen with the installer created account as well, for example due to a group policy on the computer or domain. /Magnus ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] ideas for auto-processing patches
[EMAIL PROTECTED] schrieb: On 1/4/07, Gavin Sherry [EMAIL PROTECTED] wrote: On Thu, 4 Jan 2007, Andrew Dunstan wrote: ... Pulling branches from anonvcvs regularly might be burdensome bandwidth-wise. So, like you say, a local mirror would be beneficial for patch testing. Right some sort of local mirror would definitely speed things up. Easier speedup in this regard would be using subversion instead of cvs. It transfers only diffs to your working copy (or rather, to your last checkout) so its really saving on bandwidth. Regards Tino ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org