[HACKERS] Winner of naming discussions: Synchronous Commit
Thank you to everyone for input on the naming of Feature X. The clear winner of the parameter naming was synchronous_commit, which would imply we refer to the new feature as Asynchronous Commit in the release notes and general discussions. I will be now be starting to make changes throughout the patch so that this name is used for variable names, function calls and comments. After that I'll be working on docs for this feature, though they may not be available until next week, given my current schedule. The docs will include discussions of the proof of using mixed synch/asynch commits. The ideas, in chronological order, and voices in favour of them were: transaction_guarantee Idea: Simon Riggs Supporters: None deferred_commit Idea: Tom Lane Supporters: Richard Huxton synchronous_commit Idea: Greg Stark Supporters: Simon, Josh, Tom, Bruce, Florian reliable_commit Idea: Bruce Momjian Supporters: None commit_durability Idea: Simon Riggs Supporters: None confirmed_commit Idea: Greg Smith Supporters: None synchronous_commit is a clear winner. Most of the names have difficulties as to how we would refer to them when turned off: undurable commit, unconfirmed commit, unreliable commit don't work for me, nor does undeferred commit make sense for normal commits. Also, I can't see our press release saying PostgreSQL releases Unreliable Commit. :-) -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Bugtraq: Having Fun With PostgreSQL
Steve Atkins [EMAIL PROTECTED] writes: On Jun 23, 2007, at 11:03 AM, Magnus Hagander wrote: Out of curiosity, how do other databases deal with this? MySQL installs with an empty root password for access from localhost or the machines own IP address. It also installs an account with network access to any database beginning with test and possibly some more ill-defined accounts with local access. FWIW, on mysql 5.0.42 I see only [EMAIL PROTECTED] and [EMAIL PROTECTED] in a fresh-out-of-the-box installation; not sure where you got these other accounts, maybe a distro-specific modification? But the bottom line is that mysql's out-of-the-box behavior is *exactly* like our trust-for-local-connections behavior. Anyone on the box can do mysql -u root ... and the server will accept them as being superuser (they don't even have to know to enter an empty password, in my experience). regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] msvc and vista fun
I am still very unhappy about the way the MSVC builds work. Although we have managed to make it sort of work with the buildfarm script, it is distinctly fragile. Last night, for example, I had a build failure due to a badly installed zlib. The error state didn't come back to the buildfarm script, which carried on merrily to the check stage, which also naturally failed, but also didn't manage to give back an error state to the buildfarm script, and finally got a failure at the install stage. That means that we simply can't rely on this build system to give us accurate state on the buildfarm - we might have had a code breakage but it won't always tell us that. Part of the problem, it seems to me, is that this build system is a hodge-podge of Perl programs and Windows shell scripts. In at least one case we call perl to write a dynamically generated .bat file which we then execute. Since the perl is pretty much indispensable (or at least would need replacing by an engine of similar capability), I think we should make the whole thing a perl suite with some very minimal .bat wrappers if necessary. On a somewhat related note, I have had spectacular lack of success in getting either MSVC or MinGW builds to work on Vista - so much so that I have currently abandoned my attempts on that platform and I resorted to resuscitating an old XP box for testing. Following some advice from Magnus, I added ACLs to the build root for both an admin and a non-admin user (cacls buildroot /E /T /G AdminUser:C and similarly for a non-admin user) . I can build as the admin user but when I come to run initdb it fails, complaining that it can't find the postgres executable. If I then switch to the non-admin user, it can run initdb just fine. However, that user can't build, because it gets a mysterious failure from mt.exe. MinGW is even worse - it says it can't run gcc because it can't run cc1.exe (IIRC), so it fails at the configure stage! All of this has cost me quite a few hours in the last week, with very little to show for it. Perhaps someone would like to tell me how I can remedy these problems. More importantly, this should be in an FAQ or some such. Also, I would like to know if we have really tested out on Vista the privilege surrendering code that is is supposed to work in Windows. It looks to me like it might not be working. If we can make progress on these issues then I'll be happy. If not, I think we need to look carefully at what we say we can support, and what we declare to be still experimental. cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Bugtraq: Having Fun With PostgreSQL
Tom Lane wrote: Steve Atkins [EMAIL PROTECTED] writes: On Jun 23, 2007, at 11:03 AM, Magnus Hagander wrote: Out of curiosity, how do other databases deal with this? MySQL installs with an empty root password for access from localhost or the machines own IP address. It also installs an account with network access to any database beginning with test and possibly some more ill-defined accounts with local access. FWIW, on mysql 5.0.42 I see only [EMAIL PROTECTED] and [EMAIL PROTECTED] in a fresh-out-of-the-box installation; not sure where you got these other accounts, maybe a distro-specific modification? But the bottom line is that mysql's out-of-the-box behavior is *exactly* like our trust-for-local-connections behavior. Anyone on the box can do mysql -u root ... and the server will accept them as being superuser (they don't even have to know to enter an empty password, in my experience). This is all documented. For 5.1.x see: http://dev.mysql.com/doc/refman/5.1/en/default-privileges.html Perhaps we should add a section to our docs on securing the database. cheers andredw ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] msvc and vista fun
Andrew Dunstan wrote: On a somewhat related note, I have had spectacular lack of success in getting either MSVC or MinGW builds to work on Vista - so much so that I have currently abandoned my attempts on that platform and I resorted to resuscitating an old XP box for testing. Following some advice from Magnus, I added ACLs to the build root for both an admin and a non-admin user (cacls buildroot /E /T /G AdminUser:C and similarly for a non-admin user) . I can build as the admin user but when I come to run initdb it fails, complaining that it can't find the postgres executable. Yeah, I ran into that problem as well. I'll look at my Vista box when I'm in the office tomorrow and see if I can figure out what hack fixed it for me. If I then switch to the non-admin user, it can run initdb just fine. However, that user can't build, because it gets a mysterious failure from mt.exe. MinGW is even worse - it says it can't run gcc because it can't run cc1.exe (IIRC), so it fails at the configure stage! All of this has cost me quite a few hours in the last week, with very little to show for it. And that one... Perhaps someone would like to tell me how I can remedy these problems. More importantly, this should be in an FAQ or some such. Also, I would like to know if we have really tested out on Vista the privilege surrendering code that is is supposed to work in Windows. It looks to me like it might not be working. What makes you say that? If we can make progress on these issues then I'll be happy. If not, I think we need to look carefully at what we say we can support, and what we declare to be still experimental. Lack of completely reliable buildfarm support for MSVC or trouble setting up a buildenv doesn't mean we cannot support a platform - it just means we need o be extra vigilent to ensure that regresssion tests etc. do actually pass (though that level of failure would be visible on the BF I would hope - it certainly has been until now). Regards, Dave ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] msvc and vista fun
Andrew Dunstan wrote: I am still very unhappy about the way the MSVC builds work. Although we have managed to make it sort of work with the buildfarm script, it is distinctly fragile. Last night, for example, I had a build failure due to a badly installed zlib. The error state didn't come back to the buildfarm script, which carried on merrily to the check stage, which also naturally failed, but also didn't manage to give back an error I've heard you report this before, and I've tried and failed many times to reproduce it. It has *always* come back with the proper errorlevel when I've tried. So two questions: 1) are you absolutely sure that the problem is not in the buildfarm script? 2) What exactly was the error? So I can try to reproduce that exact problem here and see if I can find out what it is. What did you do wrong, and what was the error messages if you still have the log. state to the buildfarm script, and finally got a failure at the install stage. That means that we simply can't rely on this build system to give us accurate state on the buildfarm - we might have had a code breakage but it won't always tell us that. At least it broke eventually :-) But I agree that's a big problem. Part of the problem, it seems to me, is that this build system is a hodge-podge of Perl programs and Windows shell scripts. In at least one case we call perl to write a dynamically generated .bat file which we then execute. Which one is that? I can't find it. A simple findstr through all the perl files appears to contain nothing of the sort. Since the perl is pretty much indispensable (or at least would need replacing by an engine of similar capability), I think we should make the whole thing a perl suite with some very minimal .bat wrappers if necessary. That's mostly how it is now, no? What are you referring to specifically? The only longer ones I can see are: builddoc.bat - could certainly be put into perl stuff, but it's not involved in the buildfarm process, so... clean.bat - doesn't really matter, IMO, and again not involved in the buildfarm process vcregress.bat - sure, it could be made perl, but right now you can actually run the regression tests on an installed system without having perl at all... But that's not really a requirement, so we could change that one. The rest are just thin wrappers. Speaking of wrappers, are you planning to make the buildfarm use the modules instead of the perlscripts? IIRC, I made those modules specifically for the bf - and it should make it even easier to pick up the errors from there. I'll leave the Vista parts to Dave since I haven't used it at all there. These are really completely separate issues, and should be treated as such. //Magnus ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] msvc and vista fun
Magnus Hagander wrote: Andrew Dunstan wrote: I am still very unhappy about the way the MSVC builds work. Although we have managed to make it sort of work with the buildfarm script, it is distinctly fragile. Last night, for example, I had a build failure due to a badly installed zlib. The error state didn't come back to the buildfarm script, which carried on merrily to the check stage, which also naturally failed, but also didn't manage to give back an error I've heard you report this before, and I've tried and failed many times to reproduce it. It has *always* come back with the proper errorlevel when I've tried. So two questions: 1) are you absolutely sure that the problem is not in the buildfarm script? Pretty damn sure, yes. This code has functioned correctly on the buildfarm on every other platform since its inception. 2) What exactly was the error? So I can try to reproduce that exact problem here and see if I can find out what it is. What did you do wrong, and what was the error messages if you still have the log. I did indeed keep the logs. Steps to reproduce: rename your zdll.lib and then run the buildfarm script. Output: E:\progperl run_build.pl --test [17:51:47] checking out source ... [17:52:17] checking if build run needed ... [17:52:30] copying source to pgsql.3840 ... 3152 File(s) copied [17:53:44] running configure ... [17:53:44] running make ... [18:27:50] running make check ... [18:27:50] running make install ... Branch: HEAD Stage Install failed with status 2 from the bottom of the make log: Build FAILED. .\src\backend\utils\adt\oracle_compat.c(898): warning C4090: 'function' : different 'const' qualifiers .\src\backend\utils\adt\oracle_compat.c(900): warning C4090: 'function' : different 'const' qualifiers LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' LINK : fatal error LNK1104: cannot open file 'e:\prog\pgdepend\zlib\lib\zdll.lib' 2 Warning(s) 12 Error(s) Relevant perl code executed by buildfarm: chdir $pgsql/src/tools/msvc; @makeout = `build 21`; chdir $branch_root; my $status = $? 8; Could it possibly be that the exit status has only low bits set, and isn't what should be returned by wait()? I am checking on this now (time passes) ... no, the raw status returned is in fact 0. Now, I see that we do something different in the install case, which is why the error gets trapped there. For install, we don't call the .bat file at all, we call the install script directly: chdir $pgsql/src/tools/msvc; @makeout = `perl install.pl $installdir 21`; chdir $branch_root; my $status = $? 8; So it looks like the .bat files are not collecting the exit status and passing it on correctly. state to the buildfarm script, and finally got a failure at the install stage. That means that we simply can't rely on this build system to give us accurate state on the buildfarm - we might have had a code breakage but it won't always tell us that. At least it broke eventually :-) But I agree that's a big problem. yes. Part of the problem, it seems to me, is that this build system is a hodge-podge of Perl programs and Windows shell scripts. In at least one case we call perl to write a dynamically generated .bat file which we then execute. Which one is that? I can't find it. A simple findstr through all the perl files appears to contain nothing of the sort. perl ../../src/tools/msvc/getregress.pl regress.tmp.bat call regress.tmp.bat del regress.tmp.bat Since the perl is pretty much indispensable (or at least would need replacing by an engine of similar capability), I think we should make the whole thing a perl suite with some very minimal .bat wrappers if necessary. That's mostly how it is now, no? What are you referring to specifically? The only longer ones I can see are: builddoc.bat - could certainly be put into perl stuff, but it's not involved in the buildfarm process, so... clean.bat - doesn't really matter, IMO, and again not involved in the buildfarm process vcregress.bat - sure, it
Re: [HACKERS] msvc and vista fun
Dave Page wrote: Perhaps someone would like to tell me how I can remedy these problems. More importantly, this should be in an FAQ or some such. Also, I would like to know if we have really tested out on Vista the privilege surrendering code that is is supposed to work in Windows. It looks to me like it might not be working. What makes you say that? It was just a suspicion, but maybe the fact that the admin user can't run while the non-admin user can indicates that it is working (possibly too well). If we can make progress on these issues then I'll be happy. If not, I think we need to look carefully at what we say we can support, and what we declare to be still experimental. Lack of completely reliable buildfarm support for MSVC or trouble setting up a buildenv doesn't mean we cannot support a platform - it just means we need o be extra vigilent to ensure that regresssion tests etc. do actually pass (though that level of failure would be visible on the BF I would hope - it certainly has been until now). I am not only concerned about the buildfarm, but about the robustness, maintainability and simplicity of the build process. Having to have a completely separate build system for msvc is really quite sucky. cheers andrew ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Server-side support of all encodings
Tom Lane [EMAIL PROTECTED]
ITAGAKI Takahiro [EMAIL PROTECTED] writes:
PostgreSQL suppots SJIS, BIG5, GBK, UHC and GB18030 as client encodings,
but we cannot use them as server encodings. Are there any reason for it?
Very much so --- they aren't safe ASCII-supersets, and thus for example
the parser will fail on them. Backend encodings must have the property
that all bytes of a multibyte character are = 128.
Sorry. I still cannot understand why backend encodings must have this
property. AFAIK, the parser treats characters as ASCII. So any multi-byte
characters will be treated as two or more ASCII characters. But if
the multi-byte encoding doesnot use any special ASCII characters like
single quote('), double quote() and backslash(\), I think the parser
can deal with it correctly. A quick search in
src\backend\utils\mb\Unicode\*.map tells me that no encoding uses
single quote or double quote, but JOHAB, GBK, GB18030, BIG5, SJIS
use backslash. Since pgsql doesnot accept backslash as escape character
in identity(double quoted string) or value(single quoted string)
any more, I think the parser/scanner can process multi-bytes characters
correctly.
Thanks in advance.
William ZHANG
regards, tom lane
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
---(end of broadcast)---
TIP 6: explain analyze is your friend
Re: [Fwd: Re: [HACKERS] tsearch in core patch]
Tatsuo Ishii wrote:
japanese '{ja_JP, C}'
How would we know C - japanese?
You can't do that. You can't have different languages (not locales)
mapping to the same 'tsearch language' because the stemmer doesn't know
that a specific word is in english or japanese. So you have two options:
(a) disable stemming (b) leave the language set to 'japanese' and see if
it plays well.
Ok, probably we need to copy the English stemming rule to the one for
Japanese. I think same thing (commonly used English with local
language) can be applied to Chinese and Korean.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
Re: [HACKERS] tsearch in core patch
I would be surprised if C locale defaulted to anything except English. Don't be surprised. The mechanism of collation is too simple for Japanse Kanji, and locale is not usefull for Japanse anyway. That's why Japanese installations of PostgreSQL tend to use C locale. -- Tatsuo Ishii SRA OSS, Inc. Japan I suppose it would be sensible to add a switch to allow people to select a different language. In any case, the only thing initdb would be doing would be setting up an initial value of a table entry or GUC variable, so you could always change it yourself later; it may not be worth sweating too much about this. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Server-side support of all encodings
William ZHANG [EMAIL PROTECTED] writes:
Sorry. I still cannot understand why backend encodings must have this
property. AFAIK, the parser treats characters as ASCII. So any multi-byte
characters will be treated as two or more ASCII characters. But if
the multi-byte encoding doesnot use any special ASCII characters like
single quote('), double quote() and backslash(\), I think the parser
can deal with it correctly.
You've got your attention too narrowly focused on strings inside quotes;
it's strings outside quotes that are the problem.
As an example, I see that gb18030 defines characters like 97 7e.
If someone tried to use that as a character of a SQL identifier
--- something that'd work fine for the UTF8 equivalent e6 a2 a1
--- the parser would see it as an identifier byte followed by
the operator ~.
Similarly, there are problems if we were to allow these character sets
for the pattern argument of a regular expression operator, or for any
datatype at all that can be embedded in an array constant. And for PL
languages that feed prosrc strings into external interpreters, such as
Perl or R, it gets really interesting really quickly :-(.
It is possible that some of these encodings could be allowed without
any risks, but I don't think it is worth our time to grovel through
each valid character and every possible backend situation to determine
safety. The risks are not always obvious --- see for instance the
security holes we fixed about a year ago in 8.1.4 et al --- and so
I for one would never have a lot of faith in there not being any holes.
The rule no ASCII-aliasing characters is a simple one that we can have
some confidence in.
regards, tom lane
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Re: [Fwd: Re: [HACKERS] tsearch in core patch]
Tatsuo Ishii [EMAIL PROTECTED] writes: Ok, probably we need to copy the English stemming rule to the one for Japanese. Pardon my ignorance here, but is the concept of stemming even relevant to Japanese/Chinese/Korean? What little I know about ideographic languages suggests it wouldn't work well. And surely the specific rules in the Snowball project's English stemmer wouldn't work. I think same thing (commonly used English with local language) can be applied to Chinese and Korean. Well, it's not hard at all to find chunks of English text that have embedded bits of French, Spanish, or what-have-you, but that's not an argument for trying to intermix the stemmers. I doubt that such simple bits of program could tell the language difference well enough to determine which stemming rules to apply. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [Fwd: Re: [HACKERS] tsearch in core patch]
Tatsuo Ishii [EMAIL PROTECTED] writes: Ok, probably we need to copy the English stemming rule to the one for Japanese. Pardon my ignorance here, but is the concept of stemming even relevant to Japanese/Chinese/Korean? What little I know about ideographic languages suggests it wouldn't work well. And surely the specific rules in the Snowball project's English stemmer wouldn't work. Your undestanding is correct. English stemmer would not work for Japanese non English part. What I meant was the chunks of English text in Japanese. I think same thing (commonly used English with local language) can be applied to Chinese and Korean. Well, it's not hard at all to find chunks of English text that have embedded bits of French, Spanish, or what-have-you, but that's not an argument for trying to intermix the stemmers. I doubt that such simple bits of program could tell the language difference well enough to determine which stemming rules to apply. For Japanese, it will be fairly simple: 7bit ASCII range words must be English (Note that mostly used Japanese encodings such as EUC do not allow to mix with ISO 8859). -- Tatsuo Ishii SRA OSS, Inc. Japan ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
