Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Martijn van Oosterhout
On Tue, Sep 11, 2007 at 11:27:50AM +0900, Tatsuo Ishii wrote: SELECT * FROM japanese_table ORDER BY convert(japanese_text using utf8_to_euc_jp); Without using convert(), he will get random order of data. This is because Kanji characters are in random order in UTF-8, while Kanji characters

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Jeff Davis
On Tue, 2007-09-11 at 14:50 +0900, Tatsuo Ishii wrote: On Tue, 2007-09-11 at 12:29 +0900, Tatsuo Ishii wrote: Please show me concrete examples how I could introduce a vulnerability using this kind of convert() usage. Try the sequence below. Then, try to dump and then reload the

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread db
Try the sequence below. Then, try to dump and then reload the database. When you try to reload it, you will get an error: ERROR: invalid byte sequence for encoding UTF8: 0xbd I know this could be a problem (like chr() with invalid byte pattern). And that's enough of a problem already. We

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Tatsuo Ishii
On Tue, 2007-09-11 at 14:50 +0900, Tatsuo Ishii wrote: On Tue, 2007-09-11 at 12:29 +0900, Tatsuo Ishii wrote: Please show me concrete examples how I could introduce a vulnerability using this kind of convert() usage. Try the sequence below. Then, try to dump and then reload

Re: [HACKERS] Ts_rank internals

2007-09-11 Thread Oleg Bartunov
On Tue, 11 Sep 2007, Teodor Sigaev wrote: I tried to understand how ts_rank works, but I failed. What does Cover function do? How does it work? What is the DocRepresentation data structure like? I can see the definition of the struct, and the get_docrep function to convert to that format, but

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Albe Laurenz
Andrew Dunstan wrote: Instead of the code point, I'd prefer the actual encoding of the character as argument to chr() and return value of ascii(). And frankly, I don't know how to do it sanely anyway. A character encoding has a fixed byte pattern, but a given byte pattern doesn't have a

Re: [HACKERS] Per-function search_path = per-function GUC settings

2007-09-11 Thread Zdenek Kotala
Tom Lane wrote: I thought about ways to include GUC settings directly into CREATE FUNCTION, but it seemed pretty ugly and inconsistent with the existing syntax. So I'm thinking of supporting only the above syntaxes, meaning it'll take at least two commands to create a secure SECURITY DEFINER

[HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Simon Riggs
I've completed a review of all of the LWlocking in the backends. This is documented in the enclosed file. I would propose that we use this as comments in lwlock.h or in the README, if people agree. A number of points emerge from that analysis: 1. The ProcArrayLock is acquired Exclusive-ly by

Re: [HACKERS] CSStorm occurred again by postgreSQL8.2

2007-09-11 Thread Simon Riggs
On Wed, 2006-09-13 at 21:45 -0400, Tom Lane wrote: Anyway, given that there's this one nonobvious gotcha, there might be others. My recommendation is that we take this off the open-items list for 8.2 and revisit it in the 8.3 cycle when there's more time. Well, its still 8.3 just... As

Re: [HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Marko Kreen
On 9/11/07, Simon Riggs [EMAIL PROTECTED] wrote: 5. ReadNewTransactionId() is only called now by GetNextXidAndEpoch(), but I can't find a caller of that anywhere in core or contrib. Can those now be removed? GetNextXidAndEpoch() is needed for external modules to use 8-byte transaction ids

Re: [HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Simon Riggs
On Tue, 2007-09-11 at 13:31 +0300, Marko Kreen wrote: On 9/11/07, Simon Riggs [EMAIL PROTECTED] wrote: 5. ReadNewTransactionId() is only called now by GetNextXidAndEpoch(), but I can't find a caller of that anywhere in core or contrib. Can those now be removed? GetNextXidAndEpoch() is

Re: [HACKERS] pgcrypto related backend crash on solaris 10/x86_64

2007-09-11 Thread Zdenek Kotala
Marko Kreen wrote: On 9/9/07, Stefan Kaltenbrunner [EMAIL PROTECTED] wrote: I brought back clownfish(still a bit dubious about the unexplained failures which seem vmware emulation bugs but this one seems to be easily reproduceable) onto the buildfarm and enabled --with-openssl after the the

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
On Tue, 11 Sep 2007, Tom Lane wrote: dugong has been failing contribcheck repeatably for the last day or so, with a very interesting symptom: CREATE DATABASE is failing with The reason for that is that I've been trying to switch from 9.1 to 10.0 version of the ICC compiler. A month ago, I've

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: On Tue, 11 Sep 2007, Tom Lane wrote: dugong has been failing contribcheck repeatably for the last day or so, with a very interesting symptom: CREATE DATABASE is failing with The reason for that is that I've been trying to switch from 9.1 to 10.0

Re: [HACKERS] CSStorm occurred again by postgreSQL8.2

2007-09-11 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: As discussed in the other thread Final Thoughts for 8.3 on LWLocking and Scalability, XidCacheRemoveRunningXids() is now the only holder of an X lock during normal processing, Nonsense. Main transaction exit also takes an exclusive lock, and is far more

Re: [HACKERS] Per-function search_path = per-function GUC settings

2007-09-11 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: I have a question about what does happen if search path is not defined for SECURITY DEFINER function. My expectation is that SECURITY DEFINER function should defined empty search patch in this case. Your expectation is incorrect. We are not in the

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
This could be a compiler bug, or it could be our fault --- might need a volatile on some pointer or other, for example, to prevent the compiler from making an otherwise legitimate assumption. So it seems worth chasing it down. Tom, Thank you for the directions, I'll try to do what you

Re: [HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: 1. The ProcArrayLock is acquired Exclusive-ly by only one remaining operation: XidCacheRemoveRunningXids(). Reducing things to that level is brilliant work, Florian and Tom. It would be brilliant if it were true, but it isn't. Better look again.

Re: [HACKERS] pgcrypto related backend crash on solaris 10/x86_64

2007-09-11 Thread Marko Kreen
On 9/11/07, Zdenek Kotala [EMAIL PROTECTED] wrote: Marko Kreen wrote: This is crashing because of the crippled OpenSSL on some version of Solaris. Zdenek Kotala posted a workaround for that, I am cleaning it but have not found the time to finalize it. I'll try to post v03 of Zdenek's

[HACKERS] pg_dump and money type

2007-09-11 Thread Peter Eisentraut
Since the money type has a locale dependent input and output format, there has to be some context saved when a database dump is created. For example, if your environment uses a locale that uses the opposite point-vs-comma conventions from English (e.g., de_DE), then the following will fail to

Re: [HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Simon Riggs
On Tue, 2007-09-11 at 10:21 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: 1. The ProcArrayLock is acquired Exclusive-ly by only one remaining operation: XidCacheRemoveRunningXids(). Reducing things to that level is brilliant work, Florian and Tom. It would be brilliant if

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: BTW, does ICC have any switch corresponding to gcc's -fno-strict-aliasing? I see that configure tries to feed that switch to it, but it might want some other spelling. Apparently in none of the ICC manuals -fno-strict-aliasing is described, but

Re: [HACKERS] Per-function search_path = per-function GUC settings

2007-09-11 Thread Zdenek Kotala
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: I have a question about what does happen if search path is not defined for SECURITY DEFINER function. My expectation is that SECURITY DEFINER function should defined empty search patch in this case. Your expectation is incorrect. We

Re: [HACKERS] pgcrypto related backend crash on solaris 10/x86_64

2007-09-11 Thread Zdenek Kotala
Marko Kreen wrote: On 9/11/07, Zdenek Kotala [EMAIL PROTECTED] wrote: Marko Kreen wrote: This is crashing because of the crippled OpenSSL on some version of Solaris. Zdenek Kotala posted a workaround for that, I am cleaning it but have not found the time to finalize it. I'll try to post v03

Re: [HACKERS] CSStorm occurred again by postgreSQL8.2

2007-09-11 Thread Simon Riggs
On Tue, 2007-09-11 at 09:58 -0400, Tom Lane wrote: Can we let go of this for 8.3, please? OK, we've moved forward, so its a good place to break. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 1: if

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
Well, the first thing I'd suggest is trying to localize which Assert makes it fail. From the bug's behavior I think it is highly probable that the problem is in fsync signalling, which puts it either in bgwriter.c or md.c. Try recompiling those modules separately without cassert (leaving all

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
Actually, in the log file I also see some messages about has_seq_search: КОМАНДА: CREATE DATABASE contrib_regression TEMPLATE=template0 NOTICE: database contrib_regression does not exist, skipping ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: Actually, in the log file I also see some messages about has_seq_search: =EB=EF=ED=E1=EE=E4=E1: CREATE DATABASE contrib_regression TEMPLATE=3Dtem= plate0 NOTICE: database contrib_regression does not exist, skipping ERROR: too many active

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Jeff Davis
On Mon, 2007-09-10 at 23:20 -0400, Tom Lane wrote: The reason we have a problem here is that we've been choosing convenience over safety in encoding-related issues. I wonder if we must stoop to having a strict_encoding_checks GUC variable to satisfy everyone. That would be satisfactory to

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: Actually, in the log file I also see some messages about has_seq_search: ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans BTW, I just made a commit to include

Re: [HACKERS] Per-function search_path = per-function GUC settings

2007-09-11 Thread Peter Eisentraut
Am Dienstag, 11. September 2007 15:53 schrieb Tom Lane: Zdenek Kotala [EMAIL PROTECTED] writes: I have a question about what does happen if search path is not defined for SECURITY DEFINER function. My expectation is that SECURITY DEFINER function should defined empty search patch in this

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Teodor Sigaev
It seems to me last run (http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=dugongdt=2007-09-11%2016:05:01) points to problem with hash implementation. *** ./expected/tsdicts.out Tue Sep 11 20:05:23 2007 --- ./results/tsdicts.out Tue Sep 11 20:18:38 2007 *** *** 301,306

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Teodor Sigaev [EMAIL PROTECTED] writes: It seems to me last run (http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=dugongdt=2007-09-11%2016:05:01) points to problem with hash implementation. dynahash.c is used all over the system, though. If it were broken by a compiler issue, it's hard to

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
On Tue, 11 Sep 2007, Tom Lane wrote: NOTICE: database contrib_regression does not exist, skipping ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans With

Re: [HACKERS] Per-function search_path = per-function GUC settings

2007-09-11 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Well, a SECURITY DEFINER function either sets its own search path, in which case a default search path would have no effect, or it doesn't set its own search path, in which case it's already broken (albeit in a different way). So setting a default

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: On Tue, 11 Sep 2007, Tom Lane wrote: NOTICE: database contrib_regression does not exist, skipping ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans ERROR: too

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
On Tue, 11 Sep 2007, Tom Lane wrote: Sergey E. Koposov [EMAIL PROTECTED] writes: On Tue, 11 Sep 2007, Tom Lane wrote: NOTICE: database contrib_regression does not exist, skipping ERROR: too many active hash_seq_search scans ERROR: too many active hash_seq_search scans ERROR: too many

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: Yes, indeed. After several make installcheck's I get ERROR: too many active hash_seq_search scans, cannot start one on smgr relation table ERROR: too many active hash_seq_search scans, cannot start one on smgr relation table Hm, so that must

Re: [HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Florian G. Pflug
Simon Riggs wrote: On Tue, 2007-09-11 at 10:21 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: 1. The ProcArrayLock is acquired Exclusive-ly by only one remaining operation: XidCacheRemoveRunningXids(). Reducing things to that level is brilliant work, Florian and Tom. It would

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Tom Lane
Jeff Davis [EMAIL PROTECTED] writes: On Mon, 2007-09-10 at 23:20 -0400, Tom Lane wrote: It might work the way you are expecting if the database uses SQL_ASCII encoding and C locale --- and I'd be fine with allowing convert() only when the database encoding is SQL_ASCII. I prefer this option.

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Jeff Davis
On Tue, 2007-09-11 at 14:48 -0400, Tom Lane wrote: Jeff Davis [EMAIL PROTECTED] writes: On Mon, 2007-09-10 at 23:20 -0400, Tom Lane wrote: It might work the way you are expecting if the database uses SQL_ASCII encoding and C locale --- and I'd be fine with allowing convert() only when the

Re: [HACKERS] Final Thoughts for 8.3 on LWLocking and Scalability

2007-09-11 Thread Simon Riggs
On Tue, 2007-09-11 at 19:32 +0200, Florian G. Pflug wrote: Simon Riggs wrote: On Tue, 2007-09-11 at 10:21 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: 1. The ProcArrayLock is acquired Exclusive-ly by only one remaining operation: XidCacheRemoveRunningXids(). Reducing

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Alvaro Herrera
Tom Lane wrote: Jeff Davis [EMAIL PROTECTED] writes: On Mon, 2007-09-10 at 23:20 -0400, Tom Lane wrote: It might work the way you are expecting if the database uses SQL_ASCII encoding and C locale --- and I'd be fine with allowing convert() only when the database encoding is SQL_ASCII.

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Tom Lane wrote: I think really the technically cleanest solution would be to make convert() return bytea instead of text; then we'd not have to put restrictions on what encoding or locale it's working inside of. However, it's not clear to me whether

Re: [HACKERS] pg_dump and money type

2007-09-11 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Since the money type has a locale dependent input and output format, there has to be some context saved when a database dump is created. For example, if your environment uses a locale that uses the opposite point-vs-comma conventions from

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Tom Lane
Tatsuo Ishii [EMAIL PROTECTED] writes: If we make convert() operate on bytea and return bytea, as Tom suggested, would that solve your use case? The problem is, the above use case is just one of what I can think of. Another use case is, something like this: SELECT

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Andrew Dunstan
Sergey E. Koposov wrote: With hash_seq_search ERROR, it was partially a false alarm. I've had some old postgres daemon hanging around and writing that to the log. Although I remember seeing that hash_seq_search message recently when dealing with this bug, it does not show up in the course

Re: [HACKERS] pg_dump and money type

2007-09-11 Thread Heikki Linnakangas
Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Since the money type has a locale dependent input and output format, there has to be some context saved when a database dump is created. For example, if your environment uses a locale that uses the opposite point-vs-comma

Re: [HACKERS] What is happening on buildfarm member dugong

2007-09-11 Thread Sergey E. Koposov
On Tue, 11 Sep 2007, Andrew Dunstan wrote: Your buildfarm member must be seriously misconfigured if you can get the logs from different postmasters comingled. Every run gets its own logfile in its own inst directory. No, everything I'm doing now about that bug, I'm doing in the very

[HACKERS] Preparation for PostgreSQL releases 8.2.5, 8.1.10, 8.0.14, 7.4.18, 7.3.20

2007-09-11 Thread Bruce Momjian
Preparations are being made for PostgreSQL releases 8.2.5, 8.1.10, 8.0.14, 7.4.18, 7.3.20. The CVS branches are nearly ready. The releases will happen sometime early next week. The packagers have been contacted. -- Bruce Momjian [EMAIL PROTECTED] http://momjian.us EnterpriseDB

Re: [HACKERS] Preparation for PostgreSQL releases 8.2.5, 8.1.10, 8.0.14, 7.4.18, 7.3.20

2007-09-11 Thread Alvaro Herrera
Bruce Momjian wrote: Preparations are being made for PostgreSQL releases 8.2.5, 8.1.10, 8.0.14, 7.4.18, 7.3.20. The CVS branches are nearly ready. The releases will happen sometime early next week. The packagers have been contacted. Does this mean that if I commit something in these days

Re: [HACKERS] Preparation for PostgreSQL releases 8.2.5, 8.1.10, 8.0.14, 7.4.18, 7.3.20

2007-09-11 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Preparations are being made for PostgreSQL releases 8.2.5, 8.1.10, 8.0.14, 7.4.18, 7.3.20. The CVS branches are nearly ready. The releases will happen sometime early next week. The packagers have been contacted. Does this mean that if I

[HACKERS] HOT breaks CLUSTER, a bit

2007-09-11 Thread Tom Lane
While editing Bruce's README.HOT documentation I ran into a problem. I had idly added the following text after thinking about the different sorts of snapshots we use: --- Another unpleasant consequence is that it is no longer very meaningful to use SnapshotAny in an index scan: if the index was

Re: [HACKERS] pg_dump and money type

2007-09-11 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: Tom Lane wrote: This risk seems rather overstated, as it's unlikely that someone using money would choose to reload their data into a DB with a fundamentally incompatible locale setting. It doesn't sound unlikely at all to me. For example,

Re: [HACKERS] pg_dump and money type

2007-09-11 Thread Alvaro Herrera
Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: Tom Lane wrote: This risk seems rather overstated, as it's unlikely that someone using money would choose to reload their data into a DB with a fundamentally incompatible locale setting. It doesn't sound unlikely at all to

[HACKERS] CVS HEAD is broken by flex

2007-09-11 Thread ITAGAKI Takahiro
I found CVS HEAD is broken and the following error is reported. My environment is RHEL4 Update 2 x86_64 with gcc 3.4.4 and flex 2.5.33. In file included from ../../../src/include/port.h:17, from ../../../src/include/c.h:841, from

[HACKERS] buildfarm failure after ICC configure change

2007-09-11 Thread Jeremy Drake
I just saw that my buildfarm member (running ICC 9.0 on linux) failed after the latest change to configure http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=mongoosedt=2007-09-11%2020:45:01 I was the one who sent in the first patch to configure to add the check for ICC, and as I recall at the

[HACKERS] Add function for quote_qualified_identifier?

2007-09-11 Thread Brendan Jurd
Hi hackers, I note that we currently expose the usefulness of the quote_identifier function to the user with quote_ident(text). Is there any reason we shouldn't do the same with quote_qualified_identifier? We could just add a quote_qualified_ident(text, text) ... it would make forming dynamic

Re: [HACKERS] Per-function search_path = per-function GUC settings

2007-09-11 Thread Brendan Jurd
On 9/12/07, Tom Lane [EMAIL PROTECTED] wrote: It would break functions that actually want to use a caller-specified search path, and protect themselves by explicitly schema-qualifying every other reference than one to some caller-specified object. Which admittedly is notationally a pain in

Re: [HACKERS] invalidly encoded strings

2007-09-11 Thread Tatsuo Ishii
However ISTM we would also need something like length(bytea, name) returns int -- counts the number of characters assuming that the bytea is in -- the given encoding Hmm, I wonder if counting chars is consistent regardless of the encoding the string is in. To me it sounds

Re: [HACKERS] CVS HEAD is broken by flex

2007-09-11 Thread Tom Lane
ITAGAKI Takahiro [EMAIL PROTECTED] writes: The suspect is flex (or yy template file). It defines __unused to be empty, but netdb.h uses __unused as a variable. Egad. Shouldn't there be villagers breaking down the flex hackers' front door with torches and pitchforks in hand? Why is it our

Re: [HACKERS] buildfarm failure after ICC configure change

2007-09-11 Thread Tom Lane
Jeremy Drake [EMAIL PROTECTED] writes: I just saw that my buildfarm member (running ICC 9.0 on linux) failed after the latest change to configure Argh! Can someone quote chapter and verse from the ICC manual about this? I was just following what Sergey said was the approved spelling of the

Re: [HACKERS] buildfarm failure after ICC configure change

2007-09-11 Thread Jeremy Drake
On Wed, 12 Sep 2007, Tom Lane wrote: Jeremy Drake [EMAIL PROTECTED] writes: I just saw that my buildfarm member (running ICC 9.0 on linux) failed after the latest change to configure Argh! Can someone quote chapter and verse from the ICC manual about this? I was just following what