Re: [HACKERS] Reducing contention for the LockMgrLock

2005-12-09 Thread Simon Riggs
On Thu, 2005-12-08 at 09:44 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: The output you gave wasn't anything I recognize in the code. Assuming its not already there, please can you share code you are using to find the evidence, even if its just privately in some form? See

[HACKERS] psql patch: new host/port

2005-12-09 Thread David Fetter
Folks, Please find enclosed a patch that lets you use \c to connect (optionally) to a new host and port without exiting psql. Cheers, D -- David Fetter [EMAIL PROTECTED] http://fetter.org/ phone: +1 415 235 3778 Remember to vote! Index: doc/src/sgml/ref/psql-ref.sgml

[HACKERS] psql UTF8

2005-12-09 Thread Teodor Sigaev
Working on UTF support in tsearch2 I noticed different output of psql: 1) initdb with SQL_ASCII: contrib_regression=# select 'jisa n asd n asld kl ansd contrib_regression'# asdfjkl; asdf kl;amsdf'; ?column?

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Simon Riggs
This technique and others are discussed in detail in the Intel Optimization Manual: http://apps.intel.com/scripts-util/download.asp?url=/design/PentiumII/manuals/24512701.pdftitle=Intel%AE+Architecture+Optimization+Reference+Manualfullpg=3site=Developer Similar manual exists for AMD and other

Re: [HACKERS] Replication on the backend

2005-12-09 Thread Christopher Browne
Are you sure that no way to implement a generic aproach on the backend? What does specification say? What specification are you talking about? Does Oracle 10g have a core implementation of replication (cluster)? Since replication is sold as a separate product from Oracle 10g, obviously not.

Re: [HACKERS] Reducing contention for the LockMgrLock

2005-12-09 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2005-12-08 at 09:44 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: You're looking at the number of spins to acquire each lock? Number of semop waits. I wonder whether that is the thing to measure. That measure doesn't show how

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Bruce Momjian
Do these optimizations have any affect on database software? I know call overhead shows up as a performance bottleneck, but do these others optimizations also measurably improve performance? --- Simon Riggs wrote: This

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Bruce Momjian
Kenneth Marshall wrote: The main benefit of pre-fetching optimization is to allow just- in-time data delivery to the processor. There are numerous papers illustrating the dramatic increase in data throughput by using datastructures designed to take advantage of prefetching. Factors of 3-7 can

Re: [HACKERS] Upcoming PG re-releases

2005-12-09 Thread Martijn van Oosterhout
On Thu, Dec 08, 2005 at 05:54:35PM -0500, Gregory Maxwell wrote: No, what is needed for people who care about fixing their data is a loadable strip_invalid_utf8() that works in older versions.. then just select * from bar where foo != strip_invalid_utf8(foo); The function would be useful in

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: I can see that being useful for a single-user application that doesn't have locking or I/O bottlenecks, and doesn't have a multi-stage design like a database. Do we do enough of such processing that we will _see_ an improvement, or will our code

Re: [HACKERS] Foreign key trigger timing bug?

2005-12-09 Thread Jan Wieck
On 12/8/2005 8:53 PM, Tom Lane wrote: Stephan Szabo [EMAIL PROTECTED] writes: Yeah. I really don't understand it, but it appears to me to be explicitly different in the spec for on delete cascade even compared to the rest of the referential actions. One problem I see is, what do we do if

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Kenneth Marshall
On Fri, Dec 09, 2005 at 10:37:25AM -0500, Bruce Momjian wrote: Kenneth Marshall wrote: The main benefit of pre-fetching optimization is to allow just- in-time data delivery to the processor. There are numerous papers illustrating the dramatic increase in data throughput by using

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Kenneth Marshall
The main benefit of pre-fetching optimization is to allow just- in-time data delivery to the processor. There are numerous papers illustrating the dramatic increase in data throughput by using datastructures designed to take advantage of prefetching. Factors of 3-7 can be realized and this can

[HACKERS] Log of CREATE USER statement

2005-12-09 Thread Ricardo Vaz - TCESP
Title: Log of CREATE USER statement Hi, I need to log any statement in my postgresql server. However, I would like that the password defined in CREATE USER statement was registered in MD5 format, independent of the form as it was specified in that statement. For example, if the user

Re: [HACKERS] int to inet conversion [or Re: inet to bigint?]

2005-12-09 Thread Kai
Hello All, I've been pondering the discussed subject a few times, and came along a few things that I think are missing from the default set of typeconversions within postgres. After working regularly with inet values in sql, it would be nice to be able to do this: = select

[HACKERS] Backslashes in string literals

2005-12-09 Thread Kevin Grittner
I've just been bitten by the backslash in string literals issue. I have reviewed the mailing lists and the TODO list. I see that the direction PostgreSQL is headed is to drop the nonstandard escapes, unless an extended literal is explicitly used. I've attached a patch which supports this as a

Re: [HACKERS] psql UTF8

2005-12-09 Thread Martijn van Oosterhout
On Fri, Dec 09, 2005 at 01:39:50PM +0300, Teodor Sigaev wrote: Working on UTF support in tsearch2 I noticed different output of psql: snip Length of '' string is different. This difference is a reason for installcheck failture of tsearch2 on UTF8 inited db. Is it supposed

[HACKERS] Min Xid problem proposal

2005-12-09 Thread Alvaro Herrera
Anybody remembers my patch to allow tracking the minimum Xid present in a table, allowing to update the freeze xid on a per-table basis? The motivation behind it was to remove the requirement of database-wide vacuums. The problem I found with it was that it required all tables to be vacuumed at

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: I can see that being useful for a single-user application that doesn't have locking or I/O bottlenecks, and doesn't have a multi-stage design like a database. Do we do enough of such processing that we will _see_ an

Re: [HACKERS] Upcoming PG re-releases

2005-12-09 Thread Bruce Momjian
Martijn van Oosterhout wrote: -- Start of PGP signed section. On Thu, Dec 08, 2005 at 05:54:35PM -0500, Gregory Maxwell wrote: No, what is needed for people who care about fixing their data is a loadable strip_invalid_utf8() that works in older versions.. then just select * from bar where

Re: [HACKERS] int to inet conversion [or Re: inet to bigint?]

2005-12-09 Thread Andrew Dunstan
How do you intend to handle the mask and family parts of the object in converting it to an int, not to mention the ipv6 difficulties you mention? A better way might be to add some extra functions, ISTM. cheers andrew Kai wrote: Hello All, I've been pondering the discussed subject a few

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Tom Lane
Ricardo Vaz - TCESP [EMAIL PROTECTED] writes: However, I would like that the password defined in CREATE USER statement was registered in MD5 format, independent of the form as it was specified in that statement. Not going to happen --- we are not going to add that sort of analysis to the

Re: [HACKERS] psql UTF8

2005-12-09 Thread Andrew Dunstan
The regression tests are known to fail on non-SQL_ASCII encodings for this and similar reasons. cheers andrew Teodor Sigaev wrote: Working on UTF support in tsearch2 I noticed different output of psql: 1) initdb with SQL_ASCII: contrib_regression=# select 'jisa n asd n asld kl ans

Re: [HACKERS] Upcoming PG re-releases

2005-12-09 Thread Martijn van Oosterhout
On Fri, Dec 09, 2005 at 11:34:22AM -0500, Bruce Momjian wrote: I think the problem with any kind of function-call detection is that the data has to get into the database first, and it isn't clear how someone loading a failed dump would do that aside from modifying the column to bytea in the

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Simon Riggs
On Fri, 2005-12-09 at 09:43 -0500, Bruce Momjian wrote: Do these optimizations have any affect on database software? I know call overhead shows up as a performance bottleneck, but do these others optimizations also measurably improve performance? Many of them can, but nowhere near as much

Re: [HACKERS] Backslashes in string literals

2005-12-09 Thread Bruce Momjian
I think we we will be turning on escape_string_warning in 8.2 and allow standard_conforming_strings to be optionally turned on in that releaes. I will keep the patch for us in completing that item. This has been saved for the 8.2 release:

Re: [HACKERS] Backslashes in string literals

2005-12-09 Thread Peter Eisentraut
Kevin Grittner wrote: direction PostgreSQL is headed is to drop the nonstandard escapes, unless an extended literal is explicitly used. I've attached a patch which supports this as a configure option, using a --enable-standard-strings switch. There is already a run-time configuration option

Re: [HACKERS] Backslashes in string literals

2005-12-09 Thread Bruce Momjian
Peter Eisentraut wrote: Kevin Grittner wrote: direction PostgreSQL is headed is to drop the nonstandard escapes, unless an extended literal is explicitly used. I've attached a patch which supports this as a configure option, using a --enable-standard-strings switch. There is already a

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Bruce Momjian
Tom Lane wrote: Ricardo Vaz - TCESP [EMAIL PROTECTED] writes: However, I would like that the password defined in CREATE USER statement was registered in MD5 format, independent of the form as it was specified in that statement. Not going to happen --- we are not going to add that sort of

Re: [HACKERS] Min Xid problem proposal

2005-12-09 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: My proposal to solve that problem, is to make any transaction that inserts or modifies tuples in a table that is marked as frozen, unfreeze it first. The problem I had last time was finding a good spot in the code for doing so. I'm now proposing to do

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Peter Eisentraut
Tom Lane wrote: However, I would like that the password defined in CREATE USER statement was registered in MD5 format, independent of the form as it was specified in that statement. Not going to happen --- we are not going to add that sort of analysis to the statement logging code, as the

Re: [HACKERS] Upcoming PG re-releases

2005-12-09 Thread Bruce Momjian
Martijn van Oosterhout wrote: -- Start of PGP signed section. On Fri, Dec 09, 2005 at 11:34:22AM -0500, Bruce Momjian wrote: I think the problem with any kind of function-call detection is that the data has to get into the database first, and it isn't clear how someone loading a failed dump

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: One issue I have not heard is that CREATE USER, with the visible password, is sent over the wire in cleartext, and does appear in the logs, as we discussed, so while we MD5 the password in pg_shadow so administrators do not see it, we do log the

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Users who choose a password should have the assurance that the password cannot be seen in plain-text by anyone anywhere. In a PostgreSQL system, the password can be seen in all kinds of places, like the psql history, the server log, the activity

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Bruce Momjian
Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Users who choose a password should have the assurance that the password cannot be seen in plain-text by anyone anywhere. In a PostgreSQL system, the password can be seen in all kinds of places, like the psql history, the

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Joshua D. Drake
On Fri, 2005-12-09 at 13:03 -0500, Bruce Momjian wrote: Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Users who choose a password should have the assurance that the password cannot be seen in plain-text by anyone anywhere. In a PostgreSQL system, the password can be

Re: [HACKERS] Upcoming PG re-releases

2005-12-09 Thread Martijn van Oosterhout
On Fri, Dec 09, 2005 at 12:38:21PM -0500, Bruce Momjian wrote: This means someone who is planning on upgrading to 8.1 in two months can use this function now to weed out the bad data before the upgrade even starts. Oh, so you back-load it into the old database. Interesting. I assume to

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Peter Eisentraut
Tom Lane wrote: To put that more clearly: if the point is to keep the user's cleartext password out of the hands of the DBA, then the user has already blown it by sending the password in cleartext in the first place. An untrustworthy DBA could trivially insert code into CREATE USER to log

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Peter Eisentraut
Tom Lane wrote: As I said already, if the user wishes the password to be secure, he needs to encrypt it on the client side. Maybe we should provide a backslash command in psql for secure password entry, say, \password [username]. This would then ask for the password through a somewhat

Re: [HACKERS] Min Xid problem proposal

2005-12-09 Thread Alvaro Herrera
Tom Lane wrote: I don't see any very good argument for allowing this mechanism to set minxid = FrozenXid in the first place. If there are only frozenXid in the table, set minxid = current XID. That eliminates the entire problem at a stroke. Ok, so I shall go back to the original patch,

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: As I said already, if the user wishes the password to be secure, he needs to encrypt it on the client side. Anything else is just the illusion of security. Thinking some more, does encoding on the client side really improve

Re: [HACKERS] Min Xid problem proposal

2005-12-09 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Ok, so I shall go back to the original patch, which did exactly this. Is it OK for applying? I haven't looked at it ... when did you post it exactly? (I'm using RecentXmin instead of current XID though, because a currently-running transaction could

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Bruce Momjian
Peter Eisentraut wrote: Tom Lane wrote: As I said already, if the user wishes the password to be secure, he needs to encrypt it on the client side. Maybe we should provide a backslash command in psql for secure password entry, say, \password [username]. This would then ask for the

Re: [HACKERS] Min Xid problem proposal

2005-12-09 Thread Alvaro Herrera
Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Ok, so I shall go back to the original patch, which did exactly this. Is it OK for applying? I haven't looked at it ... when did you post it exactly? From: Alvaro Herrera [EMAIL PROTECTED] To: Tom Lane [EMAIL PROTECTED] Cc: Patches

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Qingqing Zhou
On Thu, 8 Dec 2005, Min Xu (Hsu) wrote: Perhaps because P4 is already doing H/W prefetching? http://www.tomshardware.com/2000/11/20/intel/page5.html I ran the test program on an opteron 2.2G: Got slowdown. I ran it on a PIII 650M: looks like some speedup to me. Ok, I see ... so this

Re: [HACKERS] Min Xid problem proposal

2005-12-09 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: I have a version that applies cleanly to current CVS tip. Do I post it again? No need unless you think the changes are significant. I'll try to look over the patch soon. regards, tom lane ---(end of

Re: [HACKERS] Log of CREATE USER statement

2005-12-09 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Maybe we should provide a backslash command in psql for secure password entry, say, \password [username]. This would then ask for the password through a somewhat secure, unlogged channel, encrypt it, and send an ALTER ROLE command to the server.

Re: [HACKERS] Foreign key trigger timing bug?

2005-12-09 Thread Bruce Momjian
Stephan Szabo wrote: On Wed, 7 Dec 2005, Bruce Momjian wrote: I had an open 8.1 item that was: o fix foreign trigger timing issue Would someone supply text for a TODO entry on this, as I don't think we fixed it in 8.1. I'd split this into two separate items now. Fix

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Luke Lonergan
Bruce, It (the compute intensity optimization) is what we did for copy parsing, and it sped up by a factor of 100+. The rest of the copy path could use some work too. Yge virtual tuples in 8.1 are another example of grouping operations into more compact chunks instead of doing bits at a time.

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Bruce Momjian
Luke Lonergan wrote: Bruce, It (the compute intensity optimization) is what we did for copy parsing, and it sped up by a factor of 100+. The rest of the copy path could use some work too. Yge virtual tuples in 8.1 are another example of grouping operations into more compact chunks

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Andrew Dunstan
Luke Lonergan wrote: Bruce, It (the compute intensity optimization) is what we did for copy parsing, and it sped up by a factor of 100+. The changes made to COPY were portable, though. cheers andrew ---(end of broadcast)--- TIP 5:

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Tom Lane
Andrew Dunstan [EMAIL PROTECTED] writes: Luke Lonergan wrote: It (the compute intensity optimization) is what we did for copy parsing, and it sped up by a factor of 100+. The changes made to COPY were portable, though. In fact, the changes made to COPY had absolutely nada to do with any of

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Luke Lonergan
Tom, On 12/9/05 2:14 PM, Tom Lane [EMAIL PROTECTED] wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Luke Lonergan wrote: It (the compute intensity optimization) is what we did for copy parsing, and it sped up by a factor of 100+. The changes made to COPY were portable, though. In fact,

Re: [HACKERS] [COMMITTERS] pgsql: Translation typo fix

2005-12-09 Thread Peter Eisentraut
Alvaro Herrera wrote: Peter Eisentraut wrote: Am Dienstag, 15. November 2005 00:01 schrieb Alvaro Herrera: Good question. Peter, is pgtranslation supposed to be the primary source of translations? Yes, and more importantly, your changes will get overwritten by pgtranslation. Ok,

[HACKERS] gcc complains about %x

2005-12-09 Thread Tom Lane
CVS tip is giving me this warning: initdb.c: In function `locale_date_order': initdb.c:2101: warning: `%x' yields only last 2 digits of year in some locales I realize that the code does not care about that, but I care about not seeing warnings. Is there a way to avoid this construct?

Re: [HACKERS] gcc complains about %x

2005-12-09 Thread Peter Eisentraut
Tom Lane wrote: CVS tip is giving me this warning: initdb.c: In function `locale_date_order': initdb.c:2101: warning: `%x' yields only last 2 digits of year in some locales This is fixed in GCC 3.4 (released April 2004) and later. -- Peter Eisentraut

Re: [HACKERS] Foreign key trigger timing bug?

2005-12-09 Thread Stephan Szabo
On Fri, 9 Dec 2005, Jan Wieck wrote: On 12/8/2005 8:53 PM, Tom Lane wrote: Stephan Szabo [EMAIL PROTECTED] writes: Yeah. I really don't understand it, but it appears to me to be explicitly different in the spec for on delete cascade even compared to the rest of the referential

Re: [HACKERS] int to inet conversion [or Re: inet to bigint?]

2005-12-09 Thread Christopher Kings-Lynne
Hi Kai, There are some rather simplistic functions to convert 32bit inet values to and from bigints in the mysql compatibility project: http://pgfoundry.org/projects/mysqlcompat/ In the miscellaneous.sql. Chris Kai wrote: Hello All, I've been pondering the discussed subject a few times,