Re: [HACKERS] PG Killed by OOM Condition

2005-10-04 Thread Jeff Davis
It's not an easy decision. Linux isn't wrong. Solaris isn't wrong. Most people never hit these problems, and the people that do, are just as likely to hit one problem, or the other. The grass is always greener on the side of the fence that isn't hurting me right now, and all that. Cheers,

Re: [HACKERS] PG Killed by OOM Condition

2005-10-04 Thread Dennis Bjorklund
On Mon, 3 Oct 2005, Jeff Davis wrote: involved, but I could be wrong. Is it possible to be hit by the OOM killer if no applications use fork()? Sure, whenever the system is out of mem and the os can't find a free page then it kills a process. If you check the kernel log you can see if the

Re: [HACKERS] Tuning current tuplesort external sort code for 8.2

2005-10-04 Thread Simon Riggs
On Tue, 2005-10-04 at 00:21 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: AFAICS the following opportunities exist, without changing any of the theoretical algorithms or the flexibility of definable datatypes: 1. tuplesort_heap_siftup and tuplesort_heap_insert make no

Re: [HACKERS] PG Killed by OOM Condition

2005-10-04 Thread Martijn van Oosterhout
On Mon, Oct 03, 2005 at 11:47:57PM -0700, Jeff Davis wrote: I think that I've run into the OOM killer without a fork() being involved, but I could be wrong. Is it possible to be hit by the OOM killer if no applications use fork()? fork() is the obvious overcomitter. If Netscape wants to spawn

Re: [HACKERS] Tuning current tuplesort external sort code for 8.2

2005-10-04 Thread Martijn van Oosterhout
On Tue, Oct 04, 2005 at 12:37:51AM +0100, Simon Riggs wrote: On Mon, 2005-10-03 at 23:25 +0200, Martijn van Oosterhout wrote: Please note: if inlineApplySortFunction() is actually inlined (it isn't by default) Can you explain your last post some more. Thats not what I get. The inline

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Martijn van Oosterhout
On Mon, Oct 03, 2005 at 10:51:32PM +0100, Simon Riggs wrote: Basically, I recommend adding -Winline -finline-limit-1500 to the default build while we discuss other options. I add -Winline but get no warnings. Why would I use -finline-limit-1500? I'm interested, but uncertain as to what

Re: [HACKERS] Vacuum and Transactions

2005-10-04 Thread Simon Riggs
On Tue, 2005-10-04 at 00:26 -0400, Rod Taylor wrote: As I understand it vacuum operates outside of the regular transaction and if you stop it (SIGTERM, or pg_cancel_backend()) some of the work it accomplished will be kept when it rolls back. For large structures with a ton of dead entries

Re: [HACKERS] Tuning current tuplesort external sort code for 8.2

2005-10-04 Thread Martijn van Oosterhout
On Tue, Oct 04, 2005 at 11:55:58AM +0200, Martijn van Oosterhout wrote: It goes in to say that the limit is 1 for gcc 2.95, but if you examine the manual for gcc 3.3 it has the limit at 600. So it's entirely possible that at the time the person wrote that code, it *was* being inlined, but

[HACKERS] New pg_config behavior

2005-10-04 Thread Peter Eisentraut
I have some doubts about the new pg_config behavior of printing not recorded when an item is not known. I doubt that this will cause automated users of this program to behave reasonably in error scenarios. I think a more reasonable behavior would be to use a non-zero exit status and possibly

[HACKERS] memory bug debugging

2005-10-04 Thread Markus Schiltknecht
Hello hackers, I'm fiddling around with the backend and have created a memory bug. I guess I'm overriding a palloced chunk, but can't figure out where. The backend's report is: WARNING: problem in alloc set MessageContext: req size alloc size for chunk 0x8383ca8 in block 0x83834a0

Re: [HACKERS] Tuning current tuplesort external sort code for 8.2

2005-10-04 Thread Simon Riggs
On Tue, 2005-10-04 at 12:37 +0200, Martijn van Oosterhout wrote: Since we're pretty much relying on gcc to inline for performance, I still think we should add -Winline by default so we can tell when it's not doing what we want. Very much agree to this. Can we put that in the build for 8.1

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Simon Riggs
On Tue, 2005-10-04 at 12:04 +0200, Martijn van Oosterhout wrote: On Mon, Oct 03, 2005 at 10:51:32PM +0100, Simon Riggs wrote: Basically, I recommend adding -Winline -finline-limit-1500 to the default build while we discuss other options. I add -Winline but get no warnings. Why would I

Re: [HACKERS] memory bug debugging

2005-10-04 Thread Martijn van Oosterhout
On Tue, Oct 04, 2005 at 01:11:41PM +0200, Markus Schiltknecht wrote: Hello hackers, I'm fiddling around with the backend and have created a memory bug. I guess I'm overriding a palloced chunk, but can't figure out where. There are some defines (MEMORY_CONTEXT_CHECKING and

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Martijn van Oosterhout
On Tue, Oct 04, 2005 at 12:24:54PM +0100, Simon Riggs wrote: How did you determine the 1500 figure? Can you give some more info to surround that recommendation to allow everybody to evaluate it? [EMAIL PROTECTED]:~/dl/cvs/pgsql-local/src/backend/utils/sort$ gcc -finline-limit-1000 -Winline -O2

Re: [HACKERS] New Point Releases Available

2005-10-04 Thread Dave Page
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Marc G. Fournier Sent: 04 October 2005 02:34 To: pgsql-hackers@postgresql.org Subject: [HACKERS] New Point Releases Available Just bundled up 7.3.11, 7.4.9 and 8.0.4 ... please look them

Re: [HACKERS] New pg_config behavior

2005-10-04 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: I have some doubts about the new pg_config behavior of printing not recorded when an item is not known. I doubt that this will cause automated users of this program to behave reasonably in error scenarios. I think a more reasonable behavior

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Ron Peacetree
The constants related to inlining involve pcode, not actual assembly instructions, and are compiler version dependent as well as subject to change without notice by the GNU folks... from: http://gcc.gnu.org/onlinedocs/gcc-3.3.5/gcc/Optimize-Options.html#Optimize-Options -finline-limit=n By

Re: [HACKERS] memory bug debugging

2005-10-04 Thread Tom Lane
Martijn van Oosterhout kleptog@svana.org writes: On Tue, Oct 04, 2005 at 01:11:41PM +0200, Markus Schiltknecht wrote: I'm fiddling around with the backend and have created a memory bug. I guess I'm overriding a palloced chunk, but can't figure out where. There are some defines

Re: [HACKERS] \d on database with a lot of tables is slow

2005-10-04 Thread Jim C. Nasby
On Sat, Oct 01, 2005 at 02:00:12PM -0400, Tom Lane wrote: I wrote: It's presumably mostly in the pg_table_is_visible() calls. I did some profiling on a test case with 1 tables, and noticed that a big part of the problem is that the catalog caches become entirely useless: almost every

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Tom Lane
Martijn van Oosterhout kleptog@svana.org writes: A quick binary search puts the cutoff between 1200 and 1300. Given version variation I picked a nice round number, 1500. Ugh, that's for -O2, for -O3 and above it needs to be 4100 to work. Maybe we should go for 5000 or so. I'm using: gcc

Re: [HACKERS] pg_dump versioning

2005-10-04 Thread Jim C. Nasby
On Mon, Oct 03, 2005 at 12:11:49AM -0400, Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Jim C. Nasby wrote: Watching the discussion about how to handle nextval() and keep it dump compatible makes me wonder: would it be useful to encode database or dump version info into

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Simon Riggs
On Tue, 2005-10-04 at 16:30 +0200, Martijn van Oosterhout wrote: On Tue, Oct 04, 2005 at 10:06:24AM -0400, Tom Lane wrote: Martijn van Oosterhout kleptog@svana.org writes: I'm using: gcc (GCC) 3.3.5 (Debian 1:3.3.5-13) I don't know what the units of this number are, but it's apparently

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Tom Lane
Martijn van Oosterhout kleptog@svana.org writes: 1. Add -Winline so we can at least be aware of when it's (not) happening. Yeah, I agree with that part, just not with adding a fixed -finline-limit value. While on the subject of gcc warnings ... if I touch that code, I want to remove

Re: [HACKERS] effective SELECT from child tables

2005-10-04 Thread Jim C. Nasby
Is there enough for a TODO here? On Mon, Oct 03, 2005 at 11:24:30PM -0400, Greg Stark wrote: Hannu Krosing [EMAIL PROTECTED] writes: On P, 2005-10-02 at 23:00 -0400, Tom Lane wrote: Here's another interesting case to think about: ALTER TABLE ADD foo integer DEFAULT 1

Re: [HACKERS] Vacuum and Transactions

2005-10-04 Thread Rod Taylor
Is it reasonable to cancel and restart the vacuum process periodically No. How big is that table, anyway? Are you trying a VACUUM FULL, or plain vacuum? It's only about 60GB in size but appears it has been missed for nearly a month for vacuum and probably has a large percentage dead

Re: [HACKERS] Inherited indexes.

2005-10-04 Thread Jim C. Nasby
On Sun, Oct 02, 2005 at 09:46:07PM -0400, Tom Lane wrote: Fredrik Olsson [EMAIL PROTECTED] writes: To allow indexes to be inherited so unique, foreign keys and such works properly with inheritance has been on the todo for quite some time. I thought that most probably it is a very non

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Martijn van Oosterhout
On Tue, Oct 04, 2005 at 10:06:24AM -0400, Tom Lane wrote: Martijn van Oosterhout kleptog@svana.org writes: I'm using: gcc (GCC) 3.3.5 (Debian 1:3.3.5-13) I don't know what the units of this number are, but it's apparently far too gcc-version-dependent to consider putting into our build

Re: [HACKERS] Vacuum and Transactions

2005-10-04 Thread Tom Lane
Rod Taylor [EMAIL PROTECTED] writes: As I understand it vacuum operates outside of the regular transaction It's a perfectly normal transaction. Is it reasonable to cancel and restart the vacuum process periodically No. How big is that table, anyway? Are you trying a VACUUM FULL, or plain

Re: [HACKERS] effective SELECT from child tables

2005-10-04 Thread Jim C. Nasby
On Sat, Oct 01, 2005 at 07:59:11PM +0100, Simon Riggs wrote: On Sat, 2005-10-01 at 10:57 -0500, Jim C. Nasby wrote: To clarify, this is a hard-coded implementation of what I'm asking for: http://cvs.distributed.net/viewcvs.cgi/stats-sql/logdb/ in a nutshell: CREATE TABLE log_other (

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread Martijn van Oosterhout
On Tue, Oct 04, 2005 at 03:56:53PM +0100, Simon Riggs wrote: I've been using gcc 3.4 and saw no warning when using either -Winline or -O3 -Winline. Ok, I've just installed 3.4 and verified that. I examined the asm code and gcc is inlining it. I concede, at this point just throw in -Winline and

Re: [HACKERS] Vacuum and Transactions

2005-10-04 Thread Tom Lane
Rod Taylor [EMAIL PROTECTED] writes: The catch is that there are some other very active structures (like pg_listener for Slony) which after a couple of hours without vacuuming will quickly have the DB at an unreasonably high load (low tens) which seems to all but halt the vacuum on the large

Re: [HACKERS] Inherited indexes.

2005-10-04 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: On Sun, Oct 02, 2005 at 09:46:07PM -0400, Tom Lane wrote: 1. A cross-table index would need to store a table OID as well as the existing block/offset information in order to tell you what an entry is pointing at. Wouldn't it make more sense to use a

Re: [HACKERS] Inherited indexes.

2005-10-04 Thread Zeugswetter Andreas DAZ SD
Another possibility is optimizing for the special case of indexing on a partitioning key. In this case, index values would be very localized to one table, so just storing the table info on each index page (or something similar) would work well. If you have the partitioning key in the

Re: [HACKERS] Inherited indexes.

2005-10-04 Thread Jim C. Nasby
On Tue, Oct 04, 2005 at 11:05:49AM -0400, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: On Sun, Oct 02, 2005 at 09:46:07PM -0400, Tom Lane wrote: 1. A cross-table index would need to store a table OID as well as the existing block/offset information in order to tell you what an

Re: [HACKERS] Inherited indexes.

2005-10-04 Thread Jim C. Nasby
Well, I never said unique, but you're correct, it's pretty undesirable to put a global index on your partitioning key. On Tue, Oct 04, 2005 at 06:16:21PM +0200, Zeugswetter Andreas DAZ SD wrote: Another possibility is optimizing for the special case of indexing on a partitioning key. In

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-04 Thread mark
On Tue, Oct 04, 2005 at 05:23:41PM +0200, Martijn van Oosterhout wrote: On Tue, Oct 04, 2005 at 03:56:53PM +0100, Simon Riggs wrote: I've been using gcc 3.4 and saw no warning when using either -Winline or -O3 -Winline. Ok, I've just installed 3.4 and verified that. I examined the asm code

Re: [PERFORM] [HACKERS] Query in SQL statement

2005-10-04 Thread Jim C. Nasby
On Sat, Oct 01, 2005 at 12:51:08PM -0700, Roger Hand wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Jim C. Nasby Sent: Friday, September 30, 2005 4:49 PM Subject: Re: [PERFORM] [HACKERS] Query in SQL statement I suggest ditching the

Re: [HACKERS] Inherited indexes.

2005-10-04 Thread Simon Riggs
On Tue, 2005-10-04 at 18:16 +0200, Zeugswetter Andreas DAZ SD wrote: Another possibility is optimizing for the special case of indexing on a partitioning key. In this case, index values would be very localized to one table, so just storing the table info on each index page (or something

[HACKERS] Announcing Veil

2005-10-04 Thread Marc Munro
This is to announce the first, Alpha, release of Veil, a database security add-on for Postgres. It allows databases to be created with efficient row-level access controls; different users have access to different subsets of data. Veil is hosted on pgfoundry at http://veil.projects.postgresql.org/

Re: [HACKERS] Announcing Veil

2005-10-04 Thread Tom Lane
Marc Munro [EMAIL PROTECTED] writes: Since I was unable to dynamically assign a LWLock using LWLockAssign (none available), I have fairly arbitrarily overloaded the use of existing LWLocks. When the flames die down perhaps we can discuss making a small number (one would be enough for me) of

[HACKERS] prefix btree implementation

2005-10-04 Thread Qingqing Zhou
I am not sure if this idea was mentioned before. The basic prefix btree idea is quite straightforward, i.e., try to compress the key items within a data page by sharing the common prefix. Thus the fanout of the page is increased and the benefits is obvious theorectically. Consider the following