Re: [HACKERS] string_to_array eats too much memory?

2006-11-09 Thread Michael Paesold
Tom Lane writes: Tatsuo Ishii [EMAIL PROTECTED] writes: string_to_array() consumes too much memory. For example, to make ~70k array elements, string_to_array seems to eat several Gig bytes of memory. I'd argue that the problem comes from enlarging the work arrays only 64 elements at a time in

Re: [HACKERS] Introducing an advanced Frequent Update Optimization

2006-11-09 Thread Andrew Sullivan
On Mon, Nov 06, 2006 at 09:50:53PM +, Simon Riggs wrote: - There are specific issues with the optimizer's ability to understand dead row numbers, which can in some cases lead to SeqScan plans that are inappropriate when tables grow because of updates. This is a red-herring that can lead

Re: [HACKERS] 8.2 Beta 3 Now Available for Download / Testing ...

2006-11-09 Thread Andrew Dunstan
Heikki Linnakangas wrote: Andrew Dunstan wrote: Marc G. Fournier wrote: As a result of there being two *known* outstanding bugs, we have just bundled up a Beta3, to allow for testing of the recent patch concerning WAL replay ... What are the bugs? AFAIK: 1. Tuple freezing and hint

Re: [HACKERS] Introducing an advanced Frequent Update Optimization

2006-11-09 Thread Simon Riggs
On Thu, 2006-11-09 at 04:09 -0500, Andrew Sullivan wrote: On Mon, Nov 06, 2006 at 09:50:53PM +, Simon Riggs wrote: - There are specific issues with the optimizer's ability to understand dead row numbers, which can in some cases lead to SeqScan plans that are inappropriate when tables

[HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Simon Riggs
Design Overview of HOT Updates -- The objective is to increase the speed of the UPDATE case, while minimizing the overall negative effects of the UPDATE. We refer to the general requirement as *Frequent Update Optimization*, though this design proposal is for Heap

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Martijn van Oosterhout
Nice idea, just one question: On Thu, Nov 09, 2006 at 05:13:16PM +, Simon Riggs wrote: Behavioural Characteristics --- With HOT, it is easily possible that the chain of prior versions spans many blocks. The chain always starts with the block of the root tuple

Re: [HACKERS] Frequent Update Project: Design Overview of HOTUpdates

2006-11-09 Thread Simon Riggs
On Thu, 2006-11-09 at 18:49 +0100, Martijn van Oosterhout wrote: Nice idea, just one question: It seems to me that bitmap index scans will get these same characteristics also, right? The bitmap scan will have to follow the chain of any possibly matching tuple in any of the blocks that are in

Re: [HACKERS] Introducing an advanced Frequent Update Optimization

2006-11-09 Thread Simon Riggs
On Tue, 2006-11-07 at 15:00 +1300, Mark Kirkwood wrote: Simon Riggs wrote: EnterpriseDB has been running a research project to improve the performance of heavily updated tables. We have a number of approaches prototyped and we'd like to discuss the best of these now on -hackers for

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Josh Berkus
Simon, If we perform an update that meets the HOT criteria then we put the new version into the overflow relation; we describe this as a HOT UPDATE. If we perform an update that does not meet the criteria, then we carry on with the existing/old MVCC behaviour; we describe this as a non-HOT

Re: [HACKERS] Introducing an advanced Frequent Update Optimization

2006-11-09 Thread Christopher Browne
Quoth [EMAIL PROTECTED] (Simon Riggs): On Tue, 2006-11-07 at 15:00 +1300, Mark Kirkwood wrote: Simon Riggs wrote: EnterpriseDB has been running a research project to improve the performance of heavily updated tables. We have a number of approaches prototyped and we'd like to discuss the

Re: [HACKERS] plperl/plperlu interaction

2006-11-09 Thread Mark Dilger
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Anyway, it is probably not expected by many users that loading a module in plperlu makes it available to plperl - I was slightly surprised myself to see it work and I am probably more aware than most of perl and plperl subtleties. I

Re: [HACKERS] Frequent Update Project: Design Overview of HOTUpdates

2006-11-09 Thread Simon Riggs
On Thu, 2006-11-09 at 13:21 -0800, Josh Berkus wrote: Simon, If we perform an update that meets the HOT criteria then we put the new version into the overflow relation; we describe this as a HOT UPDATE. If we perform an update that does not meet the criteria, then we carry on with the

Re: [HACKERS] Introducing an advanced Frequent Update Optimization

2006-11-09 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: ...and of course it would be good if LISTEN/NOTIFY were able to use this concept also, to help Slony along also. LISTEN/NOTIFY are overdue to be rewritten to not use a table at all, so I'm not particularly worried about whether this idea is applicable to

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: As more UPDATEs take place these tuple chains would grow, making locating the latest tuple take progressively longer. This is the part that bothers me --- particularly the random-access nature of the search. I wonder whether you couldn't do something

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Josh Berkus
Tom, Actually, you omitted to mention the locking aspects of moving tuples around --- exactly how are you going to make that work without breaking concurrent scans? I believe that's the unsolved technical issue in the prototype, unless Pavan has solved it in the last two weeks. Pavan? --

[HACKERS] Various breakages in new contrib/isn module

2006-11-09 Thread Tom Lane
It occurred to me to run the regression tests type_sanity and opr_sanity over the contrib/isn code. (The easy way to do this is to copy the isn install script into the regress/sql directory and then add it to serial_schedule just before those two tests.) It turned up a couple of moderately

Re: [HACKERS] plperl/plperlu interaction

2006-11-09 Thread Andrew Dunstan
Mark Dilger wrote: Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Anyway, it is probably not expected by many users that loading a module in plperlu makes it available to plperl - I was slightly surprised myself to see it work and I am probably more aware than most of perl and

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Pavan Deolasee
On 11/10/06, Josh Berkus josh@agliodbs.com wrote: Tom, Actually, you omitted to mention the locking aspects of moving tuples around --- exactly how are you going to make that work without breaking concurrent scans?I believe that's the unsolved technical issue in the prototype, unless Pavan has

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: On 11/10/06, Josh Berkus josh@agliodbs.com wrote: I believe that's the unsolved technical issue in the prototype, unless Pavan has solved it in the last two weeks. Pavan? When an overflow tuple is copied back to the main heap, the overflow tuple is

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Pavan Deolasee
On 11/10/06, Tom Lane [EMAIL PROTECTED] wrote: Pavan Deolasee [EMAIL PROTECTED] writes: On 11/10/06, Josh Berkus josh@agliodbs.com wrote: I believe that's the unsolved technical issue in the prototype, unless Pavan has solved it in the last two weeks. Pavan? When an overflow tuple is copied

Re: [HACKERS] Frequent Update Project: Design Overview of HOT

2006-11-09 Thread Hannu Krosing
Ühel kenal päeval, N, 2006-11-09 kell 18:28, kirjutas Tom Lane: Simon Riggs [EMAIL PROTECTED] writes: As more UPDATEs take place these tuple chains would grow, making locating the latest tuple take progressively longer. This is the part that bothers me --- particularly the random-access

[HACKERS] beta3 CFLAGS issue on openbsd

2006-11-09 Thread Jeremy Drake
I was trying to compile 8.2beta3 on openbsd, and ran into an interesting issue. My account on the particular openbsd box has some restrictive ulimit settings, so I don't have a lot of memory to work with. I was getting an out of memory issue linking postgres, while I did not before. I figured

Re: [HACKERS] Frequent Update Project: Design Overview of HOT Updates

2006-11-09 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: On 11/10/06, Tom Lane [EMAIL PROTECTED] wrote: (2) Isn't this full of race conditions? I agree, there could be race conditions. But IMO we can handle those. Doubtless you can prevent races by introducing a bunch of additional locking. The question

Re: [HACKERS] Frequent Update Project: Design Overview of HOT

2006-11-09 Thread Hannu Krosing
Ühel kenal päeval, R, 2006-11-10 kell 09:06, kirjutas Hannu Krosing: Ühel kenal päeval, N, 2006-11-09 kell 18:28, kirjutas Tom Lane: Simon Riggs [EMAIL PROTECTED] writes: As more UPDATEs take place these tuple chains would grow, making locating the latest tuple take progressively longer.

Re: [HACKERS] beta3 CFLAGS issue on openbsd

2006-11-09 Thread Tom Lane
Jeremy Drake [EMAIL PROTECTED] writes: I figured out that the -g flag was being surreptitiously added to my CFLAGS. It was like pulling teeth trying to get the -g flag out. I believe that this is a default behavior of autoconf scripts. I remember having done some ugly hacks years ago to