Re: [HACKERS] Huge memory consumption during vacuum (v.8.0)

2005-01-29 Thread Oleg Bartunov
On Sun, 30 Jan 2005, Oleg Bartunov wrote: On Sun, 30 Jan 2005, Tom Lane wrote: Oleg Bartunov writes: On Sun, 30 Jan 2005, Tom Lane wrote: I'm confused. The log trace you showed us before appeared to be from a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is it ... or did you ch

Re: [HACKERS] Huge memory consumption during vacuum (v.8.0)

2005-01-29 Thread Oleg Bartunov
On Sun, 30 Jan 2005, Tom Lane wrote: Oleg Bartunov writes: On Sun, 30 Jan 2005, Tom Lane wrote: I'm confused. The log trace you showed us before appeared to be from a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is it ... or did you change? Yes, first time I tried vacuum from

Re: [HACKERS] Allowing VACUUM to time out when waiting for locks?

2005-01-29 Thread Tom Lane
Philip Warner <[EMAIL PROTECTED]> writes: > Am I correct in saying that the FSM now tracks the entire table, and that > the FSM parameters just determine how much is stored in memory? No. Any free space that can't be remembered in FSM is lost to use. (Not completely --- an update of a row on the

Re: [HACKERS] Huge memory consumption during vacuum (v.8.0)

2005-01-29 Thread Tom Lane
Oleg Bartunov writes: > On Sun, 30 Jan 2005, Tom Lane wrote: >> I'm confused. The log trace you showed us before appeared to be from >> a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is >> it ... or did you change? > Yes, first time I tried vacuum from withing psql, next time

Re: [HACKERS] Huge memory consumption during vacuum (v.8.0)

2005-01-29 Thread Oleg Bartunov
On Sun, 30 Jan 2005, Tom Lane wrote: Oleg Bartunov writes: I tried run 'vacuumdb -v -z -f wsdb > vacuum-wsdb.log 2>&1&' I'm confused. The log trace you showed us before appeared to be from a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is it ... or did you change? Yes, first t

Re: [HACKERS] Allowing VACUUM to time out when waiting for

2005-01-29 Thread Philip Warner
At 02:53 AM 30/01/2005, Tom Lane wrote: Philip Warner <[EMAIL PROTECTED]> writes: > We have a frequently updated (peak > 5/sec) table with about 1000 rows. > We run VACCUM FULL on this table every 5 minutes. Plain vacuum (perhaps executed even more often, like once a minute) will cause fewer lockin

Re: [HACKERS] Huge memory consumption during vacuum (v.8.0)

2005-01-29 Thread Tom Lane
Oleg Bartunov writes: > I tried run 'vacuumdb -v -z -f wsdb > vacuum-wsdb.log 2>&1&' I'm confused. The log trace you showed us before appeared to be from a non-FULL vacuum, but here you're saying it's VACUUM FULL. Which is it ... or did you change? regards, tom lane -

Re: [HACKERS] [BUGS] Bug in create operator and/or initdb

2005-01-29 Thread Tom Lane
Steve Atkins <[EMAIL PROTECTED]> writes: > For a replacement type, how important is it that it be completely > compatible with the existing inet/cidr types? Is anyone actually using > inet types with a non-cidr mask? If you check the archives you'll discover that our current inet/cidr types were l

[HACKERS] Huge memory consumption during vacuum (v.8.0)

2005-01-29 Thread Oleg Bartunov
Hi there, seems I have a serious problem with vacuuming of rather big table (500,000,000 rows) on dual Intel(R) Xeon(TM) CPU 2.40GHz, 1Gb RAM, running Linux 2.6.7. I have PostgreSQL 8.0 release installed with slightly changed postgresql.conf: shared_buffers = 24576 # min 16, at least max_con

Re: [HACKERS] IBM patent

2005-01-29 Thread Kevin Brown
Marc G. Fournier wrote: > On Wed, 26 Jan 2005, Christopher Browne wrote: > > >Actually, the latter isn't so. > > > >If Mammoth or Pervasive or such release their own release of > >PostgreSQL, nothing has historically mandated that they make that > >release available under the BSD license. > > > >

Re: [HACKERS] [BUGS] Bug in create operator and/or initdb

2005-01-29 Thread Steve Atkins
On Sat, Jan 29, 2005 at 10:07:30PM -0500, Tom Lane wrote: > "John Hansen" <[EMAIL PROTECTED]> writes: > > In that case may I suggest fixing the catalog so network_* functions exists > > for both datatypes! > > Redesigning the inet/cidr distinction is on the to-do list (though I'm > afraid not ver

Re: [HACKERS] [BUGS] Bug in create operator and/or initdb

2005-01-29 Thread Tom Lane
"John Hansen" <[EMAIL PROTECTED]> writes: > In that case may I suggest fixing the catalog so network_* functions exists > for both datatypes! Redesigning the inet/cidr distinction is on the to-do list (though I'm afraid not very high on the list). ISTM it should either be one type with a disting

Re: [HACKERS] [BUGS] Bug in create operator and/or initdb

2005-01-29 Thread Tom Lane
"John Hansen" <[EMAIL PROTECTED]> writes: > CREATE FUNCTION my_func (inet, inet) as '$libdir/my_func.so' LANGUAGE 'C' > IMMUTABLE STRICT; > CREATE OPERATOR <<< ( > PROCEDURE = my_func, > LEFTARG = cidr, > RIGHTARG = cidr, > RESTRICT = contsel, > JOIN = contjoinsel > )

Re: [HACKERS] [BUGS] Bug in create operator and/or initdb

2005-01-29 Thread John Hansen
> My opinion is that this is a very bogus shortcut in the > network datatype code. There are no cases outside the > inet/cidr group where an operator doesn't exactly match its > underlying function. (The whole business of inet and cidr > being almost but not quite the same type is maldesigned

[HACKERS] Bug in create operator and/or initdb

2005-01-29 Thread John Hansen
The following seems to me a bug in either initdb or create operator: CREATE FUNCTION my_func (inet, inet) as '$libdir/my_func.so' LANGUAGE 'C' IMMUTABLE STRICT; CREATE OPERATOR <<< ( PROCEDURE = my_func, LEFTARG = cidr, RIGHTARG = cidr, RESTRICT = contsel,

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Mike Rylander
On Sun, 30 Jan 2005 12:15:20 +1100, Neil Conway <[EMAIL PROTECTED]> wrote: > It might _work_, I just don't see the point. Given an attribute of a > heap relation that has N distinct values and T tuples, you need to store > > - N bitmaps, each of T bits (before compression) > - T ctids > - a way to

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Neil Conway
Mike Rylander wrote: For the initial example where the index is implemented as a set of unique keys from the table and a bitmap for each key this would look a unique index, but with an extra datum at at each index node to hold the bitmap for that key. If implemented that way an augmented B-Tree st

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Mike Rylander
On Sun, 30 Jan 2005 11:07:59 +1100, Neil Conway <[EMAIL PROTECTED]> wrote: > Mike Rylander wrote: > > For on-disk bitmap indexes, yes. I don't see any reason this couldn't > > be done with GiST > > It might be possible to do it with GiST, but GiST is designed for > implementing tree-structured in

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Neil Conway
Mike Rylander wrote: For on-disk bitmap indexes, yes. I don't see any reason this couldn't be done with GiST It might be possible to do it with GiST, but GiST is designed for implementing tree-structured indexes; I don't think it's the right tool for the job. -Neil ---(e

Re: [HACKERS] IBM patent

2005-01-29 Thread Marc G. Fournier
On Wed, 26 Jan 2005, Christopher Browne wrote: Actually, the latter isn't so. If Mammoth or Pervasive or such release their own release of PostgreSQL, nothing has historically mandated that they make that release available under the BSD license. Presumably acceptance of the patent would change that

Re: [HACKERS] IBM patent

2005-01-29 Thread Christopher Browne
Oops! [EMAIL PROTECTED] (Tommi Maekitalo) was seen spray-painting on a wall: > Hi, > > I just read about this IBM-patent-issue at www.heise.de. IBM grants > this patens to all projects, which follow one of the licenses, which > are approved by the open-source-initiative. And the BSD-license is > as

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Tom Lane
Mike Rylander <[EMAIL PROTECTED]> writes: > As a side note, wouldn't the in-memory bitmaps pretty much kill the > need for multicolumn indexes? It seems that they would be able to > join index scans on the same table, and then there would be no need > for industrial strength cross-column correlati

Re: [HACKERS] [pgsql-hackers] Allow GRANT/REVOKE permissions to be applied to all schema

2005-01-29 Thread Robert Treat
On Saturday 29 January 2005 09:14, Stephen Frost wrote: > * Tom Lane ([EMAIL PROTECTED]) wrote: > > Or just make the user enter two commands for this case. Aside from > > syntactic simplicity, that might be a good idea anyway. The NEW TABLES > > case is *fundamentally* different from every other

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Victor Yegorov
* Dawid Kuroczko <[EMAIL PROTECTED]> [29.01.2005 21:25]: > With in-memory bitmap, the search would start with index a, all > matching rows would form the bitmap; then the second search > would go through b index, forming another bitmap. Which then > would be ANDed with previous bitmap. Not only

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Dawid Kuroczko
On Sat, 29 Jan 2005 21:54:39 +0200, Victor Yegorov <[EMAIL PROTECTED]> wrote: > * Dawid Kuroczko <[EMAIL PROTECTED]> [29.01.2005 21:25]: > > > With in-memory bitmap, the search would start with index a, all > > matching rows would form the bitmap; then the second search > > would go through b inde

Re: [HACKERS] [pgsql-hackers] Patent issues and 8.1

2005-01-29 Thread Robert Treat
On Saturday 29 January 2005 11:33, Tom Lane wrote: > Robert Treat <[EMAIL PROTECTED]> writes: > > I'm not mischarecterizing, I just feel that putting out an lru based > > 8.0.x release is such a bad idea that I'd rather do (1) than gamble on > > (2). > > I don't understand why you think it's such a

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Dawid Kuroczko
On Sat, 29 Jan 2005 18:46:44 +, Mike Rylander <[EMAIL PROTECTED]> wrote: > As a side note, wouldn't the in-memory bitmaps pretty much kill the > need for multicolumn indexes? It seems that they would be able to > join index scans on the same table, and then there would be no need > for industr

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Mike Rylander
On Sat, 29 Jan 2005 19:41:20 +0200, Victor Y. Yegorov <[EMAIL PROTECTED]> wrote: > * Paweï Niewiadomski <[EMAIL PROTECTED]> [29.01.2005 17:45]: > > > I'd like to implement bitmap indexes and want your comments. Here is > > > an essence of what I've found regarding bitmaps for the last month. > > >

[HACKERS] Security bugs in tsearch2

2005-01-29 Thread Tom Lane
I just noticed that all of tsearch2's "dict init" routines are declared like this: CREATE FUNCTION dex_init(text) returns internal as 'MODULE_PATHNAME' language 'C'; This is really unacceptable, because it breaks the type safety of the "internal" pseudotype. I quote from

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Victor Y. Yegorov
* Paweï Niewiadomski <[EMAIL PROTECTED]> [29.01.2005 17:45]: > > I'd like to implement bitmap indexes and want your comments. Here is > > an essence of what I've found regarding bitmaps for the last month. > > Do you think it would be possible to work on it as a team? Yes, why not. But everythin

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Victor Y. Yegorov
* Tom Lane <[EMAIL PROTECTED]> [29.01.2005 18:24]: > "Victor Y. Yegorov" <[EMAIL PROTECTED]> writes: > > I remember, Tom Lane suggested on-disk bitmaps > > I have suggested no such thing, and in fact believe that the sort of > index structure you are proposing would be of very little use. Why? I

Re: [HACKERS] [pgsql-hackers] Patent issues and 8.1

2005-01-29 Thread Tom Lane
Robert Treat <[EMAIL PROTECTED]> writes: > I'm not mischarecterizing, I just feel that putting out an lru based 8.0.x > release is such a bad idea that I'd rather do (1) than gamble on (2). I don't understand why you think it's such a bad idea. We do have the problem of getting adequate testin

Re: [HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Tom Lane
"Victor Y. Yegorov" <[EMAIL PROTECTED]> writes: > I remember, Tom Lane suggested on-disk bitmaps I have suggested no such thing, and in fact believe that the sort of index structure you are proposing would be of very little use. What I've been hoping to look into is *in memory* bitmaps used as an

Re: [HACKERS] Allowing VACUUM to time out when waiting for locks?

2005-01-29 Thread Tom Lane
Philip Warner <[EMAIL PROTECTED]> writes: > We have a frequently updated (peak > 5/sec) table with about 1000 rows. > We run VACCUM FULL on this table every 5 minutes. I agree with Bruno's comment that you shouldn't be doing that in the first place. Plain vacuum (perhaps executed even more often,

Re: [HACKERS] [pgsql-hackers] Patent issues and 8.1

2005-01-29 Thread Robert Treat
On Friday 28 January 2005 12:36, Josh Berkus wrote: > Robert, > > > Read the law... willful vs. unknown infringement are two different > > things. > > We're not infringing anything, yet. That's a *pending* patent. > *sigh* Thats understood. But you were using the counter-argument that we migh

Re: [HACKERS] Allowing VACUUM to time out when waiting for locks?

2005-01-29 Thread Bruno Wolff III
On Sun, Jan 30, 2005 at 01:23:11 +1100, Philip Warner <[EMAIL PROTECTED]> wrote: > > We have a frequently updated (peak > 5/sec) table with about 1000 rows. > > We run VACCUM FULL on this table every 5 minutes. Why not just use plain VACUUM? The table will reach a steady state size. You should

[HACKERS] Allowing VACUUM to time out when waiting for locks?

2005-01-29 Thread Philip Warner
We have a frequently updated (peak > 5/sec) table with about 1000 rows. We run VACCUM FULL on this table every 5 minutes. The regular updates are not long in duration, and the vacuum is fast, so they do not produce noticeable delays. When we run a pg_dump on the database: - the dump takes a long st

Re: [HACKERS] [pgsql-hackers] Allow GRANT/REVOKE permissions to be applied to all schema

2005-01-29 Thread Stephen Frost
* Tom Lane ([EMAIL PROTECTED]) wrote: > Or just make the user enter two commands for this case. Aside from > syntactic simplicity, that might be a good idea anyway. The NEW TABLES > case is *fundamentally* different from every other form of GRANT, in > that it causes future actions. So it might

[HACKERS] Implementing Bitmap Indexes

2005-01-29 Thread Victor Y. Yegorov
Hello. I'd like to implement bitmap indexes and want your comments. Here is an essence of what I've found regarding bitmaps for the last month. Consider the following table So, the bitmap for attribute A will be the with 1 attribute A(int2): following: # | A Val | Bi

Re: [HACKERS] Group-count estimation statistics

2005-01-29 Thread Mischa
> From: Sailesh Krishnamurthy <[EMAIL PROTECTED]> > > "Tom" == Tom Lane <[EMAIL PROTECTED]> writes: > > Tom> The only real solution, of course, is to acquire cross-column > Tom> statistics, but I don't see that happening in the near > Tom> future. > > Another approach is a hybrid

Re: [HACKERS] some linker troubles with rc5 on sun studio 9 ...

2005-01-29 Thread Hans-Jürgen Schönig
Tom Lane wrote: =?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= <[EMAIL PROTECTED]> writes: "tuptoaster.c", line 966: member can not have variably modified type: data We've seen that before. Apparently there are some versions of Sun's compiler that are too stupid to reduce this constant expression to a

Re: [HACKERS] strange 'vacuum verbose analyze' behaviour

2005-01-29 Thread Oleg Bartunov
Tom, After almost 20 hours running vacuum I see postmaster grew a little bit: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 20458 postgres 15 0 2136m 553m 204m D 33.2 54.7 198:18.36 postmaster It's strange that I see no output since starting vacuumdb -v -z -f wsdb > vac