Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Greg Stark
Tom Lane [EMAIL PROTECTED] writes: Rod Taylor [EMAIL PROTECTED] writes: If when we have partitions, that'll be good enough. If partitions aren't available this would be quite painful to anyone with large tables -- much as the days of old used to be painful for ANALYZE. Yeah ... I am

Re: [HACKERS] bitmapscan test, no success, bs is not faster

2005-04-27 Thread Pavel Stehule
On Tue, 26 Apr 2005, Tom Lane wrote: Pavel Stehule [EMAIL PROTECTED] writes: I tested bitmap scan and maybe I didnt find good examples, but with bitmap scan is slower than hashjoin. Only when I use non otiptimized SELECT bps was little bit faster. All my SELECTs are equal. Bitmap

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Simon Riggs
On Tue, 2005-04-26 at 15:00 -0700, Gurmeet Manku wrote: 2. In a single scan, it is possible to estimate n_distinct by using a very simple algorithm: Distinct sampling for highly-accurate answers to distinct value queries and event reports by Gibbons, VLDB 2001.

Re: [HACKERS] [PATCHES] Continue transactions after errors in psql

2005-04-27 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm finding it hard to visualize a non-interactive script making any good use of such a setting. Without a way to test whether you got an error or not, it would amount to an ignore errors within transactions mode, which seems a pretty bad idea.

Re: [HACKERS] possible TODO: read-only tables, select from indexes

2005-04-27 Thread Simon Riggs
On Sat, 2005-04-23 at 15:04 -0700, Ron Mayer wrote: Bruce Momjian wrote: See this TODO: * Allow data to be pulled directly from indexes I think this is the direction we should be heading because it has more general usefulness. I think read-only tables would have a few different types

[HACKERS] populating a table via the COPY command using C code.

2005-04-27 Thread Mak, Jason
hi, I'm writing an application in C that basically converts binary data into something meaningful. My first attempt was to parse the binary and insert directly to the database in one step. But this proved to be very slow. So I decided to go with a two step process. The first step is to

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Rod Taylor
On Tue, 2005-04-26 at 19:03 -0400, Greg Stark wrote: This one looks *really* good. http://www.aladdin.cs.cmu.edu/papers/pdfs/y2001/dist_sampl.pdf It does require a single full table scan Ack.. Not by default please. I have a few large append-only tables (vacuum isn't necessary) which

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Robert Treat
There is a fairly lengthy discussion going on right now on the bizgres mailing list about this topic, if your interested in helping out you might want to join that list. Robert Treat On Tue, 2005-04-26 at 05:43, [EMAIL PROTECTED] wrote: Ok! The Links your posted are great and i guessing it

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Gurmeet Manku
Hi everybody! Perhaps the following papers are relevant to the discussion here (their contact authors have been cc'd): 1. The following proposes effective algorithms for using block-level sampling for n_distinct estimation: Effective use of block-level sampling in statistics

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Andrew Dunstan
Mischa Sandberg wrote: Perhaps I can save you some time (yes, I have a degree in Math). If I understand correctly, you're trying extrapolate from the correlation between a tiny sample and a larger sample. Introducing the tiny sample into any decision can only produce a less accurate result

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Dave Held
-Original Message- From: Gurmeet Manku [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 26, 2005 5:01 PM To: Simon Riggs Cc: Tom Lane; josh@agliodbs.com; Greg Stark; Marko Ristola; pgsql-perform; pgsql-hackers@postgresql.org; Utkarsh Srivastava; [EMAIL PROTECTED] Subject: Re:

Re: [HACKERS] [PATCHES] Continue transactions after errors in psql

2005-04-27 Thread Robert Treat
On Tue, 2005-04-26 at 10:28, Tom Lane wrote: Greg Sabino Mullane [EMAIL PROTECTED] writes: To reiterate my opinion, I think the behavior should be the same for interactive and non-interactive sessions. Not only will it prevent nasty surprises, but unless we make a third 'setting', there

Re: [HACKERS] [proposal] protocol extension to support loadable stream filters

2005-04-27 Thread Brent Verner
[2005-04-26 23:00] Tom Lane said: | Brent Verner [EMAIL PROTECTED] writes: | | I also wonder what happens when | | the client and server disagree on the meaning of a filter name. | |How this is any different than saying ...when the client and | server disagree on the meaning of a

[HACKERS] PITR and postmaster.pid

2005-04-27 Thread Rod Taylor
When I'm doing a restore and go to start the postmaster it starts listing off warnings as a result of the pid file being in place. Not a big deal, but it might be suggested that the tar or cpio process exclude the postmaster.pid. -- ---(end of

[HACKERS] Behavior of shared/exclusive row locks

2005-04-27 Thread Tom Lane
I've been reviewing Alvaro's patch for shared row locks, and come across some corner cases that may need discussion. Does anyone disagree with the following statements? 1. If several transactions are holding shared lock on a row, and one of them wants to actually modify the row (or upgrade its

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Dave Held
-Original Message- From: Greg Stark [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 27, 2005 1:00 AM To: Tom Lane Cc: Rod Taylor; Greg Stark; pgsql-hackers@postgresql.org; Gurmeet Manku Subject: Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested? Tom Lane

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Josh Berkus
Mischa, Perhaps I can save you some time (yes, I have a degree in Math). If I understand correctly, you're trying extrapolate from the correlation between a tiny sample and a larger sample. Introducing the tiny sample into any decision can only produce a less accurate result than just taking

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Dave Held
-Original Message- From: Josh Berkus [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 27, 2005 10:25 AM To: Andrew Dunstan Cc: Mischa Sandberg; pgsql-perform; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested? [...]

Re: [HACKERS] [PATCHES] Continue transactions after errors in psql

2005-04-27 Thread Bruce Momjian
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: \begin_ignore_error DROP TABLE foo; \end_ignore_error I meant it's a lot to type ;-) Well, that's just a matter of choosing good (ie short) names for the backslash commands. I was trying to be clear rather than proposing

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Greg Stark
Dave Held [EMAIL PROTECTED] writes: Actually, it's more to characterize how large of a sample we need. For example, if we sample 0.005 of disk pages, and get an estimate, and then sample another 0.005 of disk pages and get an estimate which is not even close to the first estimate,

Re: [HACKERS] [PATCHES] Continue transactions after errors in psql

2005-04-27 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: Well, that's just a matter of choosing good (ie short) names for the backslash commands. I was trying to be clear rather than proposing names I would actually want to use ;-). Any suggestions? Well, if we allowed

Re: [HACKERS] [PATCHES] Continue transactions after errors in psql

2005-04-27 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: Well, that's just a matter of choosing good (ie short) names for the backslash commands. I was trying to be clear rather than proposing names I would actually want to use ;-). Any suggestions? Well, if we

Re: [HACKERS] Behavior of shared/exclusive row locks

2005-04-27 Thread Alvaro Herrera
On Wed, Apr 27, 2005 at 11:19:34AM -0400, Tom Lane wrote: 1. If several transactions are holding shared lock on a row, and one of them wants to actually modify the row (or upgrade its lock to exclusive), it must wait for the others to end but can then do so. (I think the patch does this

Re: [HACKERS] Behavior of shared/exclusive row locks

2005-04-27 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: On Wed, Apr 27, 2005 at 11:19:34AM -0400, Tom Lane wrote: Another issue that we may need to think about is that there is no protection against starvation: a would-be acquirer of a row lock could wait forever, because there isn't any mechanism preventing

Re: [HACKERS] Behavior of shared/exclusive row locks

2005-04-27 Thread Alvaro Herrera
On Wed, Apr 27, 2005 at 05:51:47PM -0400, Tom Lane wrote: Suppose that we redo the LOCKTAGs per previous discussion (which I would like to do anyway), so that it is possible to define an lmgr lock on a particular tuple. Hm. If you want I can give you the part of the patch that dealt with

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Hannu Krosing
On T, 2005-04-26 at 16:52 -0400, Robert Treat wrote: There is a fairly lengthy discussion going on right now on the bizgres mailing list about this topic, if your interested in helping out you might want to join that list. Wow! there is a BizGres mailinglist !? And this is where the

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Josh Berkus
Hannu, There is a fairly lengthy discussion going on right now on the bizgres mailing list about this topic, if your interested in helping out you might want to join that list. Wow! there is a BizGres mailinglist !? And this is where the table-partitioning discussion is! Where is this

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Hannu Krosing
On K, 2005-04-27 at 15:40 -0700, Josh Berkus wrote: Hannu, There is a fairly lengthy discussion going on right now on the bizgres mailing list about this topic, if your interested in helping out you might want to join that list. Wow! there is a BizGres mailinglist !? And this

[HACKERS] PITR bad restore possibility?

2005-04-27 Thread Rod Taylor
What happens if for reasons of broken tape, disk, etc. you lose some of your WAL logs which happens to correspond to the middle of the snapshot backup? The equivalent would be to: 1) Start the snapshot backup (tar) 2) Stop logging usable WAL logs (say a tape jammed or disk is corrupted) 3)

Re: [HACKERS] [PATCHES] [WIP] shared locks

2005-04-27 Thread Tom Lane
Found another interesting thing while testing this. I got a core dump from the Assert in GetMultiXactIdMembers, complaining that it was being asked about a MultiXactId = nextMXact. Sure enough, there was a multixact on disk, left over from a previous core-dumped test, that was larger than the

[HACKERS] Developer Community

2005-04-27 Thread Juan Jose Costello Levien
Hello, My name is Juan, and I am interested in being a PostgreSQL Developer. I would like to contribute with items in the TODO list. I am new to the list, and would like to know if you have reservations or beliefs applicated to the members of the developer community, so if I can enter to it or I

Re: [HACKERS] PITR bad restore possibility?

2005-04-27 Thread Tom Lane
Rod Taylor [EMAIL PROTECTED] writes: What happens if for reasons of broken tape, disk, etc. you lose some of your WAL logs which happens to correspond to the middle of the snapshot backup? You're screwed ... just like if you lost part of the snapshot itself. If you're really lucky the missing

Re: [HACKERS] Developer Community

2005-04-27 Thread Jim C. Nasby
See http://www.postgresql.org/docs/faqs.FAQ_DEV.html; esp. items 1.1-1.18. On Wed, Apr 27, 2005 at 08:35:10PM -0300, Juan Jose Costello Levien wrote: Hello, My name is Juan, and I am interested in being a PostgreSQL Developer. I would like to contribute with items in the TODO list. I am new

Re: [HACKERS] PITR bad restore possibility?

2005-04-27 Thread Rod Taylor
On Wed, 2005-04-27 at 20:14 -0400, Tom Lane wrote: Rod Taylor [EMAIL PROTECTED] writes: What happens if for reasons of broken tape, disk, etc. you lose some of your WAL logs which happens to correspond to the middle of the snapshot backup? You're screwed ... just like if you lost part of

Re: [HACKERS] Developer Community

2005-04-27 Thread Juan Jose Costello Levien
Jim, So I interpret that simply proposing work and doing it are all the requirements needed. Thank you. Regards, -- Juan Jose Costello Levien E-Mail: [EMAIL PROTECTED] Web: http://jcostello.ath.cxOn 4/27/05, Jim C. Nasby [EMAIL PROTECTED] wrote:See

Re: [HACKERS] pg_restore stuck in a loop?

2005-04-27 Thread Rod Taylor
On Tue, 2005-04-26 at 23:22 -0400, Tom Lane wrote: Rod Taylor [EMAIL PROTECTED] writes: I eventually clued in and made a TOC and removed all of the Slony items, but I'm still curious to know what exactly pg_restore had been doing for the last hour or so. You tell us ;-). You've got the

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Christopher Kings-Lynne
There is a fairly lengthy discussion going on right now on the bizgres mailing list about this topic, if your interested in helping out you might want to join that list. What's the point of keeping such backend development discussion separate from the -hackers list? It's always been a mistake

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Bruce Momjian
Christopher Kings-Lynne wrote: There is a fairly lengthy discussion going on right now on the bizgres mailing list about this topic, if your interested in helping out you might want to join that list. What's the point of keeping such backend development discussion separate from the

Re: [HACKERS] Tablepartitioning: Will it be supported in Future?

2005-04-27 Thread Neil Conway
Christopher Kings-Lynne wrote: What's the point of keeping such backend development discussion separate from the -hackers list? It's always been a mistake in the past... Yeah, it struck me as a bad idea as well. -Neil ---(end of broadcast)--- TIP

[HACKERS] Increased company involvement

2005-04-27 Thread Bruce Momjian
I am very excited to see companies involved in PostgreSQL development. It gives us funding for developers and features that is new for us. We had Fujitsu funding some features for 8.0 and that really helped us. However, there was a lot of coordination that happened with Fujitsu that I don't see

Re: [HACKERS] [pgsql-advocacy] Increased company involvement

2005-04-27 Thread Joshua D. Drake
However, there was a lot of coordination that happened with Fujitsu that I don't see happening with the current companies involved. Companies are already duplicating work that is also done by community members or by other companies. That is bound to happen no matter what. Look at plJava and plJ.

Re: [HACKERS] [pgsql-advocacy] Increased company involvement

2005-04-27 Thread Christopher Kings-Lynne
And finally, we have a few companies working on features that they eventually want merged back into the PostgreSQL codebase. That is a very tricky process and usually goes badly unless the company seeks community involvement from the start, including user interface, implementation, and coding

Re: [HACKERS] [pgsql-advocacy] Increased company involvement

2005-04-27 Thread Bruce Momjian
Joshua D. Drake wrote: However, there was a lot of coordination that happened with Fujitsu that I don't see happening with the current companies involved. Companies are already duplicating work that is also done by community members or by other companies. That is bound to happen no

Re: [HACKERS] DO INSTEAD and conditional rules

2005-04-27 Thread Zeugswetter Andreas DAZ SD
If we did not define it that way, I think your example would have to error out --- how would you choose which INSTEAD rule wins? The documentation says that they evaluate in alphabetical order by name. So I would expect that the first one to have its WHERE statement evaluate to true

Re: [HACKERS] [pgsql-advocacy] Increased company involvement

2005-04-27 Thread Joshua D. Drake
Do companies want to write for Blue Hat PostgreSQL and Suza PostgreSQL because that might be what happens if we don't stay organized? In fact, it might have be happening already. Well that depends... If the companies are writing for Pervasive PostgreSQL I don't think they would have a problem

Re: [HACKERS] [pgsql-advocacy] Increased company involvement

2005-04-27 Thread Bruce Momjian
Joshua D. Drake wrote: Do companies want to write for Blue Hat PostgreSQL and Suza PostgreSQL because that might be what happens if we don't stay organized? In fact, it might have be happening already. Well that depends... If the companies are writing for Pervasive PostgreSQL I