Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Tom Lane wrote: In the meantime, though, it's not quite clear why this would lead to a buildfarm failure --- it should just mean a lot of extraneous files appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks like btree_gist has some files that used to be autogenerated and are

[HACKERS] is this trigger safe and efective? - locking (caching via triiggers)

2007-08-15 Thread Pavel Stehule
Hello I write sample about triggers and i have question. is my solution correct and exists better solution? Regards Pavel Stehule DROP SCHEMA safecache CASCADE; CREATE SCHEMA safecache; CREATE TABLE safecache.source_tbl(category int, int_value int); CREATE TABLE safecache.cache(category int,

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Tom Lane wrote: So I think that cvs rtag -d REL7_4_STABLE pgsql will fix it. I'd like someone to double-check that though. Also maybe we should back up the repository first? Just for your info: all VMs on tribble, which includes cvs, are backed up at 02:30 every day, CEST (that's 00:30 UTC

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: So I'd say your strategy looks good - backup and remove the phony tag. I'd also say we should probably be logging tag commands in taginfo. Presumably we mere mortal committers should not be doing any tagging whatsoever, and tags

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Magnus Hagander
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Since I don't think that a datatype solution is the way to go, I don't feel that we are as far away from an agreement as Bruce is worried about. Well, from where I sit, there is one person saying give me the foot gun,

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Tom Lane
Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: I'd like someone to double-check that though. Also maybe we should back up the repository first? Just for your info: all VMs on tribble, which includes cvs, are backed up at 02:30 every day, CEST Good, but the salient followup

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: I'd like someone to double-check that though. Also maybe we should back up the repository first? Just for your info: all VMs on tribble, which includes cvs, are backed up at 02:30 every day, CEST Good, but the

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Oleg Bartunov
On Wed, 15 Aug 2007, Magnus Hagander wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Since I don't think that a datatype solution is the way to go, I don't feel that we are as far away from an agreement as Bruce is worried about. Well, from where I sit,

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Tom Lane
Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: Good, but the salient followup questions to that are (1) backed up to where exactly?, and (2) how many days' past backups could we get at, if we had to? They are dumped to a NFS share on this schedule. That NFS share is dumped to

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: Good, but the salient followup questions to that are (1) backed up to where exactly?, and (2) how many days' past backups could we get at, if we had to? They are dumped to a NFS share on this schedule. That NFS share

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Alvaro Herrera
Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: Good, but the salient followup questions to that are (1) backed up to where exactly?, and (2) how many days' past backups could we get at, if we had to? They are dumped to a NFS share on this schedule. That NFS

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Marko Kreen
On 8/15/07, Tom Lane [EMAIL PROTECTED] wrote: I wrote: Kris Jurka [EMAIL PROTECTED] writes: It looks like parts of the CVS repository have been mistagged as belonging to REL7_4_STABLE or have been corrupted somehow: I have no idea how you make CVS do that, but I'm sure there is some

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Stefan Kaltenbrunner
Magnus Hagander wrote: Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: Good, but the salient followup questions to that are (1) backed up to where exactly?, and (2) how many days' past backups could we get at, if we had to? They are dumped to a NFS share on this

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Stefan Kaltenbrunner
Magnus Hagander wrote: Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: I'd like someone to double-check that though. Also maybe we should back up the repository first? Just for your info: all VMs on tribble, which includes cvs, are backed up at 02:30 every day,

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Stefan Kaltenbrunner
Stefan Kaltenbrunner wrote: Magnus Hagander wrote: Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: I'd like someone to double-check that though. Also maybe we should back up the repository first? Just for your info: all VMs on tribble, which includes cvs, are

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Peter Eisentraut
Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: we should at least log such commands, and maybe disallow to anyone except Marc's pgsql account. I don't think we should disallow it. Or otherwise we might one day be stuck if we need to release while some specific person is on vacation. I

Re: [HACKERS] Index Tuple Compression Approach?

2007-08-15 Thread Simon Riggs
On Wed, 2007-08-15 at 06:51 +0100, Heikki Linnakangas wrote: Jeff Davis wrote: On Tue, 2007-08-14 at 16:27 -0500, Decibel! wrote: Isn't this what Grouped Index Tuples is? http://community.enterprisedb.com/git/git-readme.txt It looks like GIT is a little different. GIT actually

Re: [HACKERS] Testing the async-commit patch

2007-08-15 Thread Simon Riggs
On Tue, 2007-08-14 at 12:29 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Tue, 2007-08-14 at 12:09 -0400, Tom Lane wrote: heapam.c lines 1843-1852 presume previous xmax can be hinted immediately, ditto lines 2167-2176, ditto lines 2716-2725. I think probably we should

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Simon Riggs
On Tue, 2007-08-14 at 17:41 -0400, Tom Lane wrote: I've just finished re-reading the prior thread, and here are what seem to me to be the salient points: * Oleg, Teodor, and all of the old-line users of tsearch2 are comfortable with setting up a trigger to maintain a materialized tsvector

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Simon Riggs
On Wed, 2007-08-15 at 08:10 +0200, Magnus Hagander wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Since I don't think that a datatype solution is the way to go, I don't feel that we are as far away from an agreement as Bruce is worried about. Well,

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Bruce Momjian
Magnus Hagander wrote: But I would like a design that is bulletproof in dump/reload scenarios, and I think it's fair to question that aspect of the tsearch2 design because we've seen many reports of people having trouble updating databases that use tsearch2. dump/reload is *the* biggest

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Magnus Hagander
On Wed, Aug 15, 2007 at 10:23:00AM -0400, Bruce Momjian wrote: Magnus Hagander wrote: But I would like a design that is bulletproof in dump/reload scenarios, and I think it's fair to question that aspect of the tsearch2 design because we've seen many reports of people having trouble

Re: [HACKERS] is this trigger safe and efective? - locking (caching via triiggers)

2007-08-15 Thread Decibel!
I don't like the locking... take a look at Ex 37-1 at the end of http://lnk.nu/postgresql.org/fhe.html for a better way (though, the comment below about going into an infinite loop is a good observantion, but I think perhaps after some number of fast tries it should start putting a sleep in the

Re: [HACKERS] [EMAIL PROTECTED]: Re: [GENERAL] array_to_set functions]

2007-08-15 Thread Decibel!
On Wed, Aug 15, 2007 at 06:47:05AM +0200, Pavel Stehule wrote: 2007/8/14, Decibel! [EMAIL PROTECTED]: On Tue, Aug 14, 2007 at 05:38:33PM +0200, Pavel Stehule wrote: 2007/8/14, Bruce Momjian [EMAIL PROTECTED]: TODO item? + If your life is a hard drive, Christ can be your

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: ISTM that the functional index would be considerably smaller than the additional column approach, since tsvectors can be quite long. That seems like a very desirable thing with larger textbases. However, without an additional column certain queries would

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Andrew Dunstan
Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: Good, but the salient followup questions to that are (1) backed up to where exactly?, and (2) how many days' past backups could we get at, if we had to? They are dumped to a NFS share on this

[HACKERS] Another idea for index-only scans

2007-08-15 Thread Bruce Momjian
I have added another idea for index-only scans to the TODO list: A third idea would be for a heap scan to check if all rows are visible and if so set a per-table flag which can be checked by index scans. Any change to the table would have to clear the flag. To detect changes during

[HACKERS] XID wraparound and busy databases

2007-08-15 Thread Bruce Momjian
I was talking to someone at LinuxWorld and they mentioned they often have activity of 6k SELECTs per second, and that they were needing to autovacuum every few days because of xid wraparound. I did some calculations and found that: 6000 * 60 * 60 * 24 51840 or 500

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Stefan Kaltenbrunner
Andrew Dunstan wrote: Tom Lane wrote: Magnus Hagander [EMAIL PROTECTED] writes: Tom Lane wrote: Good, but the salient followup questions to that are (1) backed up to where exactly?, and (2) how many days' past backups could we get at, if we had to? They are dumped to

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Stefan Kaltenbrunner
Peter Eisentraut wrote: Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: we should at least log such commands, and maybe disallow to anyone except Marc's pgsql account. I don't think we should disallow it. Or otherwise we might one day be stuck if we need to release while some

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Tom Lane
Stefan Kaltenbrunner [EMAIL PROTECTED] writes: Peter Eisentraut wrote: Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: we should at least log such commands, and maybe disallow to anyone except Marc's pgsql account. I don't think we should disallow it. Or otherwise we might one day be

Re: [HACKERS] [EMAIL PROTECTED]: Re: [GENERAL] array_to_set functions]

2007-08-15 Thread Pavel Stehule
2007/8/15, Decibel! [EMAIL PROTECTED]: On Wed, Aug 15, 2007 at 06:47:05AM +0200, Pavel Stehule wrote: 2007/8/14, Decibel! [EMAIL PROTECTED]: On Tue, Aug 14, 2007 at 05:38:33PM +0200, Pavel Stehule wrote: 2007/8/14, Bruce Momjian [EMAIL PROTECTED]: TODO item? + If

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Is enlarging the xid field something we should consider for 8.4? No. We just got the tuple header down to 24 bytes, we are not going to give that back and then some. If you are processing 6K transactions per second, you can afford to vacuum every couple

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Is enlarging the xid field something we should consider for 8.4? No. We just got the tuple header down to 24 bytes, we are not going to give that back and then some. If you are processing 6K transactions per second, you can afford

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Heikki Linnakangas
Bruce Momjian wrote: Is enlarging the xid field something we should consider for 8.4? Is the autovacuum I/O less then the extra I/O needed for an expanded xid fields on every row? I doubt that's going to happen... Maybe we can do something to reduce the xid consumption? For example, reuse

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Wednesday, August 15, 2007 12:11:35 +0200 Peter Eisentraut [EMAIL PROTECTED] wrote: Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: we should at least log such commands, and maybe disallow to anyone except Marc's pgsql account. I

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: Maybe we can do something to reduce the xid consumption? For example, reuse xids for read-only queries. Hmm, that's an idea. More simply, just keep the current transaction open (resetting everything but the XID) if we have made no changes by the

[HACKERS] Index Tuple Compression Approach?

2007-08-15 Thread Dawid Kuroczko
On 8/14/07, Chris Browne [EMAIL PROTECTED] wrote: I recently had a chat with someone who was pretty intimate with Adabas for a number of years who's in the process of figuring things out about PostgreSQL. We poked at bits of the respective implementations, seeing some similarities and

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Decibel!
On Wed, Aug 15, 2007 at 12:49:52PM -0400, Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: Maybe we can do something to reduce the xid consumption? For example, reuse xids for read-only queries. Hmm, that's an idea. More simply, just keep the current transaction open

Re: [HACKERS] Index Tuple Compression Approach?

2007-08-15 Thread Jeff Davis
On Wed, 2007-08-15 at 06:51 +0100, Heikki Linnakangas wrote: What Chris is suggesting is basically a special case of GIT, where all the heap tuples represented by an index tuple have the same key. I was actually thinking of adding a flag to index tuples to indicate that special case in GIT. We

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Tom Lane
Decibel! [EMAIL PROTECTED] writes: Aren't there potential issues with keeping the same XID if a transaction in the middle has modified data? I don't see any, as long as you take a new snapshot. Something else to think about... any app that's doing that kind of transaction rate is likely

Re: [HACKERS] Another idea for index-only scans

2007-08-15 Thread Mike Rylander
On 8/15/07, Bruce Momjian [EMAIL PROTECTED] wrote: I have added another idea for index-only scans to the TODO list: A third idea would be for a heap scan to check if all rows are visible and if so set a per-table flag which can be checked by index scans. Any change to the table would

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Darcy Buskermolen
On Wednesday 15 August 2007 09:49:52 Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: Maybe we can do something to reduce the xid consumption? For example, reuse xids for read-only queries. Hmm, that's an idea. More simply, just keep the current transaction open (resetting

Re: [HACKERS] Another idea for index-only scans

2007-08-15 Thread Josh Berkus
Bruce, I have added another idea for index-only scans to the TODO list: A third idea would be for a heap scan to check if all rows are visible and if so set a per-table flag which can be checked by index scans. Any change to the table would have to clear the flag. To detect changes

Re: [HACKERS] change name of redirect_stderr?

2007-08-15 Thread Josh Berkus
Heikki, Should we change the ordering of pg_settings? Well, an unfixed issue in pg_settings is that the category/subcategory relationship got represented by a non-atomic field which makes sorting on that difficult. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco

Re: [HACKERS] Another idea for index-only scans

2007-08-15 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: I have added another idea for index-only scans to the TODO list: A third idea would be for a heap scan to check if all rows are visible and if so set a per-table flag which can be checked by index scans. Any change to the table would have to

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Marc G. Fournier wrote: --On Wednesday, August 15, 2007 12:11:35 +0200 Peter Eisentraut [EMAIL PROTECTED] wrote: Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: we should at least log such commands, and maybe disallow to anyone except Marc's pgsql account. I don't think we should

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Heikki Linnakangas
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Is enlarging the xid field something we should consider for 8.4? No. We just got the tuple header down to 24 bytes, we are not going to give that back and then some. If you are processing 6K transactions per second, you can afford

Re: [HACKERS] Another idea for index-only scans

2007-08-15 Thread Luke Lonergan
A hybrid scan approach combined with this idea would fit nicely - provide results for tids that are directly visible and set a bit in a bitmap for those that need recheck and extend recheck to take a bitmap (wait - it already does :-) - Luke Msg is shrt cuz m on ma treo -Original

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Wednesday, August 15, 2007 20:57:14 +0200 Magnus Hagander [EMAIL PROTECTED] wrote: Marc G. Fournier wrote: --On Wednesday, August 15, 2007 12:11:35 +0200 Peter Eisentraut [EMAIL PROTECTED] wrote: Am Mittwoch, 15. August 2007 04:20

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Zoltan Boszormenyi
Tom Lane írta: It's hard to see how anyone could be doing 6K xacts/sec unless most are read-only. regards, tom lane In a recent stress test with our PostgreSQL-based cluster between two machines 3 million transaction were performed with pgbench -c 150 -t 2 -s

Re: [HACKERS] Another idea for index-only scans

2007-08-15 Thread Bruce Momjian
Gregory Stark wrote: Bruce Momjian [EMAIL PROTECTED] writes: I have added another idea for index-only scans to the TODO list: A third idea would be for a heap scan to check if all rows are visible and if so set a per-table flag which can be checked by index scans. Any change to

Re: [HACKERS] CVS corruption/mistagging?

2007-08-15 Thread Magnus Hagander
Marc G. Fournier wrote: If you're doing that, we should probably just delete the pgsql userid from the system? Or at least change it so it doesn't have 'dev' permissions. That way you can't do it wrong in that direction ;-) Seems reasonable? there is no pgsql user *on* cvs.postgresql.org,

Re: [HACKERS] Index Tuple Compression Approach?

2007-08-15 Thread Heikki Linnakangas
Dawid Kuroczko wrote: Some time ago I've had an idea that it might be possible to compress th index size, even if it is a unique index. Take the path example. My idea would be to to split indexed value to 8-byte chunks. For example: /var/lib/postgresql/8.2/main would be split into:

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Zoltan Boszormenyi
Heikki Linnakangas írta: Zoltan Boszormenyi wrote: Tom Lane írta: It's hard to see how anyone could be doing 6K xacts/sec unless most are read-only. In a recent stress test with our PostgreSQL-based cluster between two machines 3 million transaction were performed with pgbench

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Heikki Linnakangas
Zoltan Boszormenyi wrote: Tom Lane írta: It's hard to see how anyone could be doing 6K xacts/sec unless most are read-only. In a recent stress test with our PostgreSQL-based cluster between two machines 3 million transaction were performed with pgbench -c 150 -t 2 -s 200 in about _ten

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Bruce Momjian
Zoltan Boszormenyi wrote: used a real disk, the secondary used tmpfs as PGDATA. Say whatever you want about my disk lying about flush, its 75MB/sec transfer rate transfer rate is real. So 5 million real transaction in 24 hours is not unrealistic. 6k xacts / s is five *hundred*

Re: [HACKERS] Index Tuple Compression Approach?

2007-08-15 Thread Chris Browne
[EMAIL PROTECTED] (Dawid Kuroczko) writes: On 8/14/07, Chris Browne [EMAIL PROTECTED] wrote: I recently had a chat with someone who was pretty intimate with Adabas for a number of years who's in the process of figuring things out about PostgreSQL. We poked at bits of the respective

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Robert Treat
On Wednesday 15 August 2007 13:54, Tom Lane wrote: Decibel! [EMAIL PROTECTED] writes: Aren't there potential issues with keeping the same XID if a transaction in the middle has modified data? I don't see any, as long as you take a new snapshot. I'm a little confused, wouldnt the

Re: [HACKERS] Index Tuple Compression Approach?

2007-08-15 Thread Gregory Stark
Heikki Linnakangas [EMAIL PROTECTED] writes: That general approach of storing a common part leading part just once is called prefix compression. Yeah, it helps a lot on long text fields. Tree structures like file paths in particular. You kind of want to do avoid both the prefix and the

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Tom Lane
Robert Treat [EMAIL PROTECTED] writes: I'm a little confused, wouldnt the transaction that waits 30 minutes before modifying data need to get an XID that jives with the system when it's transaction started, not when it began manipulating data? Why? Would it really be safe to take a new

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-15 Thread Ron Mayer
Magnus Hagander wrote: I don't use the functional index part, but for new users I can see how that's certainly a *lot* easier. Can someone with modern hardware test to see if it's still quite a bit slower than the extra column. I had tried it too years ago; and found the functional index to

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Kevin Grittner
On Wed, Aug 15, 2007 at 5:06 PM, in message [EMAIL PROTECTED], Tom Lane [EMAIL PROTECTED] wrote: Robert Treat [EMAIL PROTECTED] writes: I'm a little confused, wouldnt the transaction that waits 30 minutes before modifying data need to get an XID that jives with the system when it's

Re: [HACKERS] XID wraparound and busy databases

2007-08-15 Thread Tom Lane
Kevin Grittner [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] wrote: You wouldn't take a new snapshot. The thought that occurs to me is that there's no reason that a transaction has to have an XID for itself before it takes a snapshot. We always special-case our own XID anyway. I'm

Re: [HACKERS] Another idea for index-only scans

2007-08-15 Thread Decibel!
On Aug 15, 2007, at 1:54 PM, Gregory Stark wrote: A third idea would be for a heap scan to check if all rows are visible and if so set a per-table flag which can be checked by index scans. Any change to the table would have to clear the flag. To detect changes during the heap scan a