Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Kamil Srot
Adrian Klaver wrote: I have a question. First a little history. Right now, the people who know better than I are fairly certain Postgres is not changing things on its own and the developer is certain the CMS software is not doing schema changes. As I understand it logging has been cranked up

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Kamil Srot
Tom Lane wrote: Erik Jones [EMAIL PROTECTED] writes: Also, I'd write a simple ping script to check for the table that runs every 5 seconds or so. I had gathered that the table was being touched constantly by his app, so that it'd be immediately obvious when it had gone away. If

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Kamil Srot
Alvaro Herrera wrote: Adrian Klaver wrote: I have a question. First a little history. Right now, the people who know better than I are fairly certain Postgres is not changing things on its own and the developer is certain the CMS software is not doing schema changes. As I understand it

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Martijn van Oosterhout
On Mon, Aug 27, 2007 at 05:58:26PM -0400, Andrew Sullivan wrote: On Mon, Aug 27, 2007 at 11:44:38PM +0200, Kamil Srot wrote: When this problem appeared for the first time, I had clearly the wraparound problem... I did vacuum it and partially restored the data... I don't think vacuum

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Richard Huxton
Kamil Srot wrote: In the version used on this particular server, there is no automatic/programing way of changing the schema. Upgrades are done manually and application itself doesn't need schema changes for routine operations... In that case, you can settle the matter by making sure your

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Kamil Srot
Richard Huxton wrote: Kamil Srot wrote: In the version used on this particular server, there is no automatic/programing way of changing the schema. Upgrades are done manually and application itself doesn't need schema changes for routine operations... In that case, you can settle the matter

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Jeff Amiel
Kamil Srot wrote: Heh, is the pool still open? Maybe I can make at least something from it :-D Current odds Application bug: even money Application configuration issue: 2-1 Rogue cron job or other maintenance process: 4-1 Somebody messing with you (or SQL injection): 8-1 XID

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Erik Jones
On Aug 28, 2007, at 9:41 AM, Kamil Srot wrote: Jeff Amiel wrote: My entire shop has set up a betting pool on the outcome of this...so I hope you post results regardless of the outcome, Kamil. Heh, is the pool still open? Maybe I can make at least something from it :-D (in all seriousness,

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Jeff Amiel
My entire shop has set up a betting pool on the outcome of this...so I hope you post results regardless of the outcome, Kamil. (in all seriousness, we hope you find/fix the problem before things get really ugly) Kamil Srot wrote: Richard Huxton wrote: Kamil Srot wrote: In the version used

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Kamil Srot
Jeff Amiel wrote: My entire shop has set up a betting pool on the outcome of this...so I hope you post results regardless of the outcome, Kamil. Heh, is the pool still open? Maybe I can make at least something from it :-D (in all seriousness, we hope you find/fix the problem before things

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Lincoln Yeoh
At 03:15 PM 8/28/2007, Kamil Srot wrote: Andrew, Alvaro... well, sure SQL injection is possibility I cannot ignore... (and sure as dad of this application, I think it's not the case :-) ... just kidding... As even the injected SQL will be shown in the logs, so we'll know more after some time.

Re: [GENERAL] Tables dissapearing

2007-08-28 Thread Raymond O'Donnell
On 28/08/2007 15:48, Jeff Amiel wrote: Alien or supernatural intervention: 18-1 Obscure postgresql bug nobody else has ever seen: 25-1 That's the sort of confidence in the DBMS we all like to see! :-) Ray. --- Raymond O'Donnell,

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Tom Lane
Kamil Srot [EMAIL PROTECTED] writes: One more thing: The project runs proprietal CMS system and there are more instances of it with the same database layout in different databases. Every time the lost table is the same one - the bussiest one (mostly read)... and everytime the lost table is

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread brian
Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: One more thing: The project runs proprietal CMS system and there are more instances of it with the same database layout in different databases. Every time the lost table is the same one - the bussiest one (mostly read)... and everytime

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
brian wrote: Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: One more thing: The project runs proprietal CMS system and there are more instances of it with the same database layout in different databases. Every time the lost table is the same one - the bussiest one (mostly read)...

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Martijn van Oosterhout
On Mon, Aug 27, 2007 at 06:37:17PM +0200, Kamil Srot wrote: I don't say, it's gone by itself, I'm asking for help debuging this situation and hopefully find a solution. For the first time it happened, it had the same symptoms - this specific table was missing and transaction counter was

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Martijn van Oosterhout wrote: On Mon, Aug 27, 2007 at 06:37:17PM +0200, Kamil Srot wrote: I don't say, it's gone by itself, I'm asking for help debuging this situation and hopefully find a solution. For the first time it happened, it had the same symptoms - this specific table was missing

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Martijn van Oosterhout wrote: On Mon, Aug 27, 2007 at 06:57:54PM +0200, Kamil Srot wrote: Correct...the script does echo vacuum full; | $PGDIR/bin/psql -U postgres $db for each database... Hope it's correct? Well, I'd drop the full part, it tends to bloat indexes. Also, did you

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Martijn van Oosterhout
On Mon, Aug 27, 2007 at 06:57:54PM +0200, Kamil Srot wrote: Correct...the script does echo vacuum full; | $PGDIR/bin/psql -U postgres $db for each database... Hope it's correct? Well, I'd drop the full part, it tends to bloat indexes. Also, did you check it was actually completing (no

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Alvaro Herrera
Kamil Srot wrote: Martijn van Oosterhout wrote: On Mon, Aug 27, 2007 at 06:57:54PM +0200, Kamil Srot wrote: Correct...the script does echo vacuum full; | $PGDIR/bin/psql -U postgres $db for each database... Hope it's correct? Well, I'd drop the full part, it tends to bloat

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Alvaro Herrera wrote: Kamil Srot wrote: Martijn van Oosterhout wrote: On Mon, Aug 27, 2007 at 06:57:54PM +0200, Kamil Srot wrote: Correct...the script does echo vacuum full; | $PGDIR/bin/psql -U postgres $db for each database... Hope it's correct? Well, I'd

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Andrew Sullivan
On Mon, Aug 27, 2007 at 07:15:44PM +0200, Kamil Srot wrote: OK, I'll drop the full part and do it less often... This doesn't address your problem, but when you move from VACUUM FULL to VACUUM, you want to do it _more_ often, not less. But given what you've posted, I am not even a little bit

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Andrew Sullivan wrote: On Mon, Aug 27, 2007 at 07:15:44PM +0200, Kamil Srot wrote: OK, I'll drop the full part and do it less often... This doesn't address your problem, but when you move from VACUUM FULL to VACUUM, you want to do it _more_ often, not less. Sure, I ment it like

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Andrew Sullivan
On Mon, Aug 27, 2007 at 10:03:04PM +0200, Kamil Srot wrote: Sure, I ment it like I'll do the FULL vacuum less often than daily and do daily the plain vacuum command. If you have your servers set up correctly, you should never need to perform VACUUM FULL. Well, I do list all databases

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Andrew Sullivan wrote: On Mon, Aug 27, 2007 at 10:03:04PM +0200, Kamil Srot wrote: Sure, I ment it like I'll do the FULL vacuum less often than daily and do daily the plain vacuum command. If you have your servers set up correctly, you should never need to perform VACUUM

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Andrew Sullivan
On Mon, Aug 27, 2007 at 10:31:11PM +0200, Kamil Srot wrote: The script is very simple one: Well, I don't see anything obvious, but. . . I can easily rewrite it to use the vacuumdb command, but I doubt it'll make any difference. The point is that you don't have to rewrite it. Just run

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Andrew Sullivan wrote: I can easily rewrite it to use the vacuumdb command, but I doubt it'll make any difference. The point is that you don't have to rewrite it. Just run vacuumdb -a and it vacuums _all_ databases. Oh, I have it now! It takes some time, but at the end, I'll

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Tom Lane
Kamil Srot [EMAIL PROTECTED] writes: # select xmin, age(xmin) from pg_class; xmin|age ---+ 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 236838019 |

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: # select xmin, age(xmin) from pg_class; xmin|age ---+ 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 236838019

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Erik Jones
On Aug 27, 2007, at 4:08 PM, Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: # select xmin, age(xmin) from pg_class; xmin|age ---+ 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647 2 | 2147483647

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Also, in your original post you mentioned a proprietal CMS system. Is this proprietary to your company or one that you've purchased? The fact that the same table going on multiple dbs all being run by that CMS system certainly makes it worthy of suspicion. This is software developed in

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread brian
Kamil Srot wrote: Erik Jones wrote Also, in your original post you mentioned a proprietal CMS system. Is this proprietary to your company or one that you've purchased? The fact that the same table going on multiple dbs all being run by that CMS system certainly makes it worthy of

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Andrew Sullivan
On Mon, Aug 27, 2007 at 11:44:38PM +0200, Kamil Srot wrote: When this problem appeared for the first time, I had clearly the wraparound problem... I did vacuum it and partially restored the data... I don't think vacuum would fix a wrap around problem in the way you describe. I don't think

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Tom Lane
Kamil Srot [EMAIL PROTECTED] writes: From above mentioned duplications of relatio names and what Tom wrote recently (doesn't see like WA problem), it looks like the relation name is/gets corrupted in some way and this corruption is internally taken over to another instance of relation

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Erik Jones
On Aug 27, 2007, at 4:44 PM, Kamil Srot wrote: Also, in your original post you mentioned a proprietal CMS system. Is this proprietary to your company or one that you've purchased? The fact that the same table going on multiple dbs all being run by that CMS system certainly makes it

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Scott Marlowe
On 8/27/07, Kamil Srot [EMAIL PROTECTED] wrote: I really need to make sure, the next problem (if it's necessary to let it happen) will be the last one... :-( Have you set your log_statement to 'ddl' in postgresql.conf? That way you'll have a log of every single ddl that goes through the

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
brian wrote: Kamil Srot wrote: Erik Jones wrote Also, in your original post you mentioned a proprietal CMS system. Is this proprietary to your company or one that you've purchased? The fact that the same table going on multiple dbs all being run by that CMS system certainly makes it

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Andrew Sullivan wrote: On Mon, Aug 27, 2007 at 11:44:38PM +0200, Kamil Srot wrote: When this problem appeared for the first time, I had clearly the wraparound problem... I did vacuum it and partially restored the data... I don't think vacuum would fix a wrap around problem in the way

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread brian
Kamil Srot wrote: brian wrote: IMHO, it's not at all improbable, given that this software is connecting to the same databases you are seeing affected by this phenomenon. Not to mention that it's proprietary, so fewer eyes have gone over it.[1] I suggested earlier grepping for 'drop' in your

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Erik Jones wrote: On Aug 27, 2007, at 4:44 PM, Kamil Srot wrote: Also, in your original post you mentioned a proprietal CMS system. Is this proprietary to your company or one that you've purchased? The fact that the same table going on multiple dbs all being run by that CMS system

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: From above mentioned duplications of relatio names and what Tom wrote recently (doesn't see like WA problem), it looks like the relation name is/gets corrupted in some way and this corruption is internally taken over to another instance

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
brian wrote: Just wondering: what's the name of this table? What's its function? I think you mentioned that it's pretty well static. But what is it holding? Maybe that'll provide a clue as to where to look. The system is actually management of websiteeshop with webbased UI ... the table is

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Tom Lane
Kamil Srot [EMAIL PROTECTED] writes: Erik Jones wrote: Have you verified that the table's files are still on disk after it's disappeared? Do not have any idea how to do it... I wasn't able to access it using any DML/DDL commands... can try it on a binary backup of the damaged DB if

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Kamil Srot
Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: Erik Jones wrote: Have you verified that the table's files are still on disk after it's disappeared? Do not have any idea how to do it... I wasn't able to access it using any DML/DDL commands... can try it on a binary

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Erik Jones
On Aug 27, 2007, at 7:57 PM, Kamil Srot wrote: Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: Erik Jones wrote: Have you verified that the table's files are still on disk after it's disappeared? Do not have any idea how to do it... I wasn't able to access it using any DML/DDL

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Alvaro Herrera
Adrian Klaver wrote: I have a question. First a little history. Right now, the people who know better than I are fairly certain Postgres is not changing things on its own and the developer is certain the CMS software is not doing schema changes. As I understand it logging has been cranked

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Adrian Klaver
On Monday 27 August 2007 5:57 pm, Kamil Srot wrote: Tom Lane wrote: Kamil Srot [EMAIL PROTECTED] writes: Erik Jones wrote: Have you verified that the table's files are still on disk after it's disappeared? Do not have any idea how to do it... I wasn't able to access it using any

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Tom Lane
Erik Jones [EMAIL PROTECTED] writes: Also, I'd write a simple ping script to check for the table that runs every 5 seconds or so. I had gathered that the table was being touched constantly by his app, so that it'd be immediately obvious when it had gone away. If that's not so, then definitely

Re: [GENERAL] Tables dissapearing

2007-08-27 Thread Andrew Sullivan
On Tue, Aug 28, 2007 at 02:03:29AM +0200, Kamil Srot wrote: The system is actually management of websiteeshop with webbased UI ... Um, are you sure you don't have a SQL-injection problem, and someone is doing Something Bad to you? A -- Andrew Sullivan | [EMAIL PROTECTED] The fact that