[GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Michael Graham
Hi all, I have an application that is reading from a queue table, as part of my testing I stressed the table to check performance, but after the test was completed I have the a very large empty table: SELECT relname, pg_size_pretty(pg_relation_size(oid)) AS size, reltuples::bigint FROM pg_class;

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Andy Colson
On 8/3/2011 4:47 AM, Michael Graham wrote: Hi all, I have an application that is reading from a queue table, as part of my testing I stressed the table to check performance, but after the test was completed I have the a very large empty table: SELECT relname,

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Scott Marlowe
On Wed, Aug 3, 2011 at 8:03 AM, Andy Colson a...@squeakycode.net wrote: If you have lots and lots of tables, autovacuum only checks one at a time, then wait's a bit.  Did you run your test for several days? Not true. autovac naps by default 1 minute between each db. i.e. if you have 5 dbs it

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Tom Lane
Michael Graham mgra...@bloxx.com writes: From reading the documentation I see that postgres would return this space to that system after a normal vacuum if one or more pages at the end of a table become entirely free and an exclusive table lock can be easily obtained. What does easily

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Sergey Konoplev
On 3 August 2011 18:17, Tom Lane t...@sss.pgh.pa.us wrote: Would my applications constant polling of the queue mean that the lock could not be easily obtained? Very possible, depending on what duty cycle is involved there. Is there any ways of guaranteed concurrent obtaining it?          

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Michael Graham
On Wed, 2011-08-03 at 09:03 -0500, Andy Colson wrote: Depending on how long you ran your test, and the conf settings, and the size of your database, autovacuum may never have even tried. I know that the vacuum is definitely running (in fact isn't it the vacuum that set the reltuples to 0?), the

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Michael Graham
On Wed, 2011-08-03 at 10:17 -0400, Tom Lane wrote: Michael Graham mgra...@bloxx.com writes: Would my applications constant polling of the queue mean that the lock could not be easily obtained? Very possible, depending on what duty cycle is involved there. Hmm. The clients aren't that

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Pavan Deolasee
On Wed, Aug 3, 2011 at 10:57 AM, Michael Graham mgra...@bloxx.com wrote: On Wed, 2011-08-03 at 10:17 -0400, Tom Lane wrote: Michael Graham mgra...@bloxx.com writes: Would my applications constant polling of the queue mean that the lock could not be easily obtained? Very possible,

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Scott Marlowe
On Wed, Aug 3, 2011 at 8:57 AM, Michael Graham mgra...@bloxx.com wrote: On Wed, 2011-08-03 at 10:17 -0400, Tom Lane wrote: Michael Graham mgra...@bloxx.com writes: Would my applications constant polling of the queue mean that the lock could not be easily obtained? Very possible,

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Bill Moran
In response to Michael Graham mgra...@bloxx.com: On Wed, 2011-08-03 at 10:17 -0400, Tom Lane wrote: Michael Graham mgra...@bloxx.com writes: Would my applications constant polling of the queue mean that the lock could not be easily obtained? Very possible, depending on what duty

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Tom Lane
Pavan Deolasee pavan.deola...@gmail.com writes: On Wed, Aug 3, 2011 at 10:57 AM, Michael Graham mgra...@bloxx.com wrote: On Wed, 2011-08-03 at 10:17 -0400, Tom Lane wrote: Michael Graham mgra...@bloxx.com writes: Would my applications constant polling of the queue mean that the lock could not

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Michael Graham
On Wed, 2011-08-03 at 11:40 -0400, Tom Lane wrote: The other problem is that once autovacuum has gotten the lock, it has to keep it for long enough to re-scan the truncatable pages (to make sure they're still empty). And it is set up so that any access to the table will kick autovacuum off

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Tom Lane
Michael Graham mgra...@bloxx.com writes: Ah! This looks like it is very much the issue. Since I've got around 150GB of data that should be truncatable and a select every ~2s. Just to confirm would postgres write: 2011-08-03 16:09:55 BST ERROR: canceling autovacuum task 2011-08-03

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Jerry Sievers
Michael Graham mgra...@bloxx.com writes: On Wed, 2011-08-03 at 10:17 -0400, Tom Lane wrote: Michael Graham mgra...@bloxx.com writes: Would my applications constant polling of the queue mean that the lock could not be easily obtained? Very possible, depending on what duty cycle is

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Eduardo Morras
At 16:35 03/08/2011, Michael Graham wrote: Yeah it said it last ran yesterday (and is currently running now), but I did I notice in the log: 2011-08-02 19:43:35 BST ERROR: canceling autovacuum task 2011-08-02 19:43:35 BST CONTEXT: automatic vacuum of table traffic.public.logdata5queue Which

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread John R Pierce
On 08/03/11 10:21 AM, Eduardo Morras wrote: One question, while you run your tests, does IDLE IN TRANSACTION messages happen? If you run your tests with a permanent connection to database, the tables are locked and autovacuum cannot work. its not that tables are locked, its that vacuum (auto

Re: [GENERAL] Vacuum as easily obtained locks

2011-08-03 Thread Eduardo Morras
At 19:32 03/08/2011, you wrote: On 08/03/11 10:21 AM, Eduardo Morras wrote: One question, while you run your tests, does IDLE IN TRANSACTION messages happen? If you run your tests with a permanent connection to database, the tables are locked and autovacuum cannot work. its not that tables