Re: [HACKERS] truncating pg_multixact/members

2014-02-13 Thread Andres Freund
On 2014-02-12 17:40:44 -0300, Alvaro Herrera wrote: Also, AutoVacOpts (used as part of reloptions) gained three extra fields. Since this is in the middle of StdRdOptions, it'd be somewhat more involve to put these at the end of that struct. This might be a problem if somebody has a

Re: [HACKERS] truncating pg_multixact/members

2014-02-13 Thread Alvaro Herrera
Andres Freund escribió: On 2014-02-12 17:40:44 -0300, Alvaro Herrera wrote: Also, AutoVacOpts (used as part of reloptions) gained three extra fields. Since this is in the middle of StdRdOptions, it'd be somewhat more involve to put these at the end of that struct. This might be a

Re: [HACKERS] truncating pg_multixact/members

2014-02-13 Thread Alvaro Herrera
Alvaro Herrera escribió: Yes, that's what I did --- see the attached patch, which I would apply on top of the code for master and would be only in 9.3. (Of course, these changes affect other parts of the code, in particular autovacuum.c and reloptions.c. But that's not important here). --

Re: [HACKERS] truncating pg_multixact/members

2014-02-13 Thread Andres Freund
On 2014-02-13 14:40:39 -0300, Alvaro Herrera wrote: Andres Freund escribió: On 2014-02-12 17:40:44 -0300, Alvaro Herrera wrote: Also, AutoVacOpts (used as part of reloptions) gained three extra fields. Since this is in the middle of StdRdOptions, it'd be somewhat more involve to

Re: [HACKERS] truncating pg_multixact/members

2014-02-13 Thread Alvaro Herrera
Alvaro Herrera escribió: So here are two patches -- the first one, for 9.3 and HEAD, introduce the new aging variables and use them throughout vacuum and autovacuum, including per-table options; the second one adjusts the struct declarations to avoid the ABI break in VacuumStmt and

Re: [HACKERS] truncating pg_multixact/members

2014-02-13 Thread Alvaro Herrera
Alvaro Herrera escribió: So here are two patches -- the first one, for 9.3 and HEAD, introduce the new aging variables and use them throughout vacuum and autovacuum, including per-table options; the second one adjusts the struct declarations to avoid the ABI break in VacuumStmt and

Re: [HACKERS] truncating pg_multixact/members

2014-02-12 Thread Alvaro Herrera
In this new version, I added a couple of fields to VacuumStmt node. How strongly do we feel this would cause an ABI break? Would we be more comfortable if I put them at the end of the struct for 9.3 instead? Do we expect third-party code to be calling vacuum()? Also, AutoVacOpts (used as part

Re: [HACKERS] truncating pg_multixact/members

2014-02-12 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: In this new version, I added a couple of fields to VacuumStmt node. How strongly do we feel this would cause an ABI break? Would we be more comfortable if I put them at the end of the struct for 9.3 instead? In the past we've usually added such

Re: [HACKERS] truncating pg_multixact/members

2014-02-12 Thread Alvaro Herrera
Tom Lane escribió: Alvaro Herrera alvhe...@2ndquadrant.com writes: In this new version, I added a couple of fields to VacuumStmt node. How strongly do we feel this would cause an ABI break? Would we be more comfortable if I put them at the end of the struct for 9.3 instead? In the past

Re: [HACKERS] truncating pg_multixact/members

2014-02-11 Thread Robert Haas
On Tue, Feb 11, 2014 at 5:16 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: On Mon, Jan 20, 2014 at 1:39 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: * I haven't introduced settings to tweak this per table for autovacuum. I don't think those are needed.

Re: [HACKERS] truncating pg_multixact/members

2014-01-21 Thread Robert Haas
On Mon, Jan 20, 2014 at 1:39 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: * I haven't introduced settings to tweak this per table for autovacuum. I don't think those are needed. It's not hard to do, however; if people opine against this, I will implement that. I can't think of any

Re: [HACKERS] truncating pg_multixact/members

2014-01-20 Thread Alvaro Herrera
Robert Haas escribió: On Fri, Jan 3, 2014 at 9:11 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Yeah, this stuff is definitely underdocumented relative to vacuum right now. I have added a paragraph or two. It's a (probably insufficient) start. I would like to add a sample query to

Re: [HACKERS] truncating pg_multixact/members

2014-01-20 Thread Alvaro Herrera
Alvaro Herrera escribió: Here's a first cut at this. Note I have omitted a setting equivalent to autovacuum_freeze_max_age, but I think we should have one too. Some more comments on the patch: * I haven't introduced settings to tweak this per table for autovacuum. I don't think those are

Re: [HACKERS] truncating pg_multixact/members

2014-01-20 Thread Andres Freund
Hi, On 2014-01-20 15:39:33 -0300, Alvaro Herrera wrote: * The multixact_freeze_table_age value has been set to 5 million. I feel this is a big enough number that shouldn't cause too much vacuuming churn, while at the same time not leaving excessive storage occupied by pg_multixact/members,

Re: [HACKERS] truncating pg_multixact/members

2014-01-07 Thread Andres Freund
On 2014-01-06 20:51:57 -0500, Robert Haas wrote: On Mon, Jan 6, 2014 at 7:50 PM, Jim Nasby j...@nasby.net wrote: On 1/4/14, 8:19 AM, Robert Haas wrote: Also, while multixactid_freeze_min_age should be low, perhaps a million as you suggest, multixactid_freeze_table_age should NOT be

Re: [HACKERS] truncating pg_multixact/members

2014-01-06 Thread Robert Haas
On Sat, Jan 4, 2014 at 12:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: As far as back-patching the GUCs, my thought would be to back-patch them but mark them GUC_NOT_IN_SAMPLE in 9.3, so we don't have to touch the default postgresql.conf. That seems

Re: [HACKERS] truncating pg_multixact/members

2014-01-06 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sat, Jan 4, 2014 at 12:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Keep in mind that 9.3 is still wet behind the ears and many many people haven't adopted it yet. If we do what you're suggesting then we're creating a completely useless inconsistency

Re: [HACKERS] truncating pg_multixact/members

2014-01-06 Thread Robert Haas
On Mon, Jan 6, 2014 at 2:53 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jan 4, 2014 at 12:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Keep in mind that 9.3 is still wet behind the ears and many many people haven't adopted it yet. If we do what you're

Re: [HACKERS] truncating pg_multixact/members

2014-01-06 Thread Jim Nasby
On 1/4/14, 8:19 AM, Robert Haas wrote: Also, while multixactid_freeze_min_age should be low, perhaps a million as you suggest, multixactid_freeze_table_age should NOT be lowered to 3 million or anything like it. If you do that, people who are actually doing lots of row locking will start

Re: [HACKERS] truncating pg_multixact/members

2014-01-06 Thread Robert Haas
On Mon, Jan 6, 2014 at 7:50 PM, Jim Nasby j...@nasby.net wrote: On 1/4/14, 8:19 AM, Robert Haas wrote: Also, while multixactid_freeze_min_age should be low, perhaps a million as you suggest, multixactid_freeze_table_age should NOT be lowered to 3 million or anything like it. If you do that,

Re: [HACKERS] truncating pg_multixact/members

2014-01-04 Thread Robert Haas
On Fri, Jan 3, 2014 at 9:11 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas escribió: On Mon, Dec 30, 2013 at 10:59 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: One problem I see is length of time before freezing multis: they live for far too long, causing the SLRU

Re: [HACKERS] truncating pg_multixact/members

2014-01-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: As far as back-patching the GUCs, my thought would be to back-patch them but mark them GUC_NOT_IN_SAMPLE in 9.3, so we don't have to touch the default postgresql.conf. That seems bizarre and pointless. Keep in mind that 9.3 is still wet behind the

Re: [HACKERS] truncating pg_multixact/members

2014-01-03 Thread Alvaro Herrera
Robert Haas escribió: On Mon, Dec 30, 2013 at 10:59 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: One problem I see is length of time before freezing multis: they live for far too long, causing the SLRU files to eat way too much disk space. I ran burnmulti in a loop, creating multis

Re: [HACKERS] truncating pg_multixact/members

2014-01-03 Thread Andres Freund
Hi, On 2014-01-03 11:11:13 -0300, Alvaro Herrera wrote: Yeah. Since we expect mxids to be composed at a much lower rate than xids, we can keep pg_multixact small without needing to increase the rate of full table scans. I don't think that's necessarily true - there have been several

Re: [HACKERS] truncating pg_multixact/members

2014-01-02 Thread Robert Haas
On Mon, Dec 30, 2013 at 10:59 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: One problem I see is length of time before freezing multis: they live for far too long, causing the SLRU files to eat way too much disk space. I ran burnmulti in a loop, creating multis of 3 members each, with a

Re: [HACKERS] truncating pg_multixact/members

2013-12-30 Thread Alvaro Herrera
Alvaro Herrera wrote: 1. slru.c doesn't consider file names longer than 4 hexadecimal chars. For 9.3, I propose we skip this and tweak the code to consider files whose names are 4 or 5 chars in length, to remain compatible with existing installations that have pg_multixact/member having a

Re: [HACKERS] truncating pg_multixact/members

2013-12-28 Thread Kevin Grittner
Alvaro Herrera alvhe...@2ndquadrant.com wrote: 1. slru.c doesn't consider file names longer than 4 hexadecimal chars. Fixing (1) is simple: we can have each SLRU user declare how many digits to have in file names.  All existing users but pg_multixact/members should declare 4 digits; that one

Re: [HACKERS] truncating pg_multixact/members

2013-12-28 Thread Alvaro Herrera
Kevin Grittner wrote: Alvaro Herrera alvhe...@2ndquadrant.com wrote: 1. slru.c doesn't consider file names longer than 4 hexadecimal chars. Fixing (1) is simple: we can have each SLRU user declare how many digits to have in file names.  All existing users but pg_multixact/members

[HACKERS] truncating pg_multixact/members

2013-12-27 Thread Alvaro Herrera
I started looking at bug #8673 some days ago, and I identified three separate issues that need fixing: 1. slru.c doesn't consider file names longer than 4 hexadecimal chars. 2. pg_multixact/members truncation requires more intelligence to avoid removing files that are still needed. Right now we