Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2015-01-20 Thread Robert Haas
On Mon, Jan 19, 2015 at 10:53 PM, tim_wilson tim.wil...@telogis.com wrote: Was slightly optimistic that this comment in the release notes might mean that my bug with bloat on hot tables might have been fixed in 9.4 /Make VACUUM properly report dead but not-yet-removable rows to the statistics

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2015-01-19 Thread tim_wilson
Was slightly optimistic that this comment in the release notes might mean that my bug with bloat on hot tables might have been fixed in 9.4 /Make VACUUM properly report dead but not-yet-removable rows to the statistics collector (Hari Babu) Previously these were reported as live rows./

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-09-10 Thread Bruce Momjian
On Thu, Jun 12, 2014 at 01:40:59PM +0200, Andres Freund wrote: Hi Tom, On 2014-06-06 15:44:25 -0400, Tom Lane wrote: I figured it'd be easy enough to get a better estimate by adding another counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively assuming that

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-25 Thread tim_wilson
Given that this seems to have slipped off the hackers radar (or in too hard basket) I have constructed a horrible solution. I will stop using autovacuum for this relation , I will use our own system to monitor the relation, and I will reset pgclass.reltuples on this relation after vacuum is done

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-12 Thread Andres Freund
Hi Tom, On 2014-06-06 15:44:25 -0400, Tom Lane wrote: I figured it'd be easy enough to get a better estimate by adding another counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively assuming that in-progress inserts and deletes will both commit). Did you plan to backpatch

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-11 Thread Andres Freund
On 2014-06-09 11:24:22 -0700, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: I am not sure, given predicate.c's coding, how HEAPTUPLE_DELETE_IN_PROGRESS could cause problems. Could you elaborate, since that's the

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-11 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: On 2014-06-09 11:24:22 -0700, Kevin Grittner wrote: The only way that it could be a problem is if the DELETE is in a subtransaction which might get rolled back without rolling back the INSERT. The way I understand the code in that case the subxid

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Amit Kapila
On Sat, Jun 7, 2014 at 1:28 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-06 15:44:25 -0400, Tom Lane wrote: I figured it'd be easy enough to get a better estimate by adding another counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively assuming that

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Robert Haas
On Fri, Jun 6, 2014 at 3:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: It did not use to blow this question off: back around 8.3 you got DELETE_IN_PROGRESS if the tuple had a delete pending. I think we need less laziness + fuzzy thinking here. Maybe we should have a separate

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 10:14:32 -0400, Robert Haas wrote: On Fri, Jun 6, 2014 at 3:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: It did not use to blow this question off: back around 8.3 you got DELETE_IN_PROGRESS if the tuple had a delete pending. I think we need less laziness + fuzzy thinking here.

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-06-09 10:14:32 -0400, Robert Haas wrote: I think that would be a good idea for conceptual clarity if nothing else. If callers are OK with it, then they can treat both of those codes alike; but then at least there's clear evidence as to the

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 10:30:43 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-06-09 10:14:32 -0400, Robert Haas wrote: I think that would be a good idea for conceptual clarity if nothing else. If callers are OK with it, then they can treat both of those codes alike;

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: Well, I think reverting surely wouldn't be the right cure. It fixes a somewhat nasty bug. I'd certainly be prepared to add the two lines necessary to make it return DELETE_IN_PROGRESS after trying to understand Kevin's email about predicate.c and

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 08:00:52 -0700, Kevin Grittner wrote: I'm not actually sure yet whether the current state of affairs causes a problem for the serializable transaction isolation level implementation. I'd replied in the other thread before noticing you'd replied here... From what I understand

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: On 2014-06-09 08:00:52 -0700, Kevin Grittner wrote: I tried to make things a bit clearer there - but I am not sure I've succeed. I'm certainly willing to explain things further if you can tell me which are is unclear. Thanks!  IMO, something like

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: HEAPTUPLE_INSERT_IN_PROGRESS,    /* inserting xact is still in progress */ HEAPTUPLE_DELETE_IN_PROGRESS    /* deleting xact is still in progress */ the current code will return

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: For purposes of predicate.c, if the also deleted activity might be rolled back without rolling back the insert, INSERT_IN_PROGRESS is the only correct value. That's basically the argument for the

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: I am not sure, given predicate.c's coding, how HEAPTUPLE_DELETE_IN_PROGRESS could cause problems. Could you elaborate, since that's the contentious point with Tom? Since 'both in progress' can

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: The reason this stuff is not too carefully spec'd is that when HTSV was written, there was no expectation that there was any correctness issue around which of these cases was returned.  I wonder whether SSI should be using HTSV at all. That's certainly a

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Alvaro Herrera
Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: old: 1) xmin has committed, xmax is in progress, xmax is not just a locker 2) xmin is in progress, xmax is set and not not just a locker Note that the 2) case here

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Assuming that Kevin's describing his needs correctly, we could resolve this by inventing HEAPTUPLE_INSERT_AND_DELETE_IN_PROGRESS.  VACUUM could assume that that's a probably-dead tuple, while SSI could do something different. That could work. On the other

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-06 Thread Andres Freund
On 2014-06-06 15:44:25 -0400, Tom Lane wrote: I figured it'd be easy enough to get a better estimate by adding another counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively assuming that in-progress inserts and deletes will both commit). I did that, and found that it