Re: [HACKERS] More stats about skipped vacuums

2018-02-09 Thread Kyotaro HORIGUCHI
Hello. At Thu, 08 Feb 2018 18:21:56 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180208.182156.96551245.horiguchi.kyot...@lab.ntt.co.jp> > I suppose that the problem has not been resolved yet.. I found several bugs during studying this but my

Re: [HACKERS] More stats about skipped vacuums

2018-02-08 Thread Kyotaro HORIGUCHI
At Thu, 08 Feb 2018 18:04:15 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180208.180415.112312013.horiguchi.kyot...@lab.ntt.co.jp> > > > I suggest we remove support for dynamic_shared_memory_type = none first, > > > and see if we get any

Re: [HACKERS] More stats about skipped vacuums

2018-02-08 Thread Kyotaro HORIGUCHI
Hello, At Wed, 07 Feb 2018 16:59:20 -0500, Tom Lane wrote in <3246.1518040...@sss.pgh.pa.us> > Robert Haas writes: > > It seems to me that there was a thread where Tom proposed removing > > support for dynamic_shared_memory_type = none. > > I think

Re: [HACKERS] More stats about skipped vacuums

2018-02-07 Thread Tom Lane
Robert Haas writes: > It seems to me that there was a thread where Tom proposed removing > support for dynamic_shared_memory_type = none. I think you're recalling <32138.1502675...@sss.pgh.pa.us>, wherein I pointed out that >>> Whether that's worth the trouble is

Re: [HACKERS] More stats about skipped vacuums

2018-02-07 Thread Robert Haas
On Tue, Feb 6, 2018 at 8:34 PM, Kyotaro HORIGUCHI wrote: > Based on the reason, it fails to run when > dynamic_shared_memory_type = none and it is accompanied by > several cleanup complexities. The decision there is we should go > for just using static shared

Re: [HACKERS] More stats about skipped vacuums

2018-02-06 Thread Kyotaro HORIGUCHI
At Tue, 06 Feb 2018 19:24:37 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180206.192437.229464841.horiguchi.kyot...@lab.ntt.co.jp> > At Tue, 6 Feb 2018 14:50:01 +0900, Masahiko Sawada > wrote in

Re: [HACKERS] More stats about skipped vacuums

2018-02-06 Thread Kyotaro HORIGUCHI
At Tue, 6 Feb 2018 14:50:01 +0900, Masahiko Sawada wrote in > On Mon, Dec 11, 2017 at 8:15 PM, Kyotaro HORIGUCHI > wrote: > > I considered dshash for pgstat.c and the

Re: [HACKERS] More stats about skipped vacuums

2018-02-05 Thread Masahiko Sawada
On Mon, Dec 11, 2017 at 8:15 PM, Kyotaro HORIGUCHI wrote: > At Mon, 27 Nov 2017 13:51:22 -0500, Robert Haas wrote > in >> On Mon, Nov 27, 2017 at 1:49 AM, Kyotaro

Re: [HACKERS] More stats about skipped vacuums

2017-12-11 Thread Kyotaro HORIGUCHI
At Mon, 27 Nov 2017 13:51:22 -0500, Robert Haas wrote in > On Mon, Nov 27, 2017 at 1:49 AM, Kyotaro HORIGUCHI > wrote: > > Hmmm. Okay, we must make stats collector more

Re: [HACKERS] More stats about skipped vacuums

2017-11-28 Thread Magnus Hagander
On Tue, Nov 28, 2017 at 12:16 AM, Tom Lane wrote: > Magnus Hagander writes: > > What I've been thinking about for that one before is if we could just > > invent a protocol (shmq based maybe) whereby autovacuum can ask the stats > > collector for a single

Re: [HACKERS] More stats about skipped vacuums

2017-11-27 Thread Tom Lane
Magnus Hagander writes: > What I've been thinking about for that one before is if we could just > invent a protocol (shmq based maybe) whereby autovacuum can ask the stats > collector for a single table or index stat. If autovacuum never needs to > see a consistent view

Re: [HACKERS] More stats about skipped vacuums

2017-11-27 Thread Magnus Hagander
On Mon, Nov 27, 2017 at 7:53 PM, Robert Haas wrote: > On Sun, Nov 26, 2017 at 3:19 PM, Tom Lane wrote: > > Robert Haas writes: > >> On Sat, Nov 25, 2017 at 12:09 PM, Tom Lane wrote: > >>> Mumble. It's a

Re: [HACKERS] More stats about skipped vacuums

2017-11-27 Thread Robert Haas
On Sun, Nov 26, 2017 at 3:19 PM, Tom Lane wrote: > Robert Haas writes: >> On Sat, Nov 25, 2017 at 12:09 PM, Tom Lane wrote: >>> Mumble. It's a property I'm pretty hesitant to give up, especially >>> since the stats views have

Re: [HACKERS] More stats about skipped vacuums

2017-11-27 Thread Robert Haas
On Mon, Nov 27, 2017 at 1:49 AM, Kyotaro HORIGUCHI wrote: > Hmmm. Okay, we must make stats collector more effeicient if we > want to have additional counters with smaller significance in the > table stats. Currently sizeof(PgStat_StatTabEntry) is 168 > bytes. The

Re: [HACKERS] More stats about skipped vacuums

2017-11-26 Thread Michael Paquier
On Mon, Nov 27, 2017 at 5:19 AM, Tom Lane wrote: > Robert Haas writes: >> On Sat, Nov 25, 2017 at 12:09 PM, Tom Lane wrote: >>> Mumble. It's a property I'm pretty hesitant to give up, especially >>> since the stats views have

Re: [HACKERS] More stats about skipped vacuums

2017-11-26 Thread Michael Paquier
On Mon, Nov 27, 2017 at 2:49 AM, Tom Lane wrote: > Michael Paquier writes: >> ... Would you think >> that it is acceptable to add the number of index scans that happened >> with the verbose output then? > > I don't have an objection to it, but can't

Re: [HACKERS] More stats about skipped vacuums

2017-11-26 Thread Robert Haas
On Sat, Nov 25, 2017 at 12:09 PM, Tom Lane wrote: > Robert Haas writes: >> Of course, the other obvious question is whether we really need a >> consistent snapshot, because that's bound to be pretty expensive even >> if you eliminate the I/O cost.

Re: [HACKERS] More stats about skipped vacuums

2017-11-26 Thread Tom Lane
Michael Paquier writes: > ... Would you think > that it is acceptable to add the number of index scans that happened > with the verbose output then? I don't have an objection to it, but can't you tell that from VACUUM VERBOSE already? There should be a "INFO: scanned

Re: [HACKERS] More stats about skipped vacuums

2017-11-26 Thread Michael Paquier
On Sun, Nov 26, 2017 at 9:59 AM, Tom Lane wrote: > I'd say so ... that's something the average user will never bother with, > and even if they knew to bother, it's far from obvious what to do with > the information. Besides, I don't think you could just save the number > of

Re: [HACKERS] More stats about skipped vacuums

2017-11-25 Thread Tom Lane
Michael Paquier writes: > I am not arguing about skipped vacuuum data here (don't think much of > it by the way), but of the number of index scans done by the last > vacuum or autovacuum. This helps in tunning autovacuum_work_mem and > maintenance_work_mem. The bar is

Re: [HACKERS] More stats about skipped vacuums

2017-11-25 Thread Michael Paquier
On Sun, Nov 26, 2017 at 12:34 AM, Tom Lane wrote: > Michael Paquier writes: >> On Sat, Nov 25, 2017 at 12:55 AM, Robert Haas wrote: >>> I think that's a good thing to worry about. In the past, Tom has >>> expressed

Re: [HACKERS] More stats about skipped vacuums

2017-11-25 Thread Tom Lane
Robert Haas writes: > Of course, the other obvious question is whether we really need a > consistent snapshot, because that's bound to be pretty expensive even > if you eliminate the I/O cost. Taking a consistent snapshot across > all 100,000 tables in the database even if

Re: [HACKERS] More stats about skipped vacuums

2017-11-25 Thread Robert Haas
On Sat, Nov 25, 2017 at 10:34 AM, Tom Lane wrote: > If we could get rid of the copy-to-a-temporary-file technology for > transferring the stats collector's data to backends, then this problem > would probably vanish or at least get a lot less severe. But that seems > like a

Re: [HACKERS] More stats about skipped vacuums

2017-11-25 Thread Tom Lane
Michael Paquier writes: > On Sat, Nov 25, 2017 at 12:55 AM, Robert Haas wrote: >> I think that's a good thing to worry about. In the past, Tom has >> expressed reluctance to make stats tables that have a row per table >> any wider at all, IIRC.

Re: [HACKERS] More stats about skipped vacuums

2017-11-25 Thread Michael Paquier
On Sat, Nov 25, 2017 at 12:55 AM, Robert Haas wrote: > On Tue, Nov 21, 2017 at 2:09 AM, Kyotaro HORIGUCHI > wrote: >> Yes, my concern here is how many column we can allow in a stats >> view. I think I'm a bit too warried about that. > > I

Re: [HACKERS] More stats about skipped vacuums

2017-11-24 Thread Robert Haas
On Tue, Nov 21, 2017 at 2:09 AM, Kyotaro HORIGUCHI wrote: > Yes, my concern here is how many column we can allow in a stats > view. I think I'm a bit too warried about that. I think that's a good thing to worry about. In the past, Tom has expressed reluctance

Re: [HACKERS] More stats about skipped vacuums

2017-11-21 Thread Michael Paquier
On Wed, Nov 22, 2017 at 1:08 PM, Kyotaro HORIGUCHI wrote: > At Wed, 22 Nov 2017 08:20:22 +0900, Michael Paquier > wrote in > >> On Tue, Nov 21, 2017 at 4:09 PM,

Re: [HACKERS] More stats about skipped vacuums

2017-11-21 Thread Kyotaro HORIGUCHI
Hello, At Wed, 22 Nov 2017 08:20:22 +0900, Michael Paquier wrote in > On Tue, Nov 21, 2017 at 4:09 PM, Kyotaro HORIGUCHI > wrote: > > By the way I'm uneasy that the

Re: [HACKERS] More stats about skipped vacuums

2017-11-21 Thread Michael Paquier
On Tue, Nov 21, 2017 at 4:09 PM, Kyotaro HORIGUCHI wrote: > By the way I'm uneasy that the 'last_vacuum_index_scans' (and > vacuum_fail_count in 0002 and others in 0003, 0004) is mentioning > both VACUUM command and autovacuum, while last_vacuum and > vacuum_count

Re: [HACKERS] More stats about skipped vacuums

2017-11-20 Thread Kyotaro HORIGUCHI
Thank you for the comments. At Sat, 18 Nov 2017 22:23:20 +0900, Michael Paquier wrote in

Re: [HACKERS] More stats about skipped vacuums

2017-11-18 Thread Michael Paquier
On Thu, Nov 16, 2017 at 7:34 PM, Kyotaro HORIGUCHI wrote: > At Wed, 15 Nov 2017 16:13:01 +0900, Michael Paquier > wrote in > >>

Re: [HACKERS] More stats about skipped vacuums

2017-11-16 Thread Kyotaro HORIGUCHI
Thank you for reviewing this. At Wed, 15 Nov 2017 16:13:01 +0900, Michael Paquier wrote in > pg_stat_get_mod_since_analyze(C.oid) AS n_mod_since_analyze, > +

Re: [HACKERS] More stats about skipped vacuums

2017-11-14 Thread Michael Paquier
On Mon, Oct 30, 2017 at 8:57 PM, Kyotaro HORIGUCHI wrote: > At Thu, 26 Oct 2017 15:06:30 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20171026.150630.115694437.horiguchi.kyot...@lab.ntt.co.jp> >> At Fri, 20 Oct