Re: Protect syscache from bloating with negative cache entries

At Wed, 13 Feb 2019 02:15:42 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB97CF1@G01JPEXMBYT05> > From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > > I didin't consider planning that happen within a function. If > > > 5min is the default for

Re: Protect syscache from bloating with negative cache entries

At Tue, 12 Feb 2019 20:36:28 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190212.203628.118792892.horiguchi.kyot...@lab.ntt.co.jp> > > (4) > > + hash_size = cp->cc_nbuckets * sizeof(dlist_head); > > + tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len; > > +

RE: Protect syscache from bloating with negative cache entries

From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > I didin't consider planning that happen within a function. If > > 5min is the default for catalog_cache_prune_min_age, 10% of it > > (30s) seems enough and gettieofday() with such intervals wouldn't > > affect forground jobs. I'd choose

RE: Protect syscache from bloating with negative cache entries

From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > I'm at a loss how call syscache for users. I think it is "catalog > cache". The most basic component is called catcache, which is > covered by the syscache layer, both of then are not revealed to > users, and it is shown to user as

Re: Protect syscache from bloating with negative cache entries

On 2/12/19 12:35 PM, Kyotaro HORIGUCHI wrote: > Thank you for testing and the commits, Tomas. > > At Sat, 9 Feb 2019 19:09:59 +0100, Tomas Vondra > wrote in > <74386116-0bc5-84f2-e614-0cff19aca...@2ndquadrant.com> >> On 2/7/19 1:18 PM, Kyotaro HORIGUCHI wrote: >>> At Thu, 07 Feb 2019

Re: Protect syscache from bloating with negative cache entries

At Tue, 12 Feb 2019 01:02:39 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB972A6@G01JPEXMBYT05> > From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > > Recuded frequency of dlist_move_tail by taking 1ms interval between two > > succesive updates on the

Re: Protect syscache from bloating with negative cache entries

Thank you for testing and the commits, Tomas. At Sat, 9 Feb 2019 19:09:59 +0100, Tomas Vondra wrote in <74386116-0bc5-84f2-e614-0cff19aca...@2ndquadrant.com> > On 2/7/19 1:18 PM, Kyotaro HORIGUCHI wrote: > > At Thu, 07 Feb 2019 15:24:18 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > > wrote

Re: Protect syscache from bloating with negative cache entries

At Fri, 8 Feb 2019 09:42:20 +, "Ideriha, Takeshi" wrote in <4E72940DA2BF16479384A86D54D0988A6F41EDD1@G01JPEXMBKW04> > >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > >I made a rerun of benchmark using "-S -T 30" on the server build with no > >assertion and > >-O2. The

RE: Protect syscache from bloating with negative cache entries

From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > I meant that the time-based eviction is not very good, because it > > could cause less frequently entries to vanish even when memory is not > > short. Time-based eviction reminds me of Memcached, Redis, DNS, etc. > > that evicts

Re: Protect syscache from bloating with negative cache entries

On 2/12/19 1:49 AM, Tsunakawa, Takayuki wrote: > From: Tomas Vondra >> I'm not sure what you mean by "necessary" and "unnecessary" here. What >> matters is how often an entry is accessed - if it's accessed often, it makes >> sense >> to keep it in the cache. Otherwise evict it. Entries not

RE: Protect syscache from bloating with negative cache entries

From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > Recuded frequency of dlist_move_tail by taking 1ms interval between two > succesive updates on the same entry let the degradation dissapear. > > patched : 13720 tps (+2%) What do you think contributed to this performance

RE: Protect syscache from bloating with negative cache entries

From: Tomas Vondra > I'm not sure what you mean by "necessary" and "unnecessary" here. What > matters is how often an entry is accessed - if it's accessed often, it makes > sense > to keep it in the cache. Otherwise evict it. Entries not accessed for 5 > minutes are > clearly not accessed very

Re: Protect syscache from bloating with negative cache entries

On 2/7/19 1:18 PM, Kyotaro HORIGUCHI wrote: > At Thu, 07 Feb 2019 15:24:18 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20190207.152418.139132570.horiguchi.kyot...@lab.ntt.co.jp> >> I'm going to retake numbers with search-only queries. > > Yeah, I was stupid. > > I made a

Re: Protect syscache from bloating with negative cache entries

On 2/8/19 2:27 PM, MauMau wrote: > From: Tomas Vondra >> I don't think we need to remove the expired entries right away, if >> there are only very few of them. The cleanup requires walking the >> hash table, which means significant fixed cost. So if there are >> only few expired entries (say,

Re: Protect syscache from bloating with negative cache entries

From: Tomas Vondra > I don't think we need to remove the expired entries right away, if there > are only very few of them. The cleanup requires walking the hash table, > which means significant fixed cost. So if there are only few expired > entries (say, less than 25% of the cache), we can just

RE: Protect syscache from bloating with negative cache entries

>From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] >I made a rerun of benchmark using "-S -T 30" on the server build with no >assertion and >-O2. The numbers are the best of three successive attempts. The patched >version is >running with cache_target_memory = 0,

Re: Protect syscache from bloating with negative cache entries

At Thu, 07 Feb 2019 15:24:18 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190207.152418.139132570.horiguchi.kyot...@lab.ntt.co.jp> > I'm going to retake numbers with search-only queries. Yeah, I was stupid. I made a rerun of benchmark using "-S -T 30" on the server build with no

RE: Protect syscache from bloating with negative cache entries

Hi, thanks for recent rapid work. >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] >At Tue, 5 Feb 2019 19:05:26 -0300, Alvaro Herrera >wrote in <20190205220526.GA1442@alvherre.pgsql> >> On 2019-Feb-05, Tomas Vondra wrote: >> >> > I don't think we need to remove the expired

Re: Protect syscache from bloating with negative cache entries

At Tue, 5 Feb 2019 19:05:26 -0300, Alvaro Herrera wrote in <20190205220526.GA1442@alvherre.pgsql> > On 2019-Feb-05, Tomas Vondra wrote: > > > I don't think we need to remove the expired entries right away, if there > > are only very few of them. The cleanup requires walking the hash table, > >

Re: Protect syscache from bloating with negative cache entries

On 2019-02-06 17:37:04 +0900, Kyotaro HORIGUCHI wrote: > At Wed, 06 Feb 2019 15:16:53 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20190206.151653.117382256.horiguchi.kyot...@lab.ntt.co.jp> > > > The two should have the same extent of impact on performance when > > > disabled.

Re: Protect syscache from bloating with negative cache entries

At Wed, 06 Feb 2019 15:16:53 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190206.151653.117382256.horiguchi.kyot...@lab.ntt.co.jp> > > The two should have the same extent of impact on performance when > > disabled. I'll take numbers briefly using pgbench. (pgbench -j 10 -c 10 -T

Re: Protect syscache from bloating with negative cache entries

At Wed, 06 Feb 2019 14:43:34 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190206.144334.193118280.horiguchi.kyot...@lab.ntt.co.jp> > At Tue, 5 Feb 2019 02:40:35 +, "Tsunakawa, Takayuki" > wrote in > <0A3221C70F24FB45833433255569204D1FB93A16@G01JPEXMBYT05> > > From:

Re: Protect syscache from bloating with negative cache entries

At Tue, 5 Feb 2019 02:40:35 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB93A16@G01JPEXMBYT05> > From: br...@momjian.us [mailto:br...@momjian.us] > > On Mon, Feb 4, 2019 at 08:23:39AM +, Tsunakawa, Takayuki wrote: > > > Horiguchi-san, Bruce, all, So, why don't we

Re: Protect syscache from bloating with negative cache entries

On 2/5/19 11:05 PM, Alvaro Herrera wrote: > On 2019-Feb-05, Tomas Vondra wrote: > >> I don't think we need to remove the expired entries right away, if there >> are only very few of them. The cleanup requires walking the hash table, >> which means significant fixed cost. So if there are only few

Re: Protect syscache from bloating with negative cache entries

On 2019-Feb-05, Tomas Vondra wrote: > I don't think we need to remove the expired entries right away, if there > are only very few of them. The cleanup requires walking the hash table, > which means significant fixed cost. So if there are only few expired > entries (say, less than 25% of the

Re: Protect syscache from bloating with negative cache entries

On 1/21/19 9:56 PM, Bruce Momjian wrote: > On Fri, Jan 18, 2019 at 05:09:41PM -0800, Andres Freund wrote: >> Hi, >> >> On 2019-01-18 19:57:03 -0500, Robert Haas wrote: >>> On Fri, Jan 18, 2019 at 4:23 PM and...@anarazel.de >>> wrote: My proposal for this was to attach a 'generation' to

Re: Protect syscache from bloating with negative cache entries

Hi, I find it a bit surprising there are almost no results demonstrating the impact of the proposed changes on some typical workloads. It touches code (syscache, ...) that is quite sensitive performance-wise, and adding even just a little bit of overhead may hurt significantly. Even on systems

RE: Protect syscache from bloating with negative cache entries

>From: br...@momjian.us [mailto:br...@momjian.us] >On Mon, Feb 4, 2019 at 08:23:39AM +, Tsunakawa, Takayuki wrote: >> Horiguchi-san, Bruce, all, So, why don't we make >> syscache_memory_target the upper limit on the total size of all >> catcaches, and rethink the past LRU management? > >I was

RE: Protect syscache from bloating with negative cache entries

From: br...@momjian.us [mailto:br...@momjian.us] > On Mon, Feb 4, 2019 at 08:23:39AM +, Tsunakawa, Takayuki wrote: > > Horiguchi-san, Bruce, all, So, why don't we make > > syscache_memory_target the upper limit on the total size of all > > catcaches, and rethink the past LRU management? > > I

Re: Protect syscache from bloating with negative cache entries

On Mon, Feb 4, 2019 at 08:23:39AM +, Tsunakawa, Takayuki wrote: > Horiguchi-san, Bruce, all, So, why don't we make > syscache_memory_target the upper limit on the total size of all > catcaches, and rethink the past LRU management? I was going to say that our experience with LRU has been that

RE: Protect syscache from bloating with negative cache entries

Horiguchi-san, Bruce, all, I hesitate to say this, but I think there are the following problems with the proposed approach: 1) Tries to prune the catalog tuples only when the hash table is about to expand. If no tuple is found to be eligible for eviction at first and the hash table expands,

RE: Protect syscache from bloating with negative cache entries

Horiguchi-san, Bruce, Thank you for telling me your ideas behind this feature. Frankly, I don't think I understood the proposed specification is OK, but I can't explain it well at this instant. So, let me discuss that in a subsequent mail. Anyway, here are my review comments on 0001: (1)

Re: Protect syscache from bloating with negative cache entries

On Mon, Jan 28, 2019 at 01:31:43PM +0900, Kyotaro HORIGUCHI wrote: > I'll consider the last choice and will come up with a patch. Update is recent, so I have just moved the patch to next CF. -- Michael signature.asc Description: PGP signature

Re: Protect syscache from bloating with negative cache entries

At Wed, 30 Jan 2019 05:06:30 +, "Ideriha, Takeshi" wrote in <4E72940DA2BF16479384A86D54D0988A6F4156D4@G01JPEXMBKW04> > >You don't have a direct control on syscache memory usage. When you find a > >queriy > >slowed by the default cache expiration, you can set cache_memory_taret to > >keep

RE: Protect syscache from bloating with negative cache entries

Hi >> > I suggest you go with just syscache_prune_min_age, get that into PG >> > 12, and we can then reevaluate what we need. If you want to >> > hard-code a minimum cache size where no pruning will happen, maybe >> > based on the system catalogs or typical load, that is fine. >> >> Please

Re: Protect syscache from bloating with negative cache entries

At Fri, 25 Jan 2019 07:26:46 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB70E6B@G01JPEXMBYT05> > From: Robert Haas [mailto:robertmh...@gmail.com] > > On Thu, Jan 24, 2019 at 10:02 AM Tom Lane wrote: > > > I will argue hard that we should not do it at all, ever. > >

Re: Protect syscache from bloating with negative cache entries

At Fri, 25 Jan 2019 08:14:19 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB70EFB@G01JPEXMBYT05> > Hi Horiguchi-san, Bruce, > > From: Bruce Momjian [mailto:br...@momjian.us] > > I suggest you go with just syscache_prune_min_age, get that into PG 12, > > and we can

Re: Protect syscache from bloating with negative cache entries

On Fri, Jan 25, 2019 at 08:14:19AM +, Tsunakawa, Takayuki wrote: > Hi Horiguchi-san, Bruce, > > From: Bruce Momjian [mailto:br...@momjian.us] > > I suggest you go with just syscache_prune_min_age, get that into > > PG 12, and we can then reevaluate what we need. If you want to > > hard-code a

RE: Protect syscache from bloating with negative cache entries

Hi Horiguchi-san, Bruce, From: Bruce Momjian [mailto:br...@momjian.us] > I suggest you go with just syscache_prune_min_age, get that into PG 12, > and we can then reevaluate what we need. If you want to hard-code a > minimum cache size where no pruning will happen, maybe based on the system >

RE: Protect syscache from bloating with negative cache entries

From: Robert Haas [mailto:robertmh...@gmail.com] > On Thu, Jan 24, 2019 at 10:02 AM Tom Lane wrote: > > I will argue hard that we should not do it at all, ever. > > > > There is already a mechanism for broadcasting global GUC changes: > > apply them to postgresql.conf (or use ALTER SYSTEM) and

Re: Protect syscache from bloating with negative cache entries

On Thu, Jan 24, 2019 at 10:02 AM Tom Lane wrote: > Bruce Momjian writes: > > On Thu, Jan 24, 2019 at 06:39:24PM +0900, Kyotaro HORIGUCHI wrote: > >> I also thought that there's some other features that is useful if > >> it could be turned on remotely so the remote GUC feature but it > >> was too

Re: Protect syscache from bloating with negative cache entries

Bruce Momjian writes: > On Thu, Jan 24, 2019 at 06:39:24PM +0900, Kyotaro HORIGUCHI wrote: >> I also thought that there's some other features that is useful if >> it could be turned on remotely so the remote GUC feature but it >> was too complex... > Well, I am thinking if we want to do

Re: Protect syscache from bloating with negative cache entries

On Thu, Jan 24, 2019 at 06:39:24PM +0900, Kyotaro HORIGUCHI wrote: > > Second, when would you use syscache_memory_target != 0? > > It is a suggestion upthread, we sometimes want to keep some known > amount of caches despite that expration should be activated. > > > If you had > >

Re: Protect syscache from bloating with negative cache entries

Thank you for the comments. At Wed, 23 Jan 2019 18:21:45 -0500, Bruce Momjian wrote in <20190123232145.ga8...@momjian.us> > On Wed, Jan 23, 2019 at 05:35:02PM +0900, Kyotaro HORIGUCHI wrote: > > At Mon, 21 Jan 2019 17:22:55 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > > wrote in > >

Re: Protect syscache from bloating with negative cache entries

On Wed, Jan 23, 2019 at 05:35:02PM +0900, Kyotaro HORIGUCHI wrote: > At Mon, 21 Jan 2019 17:22:55 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20190121.172255.226467552.horiguchi.kyot...@lab.ntt.co.jp> > > An option is an additional PGPROC member and interface functions. > > >

Re: Protect syscache from bloating with negative cache entries

At Mon, 21 Jan 2019 17:22:55 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190121.172255.226467552.horiguchi.kyot...@lab.ntt.co.jp> > An option is an additional PGPROC member and interface functions. > > struct PGPROC > { > ... > int syscahe_usage_track_interval; /* track

RE: Protect syscache from bloating with negative cache entries

From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > Although this doesn't put a hard cap on memory usage, it is indirectly and > softly limited by the cache_prune_min_age and cache_memory_target, which > determins how large a cache can grow until pruning happens. They are >

Re: Protect syscache from bloating with negative cache entries

On Fri, Jan 18, 2019 at 05:09:41PM -0800, Andres Freund wrote: > Hi, > > On 2019-01-18 19:57:03 -0500, Robert Haas wrote: > > On Fri, Jan 18, 2019 at 4:23 PM and...@anarazel.de > > wrote: > > > My proposal for this was to attach a 'generation' to cache entries. Upon > > > access cache entries

Re: Protect syscache from bloating with negative cache entries

Thank you for pointing out the stupidity. (Tom did earlier, though.) At Mon, 21 Jan 2019 07:12:41 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB6C78A@G01JPEXMBYT05> > From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > > 0003: Remote GUC setting > >

Re: Protect syscache from bloating with negative cache entries

Hello. At Fri, 18 Jan 2019 17:09:41 -0800, "and...@anarazel.de" wrote in <20190119010941.6ruftewah7t3k...@alap3.anarazel.de> > Hi, > > On 2019-01-18 19:57:03 -0500, Robert Haas wrote: > > On Fri, Jan 18, 2019 at 4:23 PM and...@anarazel.de > > wrote: > > > My proposal for this was to attach a

RE: Protect syscache from bloating with negative cache entries

From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > 0003: Remote GUC setting > > It is independent from the above two, and heavily arguable. > > pg_set_backend_config(pid, name, value) changes the GUC on the > backend with to . > Not having looked at the code yet, why did you

Re: Protect syscache from bloating with negative cache entries

Hi, On 2019-01-18 19:57:03 -0500, Robert Haas wrote: > On Fri, Jan 18, 2019 at 4:23 PM and...@anarazel.de wrote: > > My proposal for this was to attach a 'generation' to cache entries. Upon > > access cache entries are marked to be of the current > > generation. Whenever existing memory isn't

Re: Protect syscache from bloating with negative cache entries

On Fri, Jan 18, 2019 at 4:23 PM and...@anarazel.de wrote: > My proposal for this was to attach a 'generation' to cache entries. Upon > access cache entries are marked to be of the current > generation. Whenever existing memory isn't sufficient for further cache > entries and, on a less frequent

Re: Protect syscache from bloating with negative cache entries

Robert Haas writes: > On Thu, Jan 17, 2019 at 2:48 PM Bruce Momjian wrote: >> Unfortunately, because we have not found something we are happy with, we >> have done nothing. I agree LRU can be expensive. What if we do some >> kind of clock sweep and expiration like we do for shared buffers? I

Re: Protect syscache from bloating with negative cache entries

On Thu, Jan 17, 2019 at 2:48 PM Bruce Momjian wrote: > Well, I think everyone agrees there are workloads that cause undesired > cache bloat. What we have not found is a solution that doesn't cause > code complexity or undesired overhead, or one that >1% of users will > know how to use. > >

Re: Protect syscache from bloating with negative cache entries

At Fri, 18 Jan 2019 16:39:29 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190118.163929.229869562.horiguchi.kyot...@lab.ntt.co.jp> > Hello. > > At Fri, 18 Jan 2019 11:46:03 +1300, Gavin Flower > wrote in > <4e62e6b7-0ffb-54ae-3757-5583fcca3...@archidevsys.co.nz> > > On

Re: Protect syscache from bloating with negative cache entries

Hello. At Fri, 18 Jan 2019 11:46:03 +1300, Gavin Flower wrote in <4e62e6b7-0ffb-54ae-3757-5583fcca3...@archidevsys.co.nz> > On 18/01/2019 08:48, Bruce Momjian wrote: > > On Thu, Jan 17, 2019 at 11:33:35AM -0500, Robert Haas wrote: > >> The flaw in your thinking, as it seems to me, is that in

Re: Protect syscache from bloating with negative cache entries

On 18/01/2019 08:48, Bruce Momjian wrote: On Thu, Jan 17, 2019 at 11:33:35AM -0500, Robert Haas wrote: The flaw in your thinking, as it seems to me, is that in your concern for "the likelihood that cache flushes will simply remove entries we'll soon have to rebuild," you're apparently unwilling

Re: Protect syscache from bloating with negative cache entries

On Thu, Jan 17, 2019 at 11:33:35AM -0500, Robert Haas wrote: > The flaw in your thinking, as it seems to me, is that in your concern > for "the likelihood that cache flushes will simply remove entries > we'll soon have to rebuild," you're apparently unwilling to consider > the possibility of

Re: Protect syscache from bloating with negative cache entries

On Sun, Jan 13, 2019 at 11:41 AM Tom Lane wrote: > Putting a limit on the size of the syscaches doesn't accomplish anything > except to add cycles if your cache working set is below the limit, or > make performance fall off a cliff if it's above the limit. If you're running on a Turing machine,

RE: Protect syscache from bloating with negative cache entries

From: Tom Lane [mailto:t...@sss.pgh.pa.us] > Certainly, what I've done here doesn't preclude adding some wider solution > to > the issue of extremely large catcaches. I'm relieved to hear that. > I think it takes the pressure off > for one rather narrow problem case, and the mechanism could be

Re: Protect syscache from bloating with negative cache entries

On Tue, Jan 15, 2019 at 01:32:36PM -0500, Tom Lane wrote: > ... > > FYI, Oracle provides one parameter, shared_pool_size, that determine the > > size of a memory area that contains SQL plans and various dictionary > > objects. Oracle decides how to divide the area among constituents. So > > it

Re: Protect syscache from bloating with negative cache entries

Hi, On 2019-01-15 13:32:36 -0500, Tom Lane wrote: > Well, we *had* an LRU mechanism for the catcaches way back when. We got > rid of it --- see commit 8b9bc234a --- because (a) maintaining the LRU > info was expensive and (b) performance fell off a cliff in scenarios where > the cache size limit

Re: Protect syscache from bloating with negative cache entries

"Tsunakawa, Takayuki" writes: > But the syscache/relcache bloat still remains a problem, when there are many > live tables and application connections. Would you agree to solve this in > some way? I thought Horiguchi-san's latest patches would solve this and the > negative entries. Can we

Re: Protect syscache from bloating with negative cache entries

I'm really disappointed by the direction this thread is going in. The latest patches add an enormous amount of mechanism, and user-visible complexity, to do something that we learned was a bad idea decades ago. Putting a limit on the size of the syscaches doesn't accomplish anything except to add

RE: Protect syscache from bloating with negative cache entries

>From: Ideriha, Takeshi [mailto:ideriha.take...@jp.fujitsu.com] >The detailed comments for the source code will be provided later. Hi, I'm adding some comments to 0001 and 0002 one. [0001 patch] + /*

RE: Protect syscache from bloating with negative cache entries

Hello, Sorry for delay. The detailed comments for the source code will be provided later. >> I just thought that the pair of ageclass and nentries can be >> represented as json or multi-dimensional array but in virtual they are >> all same and can be converted each other using some functions.

Re: Protect syscache from bloating with negative cache entries

> On Tue, Nov 27, 2018 at 11:40 AM Kyotaro HORIGUCHI > wrote: > > The attached is the rebased version that has multidimentional > ageclass. Thank you, Just for the information, cfbot complains about this patch because: pgstatfuncs.c: In function ‘pgstat_get_syscache_stats’:

Re: Protect syscache from bloating with negative cache entries

Thank you for reviewing. At Thu, 15 Nov 2018 11:02:10 +, "Ideriha, Takeshi" wrote in <4E72940DA2BF16479384A86D54D0988A6F1F4165@G01JPEXMBKW04> > Hello, thank you for updating the patch. > > > >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > >At Thu, 4 Oct 2018 04:27:04

RE: Protect syscache from bloating with negative cache entries

>From: Ideriha, Takeshi [mailto:ideriha.take...@jp.fujitsu.com] >I haven't looked into the code but I'm going to do it later. Hi, I've taken a look at 0001 patch. Reviewing the rest of patch will be later. if (!IsParallelWorker())

RE: Protect syscache from bloating with negative cache entries

Hello, thank you for updating the patch. >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] >At Thu, 4 Oct 2018 04:27:04 +, "Ideriha, Takeshi" > wrote in ><4E72940DA2BF16479384A86D54D0988A6F1BCB6F@G01JPEXMBKW04> >> >As a *PoC*, in the attached patch (which applies to current

Re: Protect syscache from bloating with negative cache entries

Hello. Thank you for the comment. At Thu, 4 Oct 2018 04:27:04 +, "Ideriha, Takeshi" wrote in <4E72940DA2BF16479384A86D54D0988A6F1BCB6F@G01JPEXMBKW04> > >As a *PoC*, in the attached patch (which applies to current master), size of > >CTups are > >counted as the catcache size. > > > >It

RE: Protect syscache from bloating with negative cache entries

Hi, thank you for the explanation. >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] >> >> Can I confirm about catcache pruning? >> syscache_memory_target is the max figure per CatCache. >> (Any CatCache has the same max value.) So the total max size of >> catalog caches is

Re: Protect syscache from bloating with negative cache entries

Hello. Thank you for looking this. At Wed, 12 Sep 2018 05:16:52 +, "Ideriha, Takeshi" wrote in <4E72940DA2BF16479384A86D54D0988A6F197012@G01JPEXMBKW04> > Hi, > > >Subject: Re: Protect syscache from bloating with negative cache entries > > > >

RE: Protect syscache from bloating with negative cache entries

Hi, >Subject: Re: Protect syscache from bloating with negative cache entries > >Hello. The previous v4 patchset was just broken. >Somehow the 0004 was merged into the 0003 and applying 0004 results in >failure. I >removed 0004 part from the 0003 and rebased and repos

Re: Protect syscache from bloating with negative cache entries

Hi, On 2018-07-02 21:50:36 -0400, Alvaro Herrera wrote: > On 2018-Jul-02, Andrew Dunstan wrote: > > > Andres suggested back in March (and again privately to me) that given how > > much this has changed from the original this CF item should be marked > > Returned With Feedback and the current

Re: Protect syscache from bloating with negative cache entries

Hello. The previous v4 patchset was just broken. At Tue, 26 Jun 2018 18:00:03 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180626.180003.127457941.horiguchi.kyot...@lab.ntt.co.jp> > Hello. I rebased this patchset. .. > > The attached is the patch set including this plancache

Re: Protect syscache from bloating with negative cache entries

On 06/26/2018 05:00 AM, Kyotaro HORIGUCHI wrote: The attached is the patch set including this plancache stuff. 0001- catcache time-based expiration (The origin of this thread) 0002- introduces dynahash pruning feature 0003- implement relcache pruning using 0002 0004- (perhaps) independent

Re: Protect syscache from bloating with negative cache entries

Hello. I rebased this patchset. At Thu, 15 Mar 2018 14:12:46 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180315.141246.130742928.horiguchi.kyot...@lab.ntt.co.jp> > At Mon, 12 Mar 2018 17:34:08 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in >

Re: Protect syscache from bloating with negative cache entries

At Thu, 29 Mar 2018 18:51:45 -0700, Andres Freund wrote in <20180330015145.pvsr6kjtf6tw4...@alap3.anarazel.de> > Hi, > > On 2018-03-30 10:35:48 +0900, Kyotaro HORIGUCHI wrote: > > 0002-0004 is new, in response to the comment that caches other > > than the catcache ought to

Re: Protect syscache from bloating with negative cache entries

Hi, On 2018-03-30 10:35:48 +0900, Kyotaro HORIGUCHI wrote: > 0002-0004 is new, in response to the comment that caches other > than the catcache ought to get the same feature. These can be a > separate development from 0001 for v12. I don't find a measures > to catch the all case at once. > > If

Re: Protect syscache from bloating with negative cache entries

At Thu, 29 Mar 2018 18:22:59 -0700, Andres Freund wrote in <20180330012259.7k3442yz7jigh...@alap3.anarazel.de> > On 2018-03-23 17:01:11 +0900, Kyotaro HORIGUCHI wrote: > > Hello. > > > > At Wed, 21 Mar 2018 15:28:07 -0400, David Steele > > wrote in

Re: Protect syscache from bloating with negative cache entries

On 2018-03-23 17:01:11 +0900, Kyotaro HORIGUCHI wrote: > Hello. > > At Wed, 21 Mar 2018 15:28:07 -0400, David Steele wrote > in <43095b16-14fc-e4d8-3310-2b86eaaab...@pgmasters.net> > > On 3/15/18 1:12 AM, Kyotaro HORIGUCHI wrote: > > > At Mon, 12 Mar 2018 17:34:08 +0900

Re: Protect syscache from bloating with negative cache entries

Hello. At Wed, 21 Mar 2018 15:28:07 -0400, David Steele wrote in <43095b16-14fc-e4d8-3310-2b86eaaab...@pgmasters.net> > On 3/15/18 1:12 AM, Kyotaro HORIGUCHI wrote: > > At Mon, 12 Mar 2018 17:34:08 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > >

Re: Re: Protect syscache from bloating with negative cache entries

On 3/15/18 1:12 AM, Kyotaro HORIGUCHI wrote: > At Mon, 12 Mar 2018 17:34:08 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > > The attached is the patch set including this plancache stuff. > > 0001- catcache time-based expiration (The origin of this

Re: Protect syscache from bloating with negative cache entries

At Mon, 12 Mar 2018 17:34:08 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180312.173408.162882093.horiguchi.kyot...@lab.ntt.co.jp> > > > In short, it's not really apparent to me that negative syscache entries > > > are the major problem of this

Re: Protect syscache from bloating with negative cache entries

Oops. At Mon, 12 Mar 2018 17:34:08 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180312.173408.162882093.horiguchi.kyot...@lab.ntt.co.jp> > Something like the sttached test script causes relcache This is that. #! /usr/bin/perl # printf("drop

Re: Protect syscache from bloating with negative cache entries

At Fri, 09 Mar 2018 17:40:01 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180309.174001.202113825.horiguchi.kyot...@lab.ntt.co.jp> > > In short, it's not really apparent to me that negative syscache entries > > are the major problem of this kind.

Re: Protect syscache from bloating with negative cache entries

At Wed, 07 Mar 2018 23:12:29 -0500, Tom Lane wrote in <352.1520482...@sss.pgh.pa.us> > Kyotaro HORIGUCHI writes: > > At Thu, 8 Mar 2018 00:28:04 +, "Tsunakawa, Takayuki" > > wrote in > >

Re: Protect syscache from bloating with negative cache entries

Kyotaro HORIGUCHI writes: > At Thu, 8 Mar 2018 00:28:04 +, "Tsunakawa, Takayuki" > wrote in > <0A3221C70F24FB45833433255569204D1F8FF0D9@G01JPEXMBYT05> >> Yes. We are now facing the problem of too much memory use by

Re: Protect syscache from bloating with negative cache entries

Hello, At Thu, 8 Mar 2018 00:28:04 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1F8FF0D9@G01JPEXMBYT05> > From: Alvaro Herrera [mailto:alvhe...@alvh.no-ip.org] > > The thing that comes to mind when reading this patch is that some time

RE: Protect syscache from bloating with negative cache entries

From: Alvaro Herrera [mailto:alvhe...@alvh.no-ip.org] > The thing that comes to mind when reading this patch is that some time ago > we made fun of other database software, "they are so complicated to configure, > they have some magical settings that few people understand how to set". > Postgres

Re: Protect syscache from bloating with negative cache entries

On 2018-03-07 14:48:48 -0300, Alvaro Herrera wrote: > Oh, I wasn't suggesting to throw away the whole cache at that point; > only that that is a convenient to do whatever cleanup we want to do. But why is that better than doing so continuously? > What I'm not clear about is exactly what is the

Re: Protect syscache from bloating with negative cache entries

Hello, Andres Freund wrote: > On 2018-03-07 08:01:38 -0300, Alvaro Herrera wrote: > > I wonder if this is just because we refuse to acknowledge the notion of > > a connection pooler. If we did, and the pooler told us "here, this > > session is being given back to us by the application, we'll

Re: Protect syscache from bloating with negative cache entries

Hi, On 2018-03-07 08:01:38 -0300, Alvaro Herrera wrote: > I wonder if this is just because we refuse to acknowledge the notion of > a connection pooler. If we did, and the pooler told us "here, this > session is being given back to us by the application, we'll keep it > around until the next app

Re: Protect syscache from bloating with negative cache entries

Hi, On 2018-03-01 15:19:26 -0500, Robert Haas wrote: > On Thu, Mar 1, 2018 at 3:01 PM, Andres Freund wrote: > > On 2018-03-01 14:49:26 -0500, Robert Haas wrote: > >> On Thu, Mar 1, 2018 at 2:29 PM, Andres Freund wrote: > >> > Right. Which might be very

Re: Protect syscache from bloating with negative cache entries

On Thu, Mar 1, 2018 at 3:01 PM, Andres Freund wrote: > On 2018-03-01 14:49:26 -0500, Robert Haas wrote: >> On Thu, Mar 1, 2018 at 2:29 PM, Andres Freund wrote: >> > Right. Which might be very painful latency wise. And with poolers it's >> > pretty easy to

Re: Protect syscache from bloating with negative cache entries

On 2018-03-01 14:49:26 -0500, Robert Haas wrote: > On Thu, Mar 1, 2018 at 2:29 PM, Andres Freund wrote: > > Right. Which might be very painful latency wise. And with poolers it's > > pretty easy to get into situations like that, without the app > > influencing it. > > Really?

Re: Protect syscache from bloating with negative cache entries

On Thu, Mar 1, 2018 at 2:29 PM, Andres Freund wrote: > Right. Which might be very painful latency wise. And with poolers it's > pretty easy to get into situations like that, without the app > influencing it. Really? I'm not sure I believe that. You're talking perhaps a few

Re: Protect syscache from bloating with negative cache entries

Hi, On 2018-03-01 14:24:56 -0500, Robert Haas wrote: > On Thu, Mar 1, 2018 at 1:54 PM, Andres Freund wrote: > > So this is hardcoded, without any sort of cache pressure logic? Doesn't > > that mean we'll often *severely* degrade performance if a backend is > > idle for a

<    1   2   3   >