Re: [HACKERS] Clock with Adaptive Replacement

2018-07-09 Thread Peter Geoghegan
On Tue, May 1, 2018 at 11:58 AM, Robert Haas wrote: >> All of that having been said, maybe we have an independent low-level >> problem: we increase usage_count when we pin a buffer, even if we last >> pinned the buffer 5ms ago. IIRC a change to this went in when ARC went >> in (and came out with

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-24 Thread Bruce Momjian
On Wed, May 2, 2018 at 12:27:19PM -0400, Robert Haas wrote: > I've seen customer have very good luck going higher if it lets all the > data fit in shared_buffers, or at least all the data that is accessed > with any frequency. I think it's useful to imagine a series of > concentric working sets

Re: Clock with Adaptive Replacement

2018-05-14 Thread Andrey Borodin
Hi, Ben! > 11 мая 2018 г., в 21:57, ben.manes написал(а): > > I have been working on caching as a hobby project for the last few years... Thanks for sharing this! Your message is full of insights and W-TinyLFU seems to me very promising for use in shared buffers. Best

Re: Clock with Adaptive Replacement

2018-05-11 Thread ben.manes
I have been working on caching as a hobby project for the last few years and would be happy to collaborate on an analysis and design. I've tackled a lot of these issues, wrote widely used implementations, and co-authored a paper for ACM's Transactions on Storage on an eviction policy. A short

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-10 Thread Thomas Munro
On Wed, Apr 25, 2018 at 6:31 PM, Andrey Borodin wrote: > 4. Using O_DIRECT while writing data files One interesting thing about O_DIRECT that I haven't seen mentioned in these discussions: POSIX effectively requires writes to the same file to be serialised, meaning in

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-09 Thread Peter Geoghegan
On Thu, May 3, 2018 at 5:16 PM, Peter Geoghegan wrote: > On Thu, May 3, 2018 at 12:37 PM, Robert Haas wrote: >> On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov >> wrote: >>> Sample output can be seen here: >>>

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-09 Thread Robert Haas
On Wed, May 9, 2018 at 1:37 PM, Merlin Moncure wrote: > On Wed, May 9, 2018 at 11:00 AM Robert Haas wrote: >> Independently of that, it would be probably also useful to avoid >> bumping the reference count multiple times when a buffer is accessed >> by

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-09 Thread Merlin Moncure
On Wed, May 9, 2018 at 11:00 AM Robert Haas wrote: > Independently of that, it would be probably also useful to avoid > bumping the reference count multiple times when a buffer is accessed > by the same backend several times in quick succession. Perhaps this > could even

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-09 Thread Robert Haas
On Tue, May 8, 2018 at 10:22 AM, Юрий Соколов wrote: > "just prior to the point" means we have to have machinery for information > expiration without eviction. For example, same "clock hand" should walk > over all buffers continiously, and decrement "usage count", but

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-08 Thread Vladimir Sitnikov
>I have work loads that I can repeat, so I can help with testing. That would be great. Do you think you could use DTrace to capture the trace? For instance, https://github.com/vlsi/pgsqlstat/blob/pgsqlio/pgsqlio Vladimir

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-08 Thread Jesper Pedersen
On 05/08/2018 11:35 AM, Vladimir Sitnikov wrote: Oh, looks like I'm inventing another kind of bicycle :-( Do you think you could capture a trace or two from a more-or-less representative application/database? Discussion of algorithms makes little sense as we all lack traces to

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-08 Thread Vladimir Sitnikov
> Oh, looks like I'm inventing another kind of bicycle :-( Do you think you could capture a trace or two from a more-or-less representative application/database? Discussion of algorithms makes little sense as we all lack traces to compare/validate. Vladimir

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-08 Thread Юрий Соколов
2018-05-08 16:21 GMT+03:00 Robert Haas : > > On Mon, May 7, 2018 at 12:54 PM, Юрий Соколов wrote: > >> Even if we have that, or something with similar effects, it's still > >> desirable to avoid bumping the usage count multiple times for accesses >

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-08 Thread Robert Haas
On Mon, May 7, 2018 at 12:54 PM, Юрий Соколов wrote: >> Even if we have that, or something with similar effects, it's still >> desirable to avoid bumping the usage count multiple times for accesses >> that happen close together in time. I don't really agree with Yura >>

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-07 Thread Юрий Соколов
2018-05-07 17:15 GMT+03:00 Robert Haas : > > On Sat, May 5, 2018 at 2:16 AM, Andrey Borodin wrote: > > But you loose difference between "touched once" and "actively used". Log > > scale of usage solves this: usage count grows logarithmically, but

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-07 Thread Robert Haas
On Sat, May 5, 2018 at 2:16 AM, Andrey Borodin wrote: > But you loose difference between "touched once" and "actively used". Log > scale of usage solves this: usage count grows logarithmically, but drains > linearly. Even if we have that, or something with similar effects,

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-06 Thread Yura Sokolov
06.05.2018 20:28, Andrey Borodin пишет: > >> 6 мая 2018 г., в 20:38, Yura Sokolov написал(а): >> >> I've been playing with logarithmic scale in postgresql roughly year ago. >> I didn't found any benefits compared to current code. In fact, it looked >> to perform worse

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-06 Thread Andrey Borodin
> 6 мая 2018 г., в 20:38, Yura Sokolov написал(а): > > I've been playing with logarithmic scale in postgresql roughly year ago. > I didn't found any benefits compared to current code. In fact, it looked > to perform worse than current code. > That is why I didn't wrote

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-06 Thread Yura Sokolov
06.05.2018 11:20, Andrey Borodin пишет: > > >> 5 мая 2018 г., в 13:25, Yura Sokolov написал(а): >> >> 05.05.2018 09:16, Andrey Borodin пишет: >>> Hi! >>> 4 мая 2018 г., в 16:05, Юрий Соколов написал(а): I didn't suggest log

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-05 Thread Yura Sokolov
05.05.2018 09:16, Andrey Borodin пишет: > Hi! > >> 4 мая 2018 г., в 16:05, Юрий Соколов >> написал(а): >> >> I didn't suggest log scale of usages, but rather >> "replacement-period based" increment: usage count could be >> incremented at most once in NBlocks/32 replaced

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-05 Thread Andrey Borodin
Hi! > 4 мая 2018 г., в 16:05, Юрий Соколов написал(а): > > I didn't suggest log scale of usages, but rather "replacement-period based" > increment: usage count could be incremented at most once in NBlocks/32 > replaced items. Once it is incremented, its "replacement

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-04 Thread Vladimir Sitnikov
>I don't have time to check this out just now, but it seems like an excellent idea There are a couple of sad things: 1) DTrace probes seem to be disabled by default. At least, I had to build PostgreSQL with --enable-dtrace in my macOS. Does that (the requirement to enable dtrace within

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-04 Thread Юрий Соколов
пт, 4 мая 2018 г., 12:27 Andrey Borodin : > > > 4 мая 2018 г., в 0:37, Robert Haas написал(а): > > > > On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov > > wrote: > >> Sample output can be seen here: > >>

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-04 Thread Andrey Borodin
> 4 мая 2018 г., в 0:37, Robert Haas написал(а): > > On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov > wrote: >> Sample output can be seen here: >> https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio > > Neat. Not sure what generated

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-03 Thread Peter Geoghegan
On Thu, May 3, 2018 at 12:37 PM, Robert Haas wrote: > On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov > wrote: >> Sample output can be seen here: >> https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio > > Neat. Not sure what generated

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-03 Thread Robert Haas
On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov wrote: > Sample output can be seen here: > https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio Neat. Not sure what generated this trace, but note this part: 32718388813748820500 1663

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-02 Thread Vladimir Sitnikov
>If somebody produced a trace showing the buffer lookups in order To get things moving, I've created a DTrace script that captures buffer reads: https://github.com/vlsi/pgsqlstat/blob/pgsqlio/pgsqlio Is it something that can be used to capture live traces? Sample output can be seen here:

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-02 Thread Robert Haas
On Tue, May 1, 2018 at 6:37 PM, Peter Geoghegan wrote: > This seems to be an old idea. I'm not too surprised ... this area has been well-studied. > I've always had a problem with the 8GB/16GB upper limit on the size of > shared_buffers. Not because it's wrong, but because it's not

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-02 Thread Yura Sokolov
02.05.2018 01:37, Peter Geoghegan пишет: > On Tue, May 1, 2018 at 11:58 AM, Robert Haas wrote: >> I agree that double-counting correlated accesses is a a problem, and I >> agree that we probably want to do something about it. I am skeptical >> that using wall-clock time is

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-01 Thread Peter Geoghegan
On Tue, May 1, 2018 at 11:58 AM, Robert Haas wrote: > That's not to say that the total of all usage counts must remain equal > to a constant or something dumb like that -- there's decisions to be > made in terms of how you implement things. For example, you can > imagine a

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-01 Thread Robert Haas
On Sat, Apr 28, 2018 at 7:16 PM, Peter Geoghegan wrote: > There could very easily be an enormous range of usefulness among > buffers in shared_buffers. And yet, we only recognize the increments > that usage_count can represent (this is before we even think about the > actual

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Andrey Borodin
Hi! > 30 апр. 2018 г., в 23:15, Andres Freund написал(а): > On 2018-04-30 15:39:08 +0500, Andrey Borodin wrote: >> I suspect that performance benefits can be not that big or even >> marginal, if we do not extend comprehensive eviction strategy with >> bgwriter fixes and

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Юрий Соколов
вт, 1 мая 2018 г., 0:02 Thomas Munro : > On Mon, Apr 30, 2018 at 10:39 PM, Andrey Borodin > wrote: > >> 30 апр. 2018 г., в 0:48, Andres Freund написал(а): > >> On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: > >> 30

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Andres Freund
On 2018-05-01 12:15:21 +1200, Thomas Munro wrote: > On Thu, Apr 26, 2018 at 1:31 PM, Thomas Munro > wrote: > > ... I > > suppose when you read a page in, you could tell the kernel that you > > POSIX_FADV_DONTNEED it, and when you steal a clean PG buffer you could >

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Thomas Munro
On Thu, Apr 26, 2018 at 1:31 PM, Thomas Munro wrote: > ... I > suppose when you read a page in, you could tell the kernel that you > POSIX_FADV_DONTNEED it, and when you steal a clean PG buffer you could > tell the kernel that you POSIX_FADV_WILLNEED its former

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Thomas Munro
On Mon, Apr 30, 2018 at 10:39 PM, Andrey Borodin wrote: >> 30 апр. 2018 г., в 0:48, Andres Freund написал(а): >> On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: >> 30 апр. 2018 г., в 0:39, Юрий Соколов написал(а): >> >> (btw,

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Andres Freund
Hi, On 2018-04-30 15:39:08 +0500, Andrey Borodin wrote: > I suspect that performance benefits can be not that big or even > marginal, if we do not extend comprehensive eviction strategy with > bgwriter fixes and O_DIRECT. If so, then the improvements aren't real. Bgwriter doesn't matter for

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-30 Thread Andrey Borodin
> 30 апр. 2018 г., в 0:48, Andres Freund написал(а): > > On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: >> >> 1. Teaching BgWriter to used data from eviction strategy to aggressively >> flush data to disk (instead of ++next_to_clean ) >> 2. Implementing strategies as

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-29 Thread Andres Freund
On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: > > > > 24 апр. 2018 г., в 23:14, Andres Freund написал(а): > > > > On 2018-04-24 17:16:47 +0500, Andrey Borodin wrote: > >> But, I think that cost of development of real page eviction strategy > >> itself is neglectable

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-29 Thread Юрий Соколов
вс, 29 апр. 2018 г., 2:17 Peter Geoghegan : > On Sat, Apr 28, 2018 at 7:39 AM, Stephen Frost wrote: > > Perhaps I'm misunderstanding the case here, but with the ring buffer we > > use for sequential access, aren't we already discouraging keeping heap > > pages

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-28 Thread Peter Geoghegan
On Sat, Apr 28, 2018 at 7:39 AM, Stephen Frost wrote: > Perhaps I'm misunderstanding the case here, but with the ring buffer we > use for sequential access, aren't we already discouraging keeping heap > pages around, particularly when they're coming from a sequential scan? > >

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-28 Thread Thomas Munro
On Sun, Apr 29, 2018 at 2:39 AM, Stephen Frost wrote: > * Peter Geoghegan (p...@bowt.ie) wrote: >> On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: >> > I'm personally not very excited about making rules like "index pages >> > are more valuable than

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-28 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: > > I'm personally not very excited about making rules like "index pages > > are more valuable than heap pages". Such rules will in many cases be > > true, but it's

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-26 Thread Peter Geoghegan
On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: > I think it is. We haven't done anything to address it. I think if we > want to move to direct I/O -- which may be something we need or want > to do -- we're going to have to work a lot harder at making good page >

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-26 Thread Robert Haas
On Wed, Apr 25, 2018 at 6:54 PM, Peter Geoghegan wrote: > Before some of the big shared_buffers bottlenecks were alleviated > several years ago, it was possible to observe shared_buffers evictions > occurring essentially at random. I have no idea if that's still true, > but it could

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-25 Thread Peter Geoghegan
On Wed, Apr 25, 2018 at 6:31 PM, Thomas Munro wrote: > Huh. Right. So it's not truly uniform. I wonder if any of these > algorithms are sensitive to the higher value of the leaf pages than > the heap pages. It seems quite subtle: even though we can see that >

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-25 Thread Thomas Munro
On Thu, Apr 26, 2018 at 10:54 AM, Peter Geoghegan wrote: > On Wed, Apr 25, 2018 at 5:26 AM, Thomas Munro > wrote: >> I think pgbench isn't a very interesting workload though. I don't >> have time right now but it would be fun to dig further and try

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-25 Thread Peter Geoghegan
On Wed, Apr 25, 2018 at 5:26 AM, Thomas Munro wrote: > I think pgbench isn't a very interesting workload though. I don't > have time right now but it would be fun to dig further and try to > construct workloads that generate pathological searches for buffers (I >

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-25 Thread Thomas Munro
On Tue, Apr 24, 2018 at 5:04 PM, Andrey Borodin wrote: > Here's what we have about size of shared buffer now [1] (taken from [2]). I > believe right hill must be big enough to reduce central pit to zero and make > function monotonic: OS page cache knows less about data

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-25 Thread Andrey Borodin
> 24 апр. 2018 г., в 23:14, Andres Freund написал(а): > > On 2018-04-24 17:16:47 +0500, Andrey Borodin wrote: >> But, I think that cost of development of real page eviction strategy >> itself is neglectable small compared to infrastructure changes needed >> by any non-CS5

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-24 Thread Andres Freund
Hi, On 2018-04-24 17:16:47 +0500, Andrey Borodin wrote: > But, I think that cost of development of real page eviction strategy > itself is neglectable small compared to infrastructure changes needed > by any non-CS5 strategy. What problems are you seeing? This isn't a lot of code? Greetings,

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-24 Thread Юрий Соколов
вт, 24 апр. 2018 г., 15:16 Andrey Borodin : > > > > 24 апр. 2018 г., в 11:31, Юрий Соколов > написал(а): > > > > Before implementing algorithms within PostgreSQL it will be great to > test them outside with real traces. > > > > I think, there should

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-24 Thread Andrey Borodin
> 24 апр. 2018 г., в 11:31, Юрий Соколов написал(а): > > Before implementing algorithms within PostgreSQL it will be great to test > them outside with real traces. > > I think, there should be mechamism to collect traces from real-world > postgresql instalations:

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-24 Thread Юрий Соколов
вт, 24 апр. 2018 г., 8:04 Andrey Borodin : > Hi, Thomas! > > > 24 апр. 2018 г., в 2:41, Thomas Munro > написал(а): > > > > On Fri, Feb 12, 2016 at 10:02 AM, Konstantin Knizhnik > > wrote: > >> Are there some well

Re: [HACKERS] Clock with Adaptive Replacement

2018-04-23 Thread Thomas Munro
On Fri, Feb 12, 2016 at 10:02 AM, Konstantin Knizhnik wrote: > Hi hackers, > > What do you think about improving cache replacement clock-sweep algorithm in > PostgreSQL with adaptive version proposed in this article: > >