Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-04-04 Thread Andres Freund
Hi, On 2014-01-14 20:58:20 +0900, KONDO Mitsumasa wrote: I will test some join sqls performance and TPC-3 benchmark in this or next week. This patch has been marked as Waiting For Author for nearly two months now. Marked as Returned with Feedback. Greetings, Andres Freund -- Sent via

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread KONDO Mitsumasa
Hi, I fix and submit this patch in CF4. In my past patch, it is significant bug which is mistaken caluculation of offset in posix_fadvise():-( However it works well without problem in pgbench. Because pgbench transactions are always random access... And I test my patch in DBT-2 benchmark.

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 8:58 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: In my past patch, it is significant bug which is mistaken caluculation of offset in posix_fadvise():-( However it works well without problem in pgbench. Because pgbench transactions are always random

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 8:58 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: In my past patch, it is significant bug which is mistaken caluculation of offset in posix_fadvise():-( However it works well without problem in pgbench. Because pgbench transactions are always random

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread KONDO Mitsumasa
Hi, I fixed the patch to improve followings. - Can compile in MacOS. - Change GUC name enable_kernel_readahead to readahead_strategy. - Change POSIX_FADV_SEQUNENTIAL to POISX_FADV_NORMAL when we select sequential access strategy, this reason is later... I tested simple two access

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread Simon Riggs
On 17 December 2013 11:50, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Unpatched PG is faster than patched in A and B query. It was about 1.3 times faster. Result of A query as expected, because patched PG cannot execute readahead at all. So cache cold situation is bad for patched PG.

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread KONDO Mitsumasa
(2013/12/17 21:29), Simon Riggs wrote: These are interesting results. Good research. Thanks! They also show that the benefit of this is very specific to the exact task being performed. I can't see any future for a setting that applies to everything or nothing. We must be more selective. This

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 9:30), Claudio Freire wrote: On Wed, Dec 11, 2013 at 3:14 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: enable_readahead=os|fadvise with os = on, fadvise = off Hmm. fadvise is method and is not a purpose. So I consider another idea of this GUC. Yeah, I was thinking

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-12 Thread Simon Riggs
On 14 November 2013 12:09, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: For your information of effect of this patch, I got results of pgbench which are in-memory-size database and out-memory-size database, and postgresql.conf settings are always used by us. It seems to improve

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-12 Thread Mitsumasa KONDO
2013/12/12 Simon Riggs si...@2ndquadrant.com On 14 November 2013 12:09, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: For your information of effect of this patch, I got results of pgbench which are in-memory-size database and out-memory-size database, and postgresql.conf

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-12 Thread Simon Riggs
On 12 December 2013 13:43, Mitsumasa KONDO kondo.mitsum...@gmail.com wrote: Your tests seem to relate to pgbench. Do we have tests on more BI related tasks? Yes, off-course! We will need another benchmark test before conclusion of this patch. What kind of benchmark do you have? I suggest

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-11 Thread Claudio Freire
On Wed, Dec 11, 2013 at 3:14 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: enable_readahead=os|fadvise with os = on, fadvise = off Hmm. fadvise is method and is not a purpose. So I consider another idea of this GUC. Yeah, I was thinking of opening the door for readahead=aio,

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-10 Thread Claudio Freire
On Tue, Dec 10, 2013 at 5:03 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version. When this GUC is on(default), it works in

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-10 Thread KONDO Mitsumasa
(2013/12/10 22:55), Claudio Freire wrote: On Tue, Dec 10, 2013 at 5:03 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version.

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-09 Thread KONDO Mitsumasa
Hi, I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version. When this GUC is on(default), it works in POSIX_FADV_NORMAL which is general readahead in OS. And when it is off, it

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread KONDO Mitsumasa
(2013/11/15 13:48), Claudio Freire wrote: On Thu, Nov 14, 2013 at 11:13 PM, KONDO Mitsumasa I use CentOS 6.4 which kernel version is 2.6.32-358.23.2.el6.x86_64 in this test. That's close to the kernel version I was using, so you should see the same effect. OK. You proposed readahead maximum

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread Claudio Freire
On Sun, Nov 17, 2013 at 11:02 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: However, my patch is on the way and needed to more improvement. I am going to add method of controlling readahead by GUC, for user can freely select readahed parameter in their transactions. Rather, I'd

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread KONDO Mitsumasa
(2013/11/18 11:25), Claudio Freire wrote: On Sun, Nov 17, 2013 at 11:02 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: However, my patch is on the way and needed to more improvement. I am going to add method of controlling readahead by GUC, for user can freely select readahed

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-15 Thread Peter Eisentraut
On 11/14/13, 7:09 AM, KONDO Mitsumasa wrote: I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice() in various disk-read situations. Various compiler warnings: tablecmds.c: In function

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread Claudio Freire
On Thu, Nov 14, 2013 at 9:09 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice() in various disk-read situations. In general

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread Fujii Masao
On Thu, Nov 14, 2013 at 9:09 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Hi, I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice() in various disk-read situations. When I

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
Hi Claudio, (2013/11/14 22:53), Claudio Freire wrote: On Thu, Nov 14, 2013 at 9:09 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 2:03), Fujii Masao wrote: On Thu, Nov 14, 2013 at 9:09 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Hi, I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice() in

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread Peter Geoghegan
On Thu, Nov 14, 2013 at 6:18 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I will fix it. Could you tell me your Mac OS version and gcc version? I have only mac book air with Maverick OS(10.9). I have an idea that Mac OSX doesn't have posix_fadvise at all. Didn't you use the relevant

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 11:17), Peter Geoghegan wrote: On Thu, Nov 14, 2013 at 6:18 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I will fix it. Could you tell me your Mac OS version and gcc version? I have only mac book air with Maverick OS(10.9). I have an idea that Mac OSX doesn't have

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread Claudio Freire
On Thu, Nov 14, 2013 at 11:13 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Hi Claudio, (2013/11/14 22:53), Claudio Freire wrote: On Thu, Nov 14, 2013 at 9:09 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I create a patch that is improvement of disk-read and OS file