Re: [HACKERS] gaussian distribution pgbench

2014-03-18 Thread KONDO Mitsumasa
(2014/03/17 22:37), Tom Lane wrote: KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp writes: (2014/03/17 18:02), Heikki Linnakangas wrote: On 03/17/2014 10:40 AM, KONDO Mitsumasa wrote: There is an infinite number of variants of the TPC-B test that we could include in pgbench. If we start

Re: [HACKERS] gaussian distribution pgbench

2014-03-18 Thread KONDO Mitsumasa
(2014/03/17 23:29), Robert Haas wrote: On Sat, Mar 15, 2014 at 4:50 AM, Mitsumasa KONDO kondo.mitsum...@gmail.com wrote: There are explanations and computations as comments in the code. If it is about the documentation, I'm not sure that a very precise mathematical definition will help a lot of

Re: [HACKERS] gaussian distribution pgbench

2014-03-18 Thread KONDO Mitsumasa
And I find new useful point of this feature. Under following results are '--gaussian=20' case and '--gaussian=2' case, and postgresql setting is same. [mitsu-ko@pg-rex31 pgbench]$ ./pgbench -c8 -j4 --gaussian=20 -T30 -P 5 starting vacuum...end. progress: 5.0 s, 4285.8 tps, lat 1.860 ms stddev

Re: [HACKERS] gaussian distribution pgbench

2014-03-17 Thread KONDO Mitsumasa
Hi Heikki-san, (2014/03/17 14:39), KONDO Mitsumasa wrote: (2014/03/15 15:53), Fabien COELHO wrote: Hello Heikki, A couple of comments: * There should be an explicit \setrandom ... uniform option too, even though you get that implicitly if you don't specify the distribution Fix. We can use

Re: [HACKERS] gaussian distribution pgbench

2014-03-17 Thread KONDO Mitsumasa
(2014/03/17 17:46), Heikki Linnakangas wrote: On 03/15/2014 08:53 AM, Fabien COELHO wrote: * Does min and max really make sense for gaussian and exponential distributions? For gaussian, I would expect mean and standard deviation as the parameters, not min/max/threshold. Yes... and no:-) The

Re: [HACKERS] gaussian distribution pgbench

2014-03-17 Thread KONDO Mitsumasa
(2014/03/17 18:02), Heikki Linnakangas wrote: On 03/17/2014 10:40 AM, KONDO Mitsumasa wrote: By the way, you seem to want to remove --gaussian=NUM and --exponential=NUM command options. Can you tell me the objective reason? I think pgbench is the benchmark test on PostgreSQL and default

Re: [HACKERS] gaussian distribution pgbench

2014-03-16 Thread KONDO Mitsumasa
(2014/03/15 15:53), Fabien COELHO wrote: Hello Heikki, A couple of comments: * There should be an explicit \setrandom ... uniform option too, even though you get that implicitly if you don't specify the distribution Indeed. I agree. I suggested it, but it got lost. * What exactly does

Re: [HACKERS] gaussian distribution pgbench

2014-03-13 Thread KONDO Mitsumasa
Hi, (2014/03/14 4:21), Fabien COELHO wrote: We should do the same discussion for the UI of command-line option? The patch adds two options --gaussian and --exponential, but this UI seems to be a bit inconsistent with the UI for \setrandom. Instead, we can use something like

Re: [HACKERS] gaussian distribution pgbench

2014-03-13 Thread KONDO Mitsumasa
(2014/03/13 23:00), Fujii Masao wrote: On Thu, Mar 13, 2014 at 10:51 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 03/13/2014 03:17 PM, Fujii Masao wrote: On Tue, Mar 11, 2014 at 1:49 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: (2014/03/09 1:49), Fabien COELHO wrote

Re: [HACKERS] gaussian distribution pgbench

2014-03-10 Thread KONDO Mitsumasa
(2014/03/09 1:49), Fabien COELHO wrote: Hello Mitsumasa-san, New \setrandom interface is here. \setrandom var min max [gaussian threshold | exponential threshold] Attached patch realizes this interface, but it has little bit ugly codeing in executeStatement() and process_commands().. I

Re: [HACKERS] gaussian distribution pgbench

2014-03-06 Thread KONDO Mitsumasa
Hi, (2014/03/04 17:42), KONDO Mitsumasa wrote: (2014/03/04 17:28), Fabien COELHO wrote: OK. I'm not sure which idia is the best. So I wait for comments in community:) Hmmm. Maybe you can do what Tom voted for, he is the committer:-) Yeah, but he might change his mind by our disscuttion. So

Re: [HACKERS] gaussian distribution pgbench

2014-03-06 Thread KONDO Mitsumasa
(2014/03/07 16:02), KONDO Mitsumasa wrote: And other cases are classified under following. \setrandom var min max gaussian #hoge -- uniform Oh, it's wrong... It will be.. \setrandom var min max gaussian #hoge -- ERROR Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via

Re: [HACKERS] gaussian distribution pgbench

2014-03-04 Thread KONDO Mitsumasa
(2014/03/04 17:28), Fabien COELHO wrote: OK. I'm not sure which idia is the best. So I wait for comments in community:) Hmmm. Maybe you can do what Tom voted for, he is the committer:-) Yeah, but he might change his mind by our disscuttion. So I wait untill tomorrow, and if nothing to comment,

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-03-04 Thread KONDO Mitsumasa
Hi all, I think this patch is completely forgotten, and feel very unfortunate:( Min, max, and stdev is basic statistics in general monitoring tools, So I'd like to push it. (2014/02/12 15:45), KONDO Mitsumasa wrote: (2014/01/29 17:31), Rajeev rastogi wrote: No Issue, you can share me

Re: [HACKERS] gaussian distribution pgbench

2014-03-03 Thread KONDO Mitsumasa
(2014/03/03 16:51), Fabien COELHO wrote:\setrandom foo 1 10 [uniform] \setrandom foo 1 :size gaussian 3.6 \setrandom foo 1 100 exponential 7.2 It's good design. I think it will become more low overhead at part of parsing in pgbench, because comparison of strings will be

Re: [HACKERS] gaussian distribution pgbench

2014-03-02 Thread KONDO Mitsumasa
(2014/03/02 22:32), Fabien COELHO wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Seems that in the review so far, Fabien has focused mainly in the mathematical properties of the new random number generation. That seems perfectly fine, but no comment has been made about the chosen UI

Re: [HACKERS] What behavior is in this loop?

2014-02-28 Thread KONDO Mitsumasa
(2014/02/27 20:19), Heikki Linnakangas wrote: On 02/27/2014 12:38 PM, KONDO Mitsumasa wrote: I found interesting for and while loop in WaitForWALToBecomeAvailable() in xlog.c. Can you tell me this behavior? for (;;) { ~ } while (StanbyMode) I confirmed this code is no problem in gcc compiler

[HACKERS] What behavior is in this loop?

2014-02-27 Thread KONDO Mitsumasa
Hi, I found interesting for and while loop in WaitForWALToBecomeAvailable() in xlog.c. Can you tell me this behavior? for (;;) { ~ } while (StanbyMode) I confirmed this code is no problem in gcc compiler:) Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via pgsql-hackers

Re: [HACKERS] Unfortunate choice of short switch name in pgbench

2014-02-27 Thread KONDO Mitsumasa
(2014/02/28 2:39), Tom Lane wrote: Fabien COELHO coe...@cri.ensmp.fr writes: Yeah, but they don't make -P take an integer argument. It's that little frammish that makes this problem significant. I do not see a strong case to make options with arguments case insensitive as a general rule.

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-02-18 Thread KONDO Mitsumasa
(2014/02/17 21:44), Rajeev rastogi wrote: It got compiled successfully on Windows. Thank you for checking on Windows! It is very helpful for me. Can we allow to add three statistics? I think only adding stdev is difficult to image for user. But if there are min and max, we can image each

Re: [HACKERS] gaussian distribution pgbench

2014-02-17 Thread KONDO Mitsumasa
(2014/02/16 7:38), Fabien COELHO wrote: I have updated the patch (v7) based on Mitsumasa latest v6: - some code simplifications formula changes. - I've added explicit looping probability computations in comments to show the (low) looping probability of the iterative search. -

Re: [HACKERS] Exposing currentTransactionWALVolume

2014-02-17 Thread KONDO Mitsumasa
(2014/02/15 23:04), Andres Freund wrote: Hi Simon, On 2014-01-14 17:12:35 +, Simon Riggs wrote: /* - * MarkCurrentTransactionIdLoggedIfAny + * ReportTransactionInsertedWAL * - * Remember that the current xid - if it is assigned - now has been wal logged. + * Remember that the

Re: [HACKERS] gaussian distribution pgbench

2014-02-13 Thread KONDO Mitsumasa
Hi Febien, Thank you very much for your very detail and useful comments! I read your comment, I agree most of your advice:) Attached patch is fixed for your comment. That are... - Remove redundant long-option. - We can use --gaussian=NUM -S or --gaussian=NUMN -N options. - Add sentence in

Re: [HACKERS] gaussian distribution pgbench

2014-02-13 Thread KONDO Mitsumasa
Sorry, previos attached patch has small bug. Please use latest one. 134 - return min + (int64) (max - min + 1) * rand; 134 + return min + (int64)((max - min + 1) * rand); Regards, -- Mitsumasa KONDO NTT Open Source Software Center *** a/contrib/pgbench/pgbench.c ---

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-02-11 Thread KONDO Mitsumasa
Hi Rajeev, (2014/01/29 17:31), Rajeev rastogi wrote: No Issue, you can share me the test cases, I will take the performance report. Attached patch is supported to latest pg_stat_statements. It includes min, max, and stdev statistics. Could you run compiling test on your windows enviroments? I

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-30 Thread KONDO Mitsumasa
(2014/01/30 8:29), Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: I could live with just stddev. Not sure others would be so happy. FWIW, I'd vote for just stddev, on the basis that min/max appear to add more to the counter update time than stddev does; you've got this: +

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-29 Thread KONDO Mitsumasa
(2014/01/29 16:58), Peter Geoghegan wrote: On Tue, Jan 28, 2014 at 10:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp writes: By the way, latest pg_stat_statement might affect performance in Windows system. Because it uses fflush() system call every

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-29 Thread KONDO Mitsumasa
(2014/01/29 17:31), Rajeev rastogi wrote: On 28th January, Mitsumasa KONDO wrote: By the way, latest pg_stat_statement might affect performance in Windows system. Because it uses fflush() system call every creating new entry in pg_stat_statements, and it calls many fread() to warm file cache.

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-28 Thread KONDO Mitsumasa
:54, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp wrote: Rebased patch is attached. Does this fix the Windows bug reported by Kumar on 20/11/2013 ? Sorry, I was misunderstanding. First name of Mr. Rajeev Rastogi is Kumar! I searched

Re: [HACKERS] drop duplicate buffers in OS

2014-01-28 Thread KONDO Mitsumasa
/01/29 8:20), Jeff Janes wrote: On Wed, Jan 15, 2014 at 10:34 AM, Robert Haas robertmh...@gmail.com mailto:robertmh...@gmail.com wrote: On Wed, Jan 15, 2014 at 1:53 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp wrote: I create patch that can

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-28 Thread KONDO Mitsumasa
(2014/01/29 15:51), Tom Lane wrote: KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp writes: By the way, latest pg_stat_statement might affect performance in Windows system. Because it uses fflush() system call every creating new entry in pg_stat_statements, and it calls many fread() to warm

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-27 Thread KONDO Mitsumasa
(2014/01/23 23:18), Andrew Dunstan wrote: What is more, if the square root calculation is affecting your benchmarks, I suspect you are benchmarking the wrong thing. I run another test that has two pgbench-clients in same time, one is select-only-query and another is executing 'SELECT * 

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-27 Thread KONDO Mitsumasa
(2014/01/26 17:43), Mitsumasa KONDO wrote: 2014-01-26 Simon Riggs si...@2ndquadrant.com mailto:si...@2ndquadrant.com On 21 January 2014 19:48, Simon Riggs si...@2ndquadrant.com mailto:si...@2ndquadrant.com wrote: On 21 January 2014 12:54, KONDO Mitsumasa kondo.mitsum

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-23 Thread KONDO Mitsumasa
(2014/01/23 10:28), Peter Geoghegan wrote: On Wed, Jan 22, 2014 at 5:28 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Oh, thanks to inform me. I think essential problem of my patch has bottle neck in sqrt() function and other division caluculation. Well, that's a pretty easy theory

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/22 9:34), Simon Riggs wrote: AFAICS, all that has happened is that people have given their opinions and we've got almost the same identical patch, with a rush-rush comment to commit even though we've waited months. If you submit a patch, then you need to listen to feedback and be clear

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: OK, Kondo, please demonstrate benchmarks that show we have 1% impact from this change. Otherwise we may need a config parameter to allow the calculation. OK, testing

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/23 12:00), Andrew Dunstan wrote: On 01/22/2014 08:28 PM, KONDO Mitsumasa wrote: (2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: OK, Kondo, please demonstrate benchmarks that show we have 1% impact from

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-21 Thread KONDO Mitsumasa
Rebased patch is attached. pg_stat_statements in PG9.4dev has already changed table columns in. So I hope this patch will be committed in PG9.4dev. Regards, -- Mitsumasa KONDO NTT Open Source Software Center *** a/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql ---

Re: [HACKERS] drop duplicate buffers in OS

2014-01-16 Thread KONDO Mitsumasa
(2014/01/16 21:38), Aidan Van Dyk wrote: Can we just get the backend that dirties the page to the posix_fadvice DONTNEED? No, it can remove clean page in OS file caches. Because if page is dirtied, it cause physical-disk-writing. However, it is experimental patch so it might be changed by

Re: [HACKERS] drop duplicate buffers in OS

2014-01-16 Thread KONDO Mitsumasa
(2014/01/16 3:34), Robert Haas wrote: On Wed, Jan 15, 2014 at 1:53 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I create patch that can drop duplicate buffers in OS using usage_count alogorithm. I have developed this patch since last summer. This feature seems to be discussed in hot

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread KONDO Mitsumasa
Hi, I fix and submit this patch in CF4. In my past patch, it is significant bug which is mistaken caluculation of offset in posix_fadvise():-( However it works well without problem in pgbench. Because pgbench transactions are always random access... And I test my patch in DBT-2 benchmark.

[HACKERS] drop duplicate buffers in OS

2014-01-14 Thread KONDO Mitsumasa
Hi, I create patch that can drop duplicate buffers in OS using usage_count alogorithm. I have developed this patch since last summer. This feature seems to be discussed in hot topic, so I submit it more faster than my schedule. When usage_count is high in shared_buffers, they are hard to drop

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread KONDO Mitsumasa
Hi, I fixed the patch to improve followings. - Can compile in MacOS. - Change GUC name enable_kernel_readahead to readahead_strategy. - Change POSIX_FADV_SEQUNENTIAL to POISX_FADV_NORMAL when we select sequential access strategy, this reason is later... I tested simple two access

Re: [HACKERS] pg_rewarm status

2013-12-17 Thread KONDO Mitsumasa
(2013/12/18 5:33), Robert Haas wrote: Sounds like it might be worth dusting the patch off again... I'd like to request you to add all_index option and usage_count option. When all_index option is selected, all index become rewarm nevertheless user doesn't input relation name. And usage_count

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread KONDO Mitsumasa
(2013/12/17 21:29), Simon Riggs wrote: These are interesting results. Good research. Thanks! They also show that the benefit of this is very specific to the exact task being performed. I can't see any future for a setting that applies to everything or nothing. We must be more selective. This

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 7:23), Fabrízio de Royes Mello wrote: On Wed, Dec 11, 2013 at 7:47 PM, Andres Freund and...@2ndquadrant.com * hot_standby=off: Makes delay useable with wal_level=archive (and thus a lower WAL volume) * standby_mode=off: Configurations that use tools like pg_standby and

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 18:09), Simon Riggs wrote: On 9 December 2013 10:54, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: (2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp Hi Fabrízio, I test your

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 9:30), Claudio Freire wrote: On Wed, Dec 11, 2013 at 3:14 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: enable_readahead=os|fadvise with os = on, fadvise = off Hmm. fadvise is method and is not a purpose. So I consider another idea of this GUC. Yeah, I was thinking

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-10 Thread KONDO Mitsumasa
(2013/12/11 10:25), Tom Lane wrote: Jeff Janes jeff.ja...@gmail.com writes: On Tue, Dec 3, 2013 at 11:39 PM, Claudio Freire klaussfre...@gmail.comwrote: Problem is, Postgres relies on a working kernel cache for checkpoints. Checkpoint logic would have to be heavily reworked to account for an

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-10 Thread KONDO Mitsumasa
(2013/12/10 22:55), Claudio Freire wrote: On Tue, Dec 10, 2013 at 5:03 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version

Re: [HACKERS] Time-Delayed Standbys

2013-12-10 Thread KONDO Mitsumasa
(2013/12/10 18:38), Andres Freund wrote: master PITR? What's that? All PITR is based on recovery.conf and thus not really a master? master PITR is PITR with standby_mode = off. It's just recovery from basebackup. They have difference between master PITR and standby that the former will be

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo 49 -# commited transactions from the master, specify a recovery time delay. 49 +# committed transactions from the master, specify a recovery time delay. * Fix white space 177 - if (secs = 0 microsecs

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 19:36), KONDO Mitsumasa wrote: * Problem 1 I read your wittened document. There is PITR has not affected. However, when I run PITR with min_standby_apply_delay=300, it cannot start server. The log is under following. [mitsu-ko@localhost postgresql]$ bin/pg_ctl -D data2 start

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo 49 -# commited transactions from the master, specify

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 20:29), Andres Freund wrote: On 2013-12-09 19:51:01 +0900, KONDO Mitsumasa wrote: Add my comment. We have to consider three situations. 1. PITR 2. replication standby 3. replication standby with restore_command I think this patch cannot delay in 1 situation. Why? I have three

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-09 Thread KONDO Mitsumasa
Hi, I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version. When this GUC is on(default), it works in POSIX_FADV_NORMAL which is general readahead in OS. And when it is off, it

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-08 Thread KONDO Mitsumasa
(2013/12/05 23:42), Greg Stark wrote: On Thu, Dec 5, 2013 at 8:35 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Yes. And using something efficiently DirectIO is more difficult than BufferedIO. If we change write() flag with direct IO in PostgreSQL, it will execute hardest ugly

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread KONDO Mitsumasa
(2013/12/04 16:39), Claudio Freire wrote: On Wed, Dec 4, 2013 at 4:28 AM, Tatsuo Ishii is...@postgresql.org wrote: Can we avoid the Linux kernel problem by simply increasing our shared buffer size, say up to 80% of memory? It will be swap more easier. Is that the case? If the system has not

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/11/30 5:34), Fabrízio de Royes Mello wrote: On Fri, Nov 29, 2013 at 5:49 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp mailto:kondo.mitsum...@lab.ntt.co.jp wrote: * Problem1 Your patch does not code recovery.conf.sample about recovery_time_delay. Please add it. Fixed. OK

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 4:00), Andres Freund wrote: On 2013-12-03 13:46:28 -0500, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse christ...@2ndquadrant.com wrote: Hi Fabrizio, looks good to me. I

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 11:28), Tatsuo Ishii wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus j...@agliodbs.com wrote: Would certainly be nice. Realistically, getting good automated performace tests will require paying someone like Greg S., Mark or me for 6

Re: [HACKERS] Time-Delayed Standbys

2013-11-28 Thread KONDO Mitsumasa
Hi Royes, I'm sorry for my late review... I feel potential of your patch in PG replication function, and it might be something useful for all people. I check your patch and have some comment for improvement. I haven't executed detail of unexpected sutuation yet. But I think that under

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/18 20:16), Haribabu kommi wrote: On 18 October 2013 13:35 KONDO Mitsumasa wrote: This patch conflicts pg_stat_statement_min_max_exectime patch which I submitted, and pg_stat_statement_min_max_exectime patch also adds new columns which are min_time and max_time. So I'd like to change

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/19 3:56), Peter Geoghegan wrote: On Mon, Nov 18, 2013 at 10:49 AM, Fujii Masao masao.fu...@gmail.com wrote: The same idea was proposed before but not committed because Itagaki thought that pg_stat_statements view should report only raw values. Please read the following thread. I have

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/19 11:12), KONDO Mitsumasa wrote: (2013/11/19 3:56), Peter Geoghegan wrote: On Mon, Nov 18, 2013 at 10:49 AM, Fujii Masao masao.fu...@gmail.com wrote: The same idea was proposed before but not committed because Itagaki thought that pg_stat_statements view should report only raw values

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/19 12:03), Peter Geoghegan wrote: On Mon, Nov 18, 2013 at 6:12 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I confirmed that Itagaki-san and Mr Cerdic disscution. He said that raw values be just simple. However, were his changes just simple? I cannot understand his

Re: [HACKERS] Logging WAL when updating hintbit

2013-11-18 Thread KONDO Mitsumasa
(2013/11/15 19:27), Sawada Masahiko wrote: On Thu, Nov 14, 2013 at 7:51 PM, Florian Weimer fwei...@redhat.com wrote: On 11/14/2013 07:02 AM, Sawada Masahiko wrote: I attached patch adds new wal_level 'all'. Shouldn't this be a separate setting? It's useful for storage which requires

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread KONDO Mitsumasa
(2013/11/15 13:48), Claudio Freire wrote: On Thu, Nov 14, 2013 at 11:13 PM, KONDO Mitsumasa I use CentOS 6.4 which kernel version is 2.6.32-358.23.2.el6.x86_64 in this test. That's close to the kernel version I was using, so you should see the same effect. OK. You proposed readahead maximum

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread KONDO Mitsumasa
(2013/11/18 11:25), Claudio Freire wrote: On Sun, Nov 17, 2013 at 11:02 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: However, my patch is on the way and needed to more improvement. I am going to add method of controlling readahead by GUC, for user can freely select readahed

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/10/21 20:17), KONDO Mitsumasa wrote: (2013/10/18 22:21), Andrew Dunstan wrote: If we're going to extend pg_stat_statements, even more than min and max I'd like to see the standard deviation in execution time. OK. I do! I am making some other patches, please wait more! I add stddev_time

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
Oh! Sorry... I forgot to attach my latest patch. Regards, -- Mitsumasa KONDO NTT Open Source Software Center diff --git a/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql b/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql new file mode 100644 index 000..929d623 ---

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
Hi Claudio, (2013/11/14 22:53), Claudio Freire wrote: On Thu, Nov 14, 2013 at 9:09 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 2:03), Fujii Masao wrote: On Thu, Nov 14, 2013 at 9:09 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: Hi, I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/11/14 7:11), Peter Geoghegan wrote: On Wed, Oct 23, 2013 at 8:52 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, now if we had portable atomic addition, so that we could spare the spinlock ... And adding a histogram or min/max for something like execution time isn't an approach

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 11:17), Peter Geoghegan wrote: On Thu, Nov 14, 2013 at 6:18 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I will fix it. Could you tell me your Mac OS version and gcc version? I have only mac book air with Maverick OS(10.9). I have an idea that Mac OSX doesn't have

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 11:31), Peter Geoghegan wrote: On Thu, Nov 14, 2013 at 6:28 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: It is confirmation just to make sure, does this patch mean my patch? I agree with you about not adding another lock implementation. It will becomes overhead. Yes, I

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 2:09), Fujii Masao wrote: Agreed. Could you tell me your agreed reason? I am sorry that I suspect you doesn't understand this disccusion enough:-( Regards, -- Mitsumasa KONDO NTT Open Source Software Ceter -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Compression of full-page-writes

2013-10-22 Thread KONDO Mitsumasa
(2013/10/22 12:52), Fujii Masao wrote: On Tue, Oct 22, 2013 at 12:47 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Oct 21, 2013 at 4:40 PM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: (2013/10/19 14:58), Amit Kapila wrote: On Tue, Oct 15, 2013 at 11:41 AM, KONDO Mitsumasa

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-22 Thread KONDO Mitsumasa
Hi All, (2013/10/22 22:26), Stephen Frost wrote: * Dimitri Fontaine (dimi...@2ndquadrant.fr) wrote: In our case, what I keep experiencing with tuning queries is that we have like 99% of them running under acceptable threshold and 1% of them taking more and more time. This is usually

Re: [HACKERS] Who is pgFoundery administrator?

2013-10-21 Thread KONDO Mitsumasa
(2013/10/02 18:57), Michael Paquier wrote: kondo.mitsum...@lab.ntt.co.jp wrote: Who is pgFoundery administrator or board member now? I would like to send e-mail them. At least, it does not have information and support page in pgFoundery homepage. Why don't you consider github as a potential

Re: [HACKERS] Compression of full-page-writes

2013-10-21 Thread KONDO Mitsumasa
(2013/10/19 14:58), Amit Kapila wrote: On Tue, Oct 15, 2013 at 11:41 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I think in general also snappy is mostly preferred for it's low CPU usage not for compression, but overall my vote is also for snappy. I think low CPU usage

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-21 Thread KONDO Mitsumasa
(2013/10/18 22:21), Andrew Dunstan wrote: If we're going to extend pg_stat_statements, even more than min and max I'd like to see the standard deviation in execution time. OK. I do! I am making some other patches, please wait more! Regards, -- Mitsumasa KONDO NTT Open Source Software Center.;

[HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-18 Thread KONDO Mitsumasa
I submit patch adding min and max execute statement time in pg_stat_statement in next CF. pg_stat_statement have execution time, but it is average execution time and does not provide detail information very much. So I add min and max execute statement time in pg_stat_statement columns. Usage is

[HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-10-18 Thread KONDO Mitsumasa
Hi, I submit improvement of pg_stat_statement usage patch in CF3. In pg_stat_statement, I think buffer hit ratio is very important value. However, it is difficult to calculate it, and it need complicated SQL. This patch makes it more simple usage and documentation. -bench=# SELECT query,

Re: [HACKERS] Compression of full-page-writes

2013-10-15 Thread KONDO Mitsumasa
(2013/10/15 13:33), Amit Kapila wrote: Snappy is good mainly for un-compressible data, see the link below: http://www.postgresql.org/message-id/CAAZKuFZCOCHsswQM60ioDO_hk12tA7OG3YcJA8v=4yebmoa...@mail.gmail.com This result was gotten in ARM architecture, it is not general CPU. Please see detail

Re: [HACKERS] Compression of full-page-writes

2013-10-15 Thread KONDO Mitsumasa
(2013/10/15 22:01), k...@rice.edu wrote: Google's lz4 is also a very nice algorithm with 33% better compression performance than snappy and 2X the decompression performance in some benchmarks also with a bsd license: https://code.google.com/p/lz4/ If we judge only performance, we will select

Re: [HACKERS] Compression of full-page-writes

2013-10-14 Thread KONDO Mitsumasa
(2013/10/13 0:14), Amit Kapila wrote: On Fri, Oct 11, 2013 at 10:36 PM, Andres Freund and...@2ndquadrant.com wrote: But maybe pglz is just not a good fit for this, it really isn't a very good algorithm in this day and aage. +1. This compression algorithm is needed more faster than pglz which is

Re: [HACKERS] Release note fix for timeline item

2013-10-14 Thread KONDO Mitsumasa
Sorry for my reply late... (2013/10/08 23:26), Bruce Momjian wrote: First, I want to apologize for not completing the release notes earlier so that others could review them. I started working on the release notes on Friday, but my unfamiliarity with the process and fear of making a mistake

Re: [HACKERS] Compression of full-page-writes

2013-10-08 Thread KONDO Mitsumasa
(2013/10/08 17:33), Haribabu kommi wrote: The checkpoint_timeout and checkpoint_segments are increased to make sure no checkpoint happens during the test run. Your setting is easy occurred checkpoint in checkpoint_segments = 256. I don't know number of disks in your test server, in my test

Re: [HACKERS] Compression of full-page-writes

2013-10-08 Thread KONDO Mitsumasa
Hi, I tested dbt-2 benchmark in single instance and synchronous replication. Unfortunately, my benchmark results were not seen many differences... * Test server Server: HP Proliant DL360 G7 CPU:Xeon E5640 2.66GHz (1P/4C) Memory: 18GB(PC3-10600R-9) Disk: 146GB(15k)*4 RAID1+0

Re: [HACKERS] Compression of full-page-writes

2013-10-08 Thread KONDO Mitsumasa
(2013/10/08 20:13), Haribabu kommi wrote: I chosen the sync_commit=off mode because it generates more tps, thus it increases the volume of WAL. I did not think to there. Sorry... I will test with sync_commit=on mode and provide the test results. OK. Thanks! -- Mitsumasa KONDO NTT Open

[HACKERS] Who is pgFoundery administrator?

2013-10-02 Thread KONDO Mitsumasa
Hi, I want to submit new project in pgFoundery project. I submitted new project which is WAL archive copy tool with directIO method in pgFoundery homepage 2 weeks ago, but it does not have approved and responded at all:-( Who is pgFoundery administrator or board member now? I would like to send

Re: [HACKERS] Who is pgFoundery administrator?

2013-10-02 Thread KONDO Mitsumasa
(2013/10/02 17:37), KONDO Mitsumasa wrote: I want to submit new project in pgFoundery project. Our new project was approved yesterday! Thanks very much for pgFoundery crew. Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Compression of full-page-writes

2013-09-30 Thread KONDO Mitsumasa
(2013/09/30 13:55), Amit Kapila wrote: On Mon, Sep 30, 2013 at 10:04 AM, Fujii Masao masao.fu...@gmail.com wrote: Yep, please! It's really helpful! OK! I test with single instance and synchronous replication constitution. By the way, you posted patch which is sync_file_range() WAL writing

Re: [HACKERS] gaussian distribution pgbench

2013-09-29 Thread KONDO Mitsumasa
Sorry for my delay reply. Since I have had vacation last week, I replyed from gmail. However, it was stalled post to pgsql-hackers:-( (2013/09/21 6:05), Kevin Grittner wrote: You had accidentally added to the CF In Progress. Oh, I had completely mistook this CF schedule :-) Maybe, Horiguchi-san

Re: [HACKERS] gaussian distribution pgbench

2013-09-29 Thread KONDO Mitsumasa
Sorry for my delay reply. Since I have had vacation last week, I replied from gmail. However, it was stalled post to pgsql-hackers:-( (2013/09/21 7:54), Fabien COELHO wrote: However this pattern induces stronger cache effects which are maybe not too realistic, because neighboring keys in the

Re: [HACKERS] gaussian distribution pgbench

2013-09-29 Thread KONDO Mitsumasa
(2013/09/27 5:29), Peter Eisentraut wrote: This patch no longer applies. I will try to create this patch in next commit fest. If you have nice idea, please send me! Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Compression of full-page-writes

2013-09-29 Thread KONDO Mitsumasa
Hi Fujii-san, (2013/09/30 12:49), Fujii Masao wrote: On second thought, the patch could compress WAL very much because I used pgbench. I will do the same measurement by using another benchmark. If you hope, I can test this patch in DBT-2 benchmark in end of this week. I will use under

Re: [HACKERS] 9.4 regression

2013-09-05 Thread KONDO Mitsumasa
(2013/09/05 0:04), Andres Freund wrote: I'd vote for adding zeroing *after* the fallocate() first. +1, with FALLOC_FL_KEEP_SIZE flag. At least, fallocate with FALLOC_FL_KEEP_SIZE flag is faster than nothing in my developing sorted checkpoint. I adopted it to relation file, so I don't know

[HACKERS] Add pgbench option: CHECKPOINT before starting benchmark

2013-08-30 Thread KONDO Mitsumasa
Hi, I add checkpoint option to pgbench. pgbench is simple and useful benchmark for every user. However, result of benchmark greatly changes by some situations which are in executing checkpoint, number of dirty buffers in share_buffers, and so on. For such a problem, it is custom to carry out a

  1   2   >