Re: [HACKERS] New pgbench functions are misnamed

2016-05-05 Thread Fabien COELHO
I noticed that commit 7e137f846 added functions named max() and min() to pgbench's expression syntax. Unfortunately, these functions have zilch to do with what max() and min() do in SQL. They're actually more like the greatest() and least() server-side functions. Yep. While I can't imagine

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-05-05 Thread Fabien COELHO
Hello Tom, I understood the point and I do not see real disadvantages. The C standard really says that an enum is an int, and compilers just do that. No, it doesn't say that, and compilers don't just do that. A compiler is specifically allowed to store an enum in char or short if the enum's

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-05-04 Thread Fabien COELHO
Hello Tom, There's not really a point in using an enum if you use neither the type (which you shouldn't because if you or the bitmask value you have types outside the range of the enum), nor the autogenerated numbers. I do not think that there is such a constraint in C, you can use the enum

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-05-03 Thread Fabien COELHO
Hello Andres, An enum doesn't have a benefit for a bitmask imo - you can't "legally" use it as a type for functions accepting the bitmask. I do not understand. I suggested to use enum to enumerate the bitmask constants, ISTM that it does not preclude to use it as a bitmask as you do, it is

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-05-02 Thread Fabien COELHO
Hello, I'm unsure about switching enum to #define, could be an enum still with explicit values set, something like: An enum doesn't have a benefit for a bitmask imo - you can't "legally" use it as a type for functions accepting the bitmask. I do not understand. I suggested to use enum to

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-05-02 Thread Fabien COELHO
Hello Andres, I'm not sure this is the best way to go about this. I can see valid arguments for *always* using _mdfd_openseg() in mdsync(); and I'm wondering whether we shouldn't make EXTENSION_* into a bitmask (extend,extend_recovery,return_null,open_deleted). I thought about that when I

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-05-02 Thread Fabien COELHO
Hello Andres, Sure, attached. For what it is worth, it works for me on head. I'm wondering that if _mdfd_openseg may return NULL, then ISTM that "opened_directly" should logically be false, because it was not opened? If so, maybe consider something like: opened_directy = (seg !=

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-04-29 Thread Fabien COELHO
An interesting complement about the previous failing test: Although I disabled the "flushing" feature... sh> grep flush_after xxx/postgresql.conf bgwriter_flush_after = 0# 0 disables, backend_flush_after = 0 # 0 disables, wal_writer_flush_after = 0

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-04-28 Thread Fabien COELHO
createdb: database creation failed: ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details. Attached is a simpler/faster version, based on the previous script, I just added a CHECKPOINT after the EXPLAIN ANALYSE. It fails on head with: HINT:

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-04-25 Thread Fabien COELHO
Hello Andres, Fixes it for me too. Yep, for me too. I'm not at ease with the part of the API the code is dealing with, so I do not feel like a competent reviewer. I agree with the "more comments" suggested by Robert. I have just a small naming point: /* ereport if segment not

Re: [HACKERS] [COMMITTERS] pgsql: doc: Fix typos

2016-04-24 Thread Fabien COELHO
Hello Peter, http://git.postgresql.org/pg/commitdiff/b87b2f4bda1a3b98f8dea867b8bc419ace7a9ea9 doc/src/sgml/ref/pgbench.sgml | 8 - Here is example outputs: + Here is example output: I think that something is missing now on this line. Should it rather be "Here is an example

Re: [HACKERS] Pgbench with -f and -S

2016-04-17 Thread Fabien COELHO
For the future 9.6, scripts options are cumulatives, so -f & -S can be combined. Indeed, for the <= 9.5 it seems that some options are silently ignores, when combining -S/-f, only the last one is kept, it should be warned about. ​ ​Thanks Fabien, for confirming about the missing warning.

Re: [HACKERS] Pgbench with -f and -S

2016-04-17 Thread Fabien COELHO
Pgbench allows -f and -S combinations together where the doc says that -S effectively uses the internal select-only script. Is it okay to assume that -f is disregarded here? Or are they run in round-robin fashion (although then, how does it know which read-only part of my script to run?) or

Re: [HACKERS] [BUGS] Breakage with VACUUM ANALYSE + partitions

2016-04-12 Thread Fabien COELHO
Hello Andres, I can think of a number of relatively easy ways to address this: 1) Just zap (or issue?) all pending flush requests when getting an smgrinval/smgrclosenode 2) Do 1), but filter for the closed relnode 3) Actually handle the case of the last open segment not being RELSEG_SIZE

Re: [HACKERS] pgbench randomness initialization

2016-04-07 Thread Fabien COELHO
It means that you can't separate between OS caused, and pgbench order caused performance differences. I'm not objecting to adding an option for this; but I think Fabien is right that it shouldn't be the default. Yep. Andres, attached is a simple POC with an option & environment variable

Re: [HACKERS] pgbench randomness initialization

2016-04-07 Thread Fabien COELHO
Hello Andres, If you run the test for longer... Or explicitly iterate over IVs. At the very least we need to make pgbench output the IV used, to have some chance of repeating tests. Note that I'm not against providing a way to repeat tests "exactly", and I have suggested two means:

Re: [HACKERS] pgbench randomness initialization

2016-04-07 Thread Fabien COELHO
(2) runs which really vary from one to the next, so as to have an idea about how much it may vary, what is the performance stability. I don't think this POV makes all that much sense. If you do something non-comparable, then the results aren't, uh, comparable. Which also means

Re: [HACKERS] pgbench randomness initialization

2016-04-07 Thread Fabien COELHO
Hello Andres, et al I was wondering why it's a good idea for pgbench to do INSTR_TIME_SET_CURRENT(start_time); srandom((unsigned int) INSTR_TIME_GET_MICROSEC(start_time)); to initialize randomness and then for (i = 0; i < nthreads; i++)

Re: [HACKERS] large regression for parallel COPY

2016-04-06 Thread Fabien COELHO
Hello Robert, I tried the same test mentioned in the original post on cthulhu (EDB machine, CentOS 7.2, 8 sockets, 8 cores per socket, 2 threads per core, Xeon E7-8830 @ 2.13 GHz). I attempted to test both the effects of multi_extend_v21 and the *_flush_after settings. I'm not sure of

Re: [HACKERS] pgbench more operators & functions

2016-04-05 Thread Fabien COELHO
Please note that the checkpointer patch has two open items that perhaps you can help with --- see https://wiki.postgresql.org/wiki/Open_Items Indeed, I just looked at the commitfest, and I did not notice the other threads. I do not have an OSX available, but I'll have a look at the other

Re: [HACKERS] pgbench more operators & functions

2016-04-04 Thread Fabien COELHO
Hello Andres, I don't see much point in asking people to postpone. I do think however it can make sense to respond with something like: Fabien, you've been submitting a lot of patches over the last year. Thanks for the that! To keep up with the amount of incoming work the prject relies on

Re: [HACKERS] pgbench more operators & functions

2016-04-03 Thread Fabien COELHO
Hello Simon, Here is a simple patch... The patch deadline has passed and we are in the last CF of 9.6, as I'm sure you know. Yes I know, I'm ok with that, I was just putting stuff in the queue for later, I was not asking for the patch to be considered right now. Another minor patch on

[HACKERS] pgbench more operators & functions

2016-04-02 Thread Fabien COELHO
Hello, Here is a simple patch which adds a bunch of operators (bitwise: & | ^ ~, comparisons: =/== <>/!= < <= > >=, logical: and/&& or/|| xor/^^ not/!) and functions (exp ln if) to pgbench. I've tried to be pg's SQL compatible where appropriate. Also attached is a simple test script. Some

Re: [HACKERS] pgbench - remove unused clientDone parameter

2016-04-02 Thread Fabien COELHO
Remove pgbench clientDone unused "ok" parameter. Seems useless, yeah, removed. Good! Actually it was introduced as an unused argument in 3da0dfb4b146. Strange indeed. I did not thought to dig into the history. I switchted the corresponding commitfest entry as committed. Thanks, --

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-29 Thread Fabien COELHO
Please split the patch into one part for backporting and one part for master-only and post both patches, clearly indicating which is which. Attached are the full patch for head and the backport part (the patch name ends with "backport") separated. That's not really what I wanted; the full

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-29 Thread Fabien COELHO
Indeed. The documentation is manually edited when submitting changes so as to minimize diffs, but then it does not correspond anymore to any actual output, so it is easy to do it wrong. Well, you fixed the "latency stddev" line to the sample output too, but in my trial run that line was not

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-29 Thread Fabien COELHO
Hello, In doing this, I noticed that the latency output is wrong if you use -T instead of -t; it always says the latency is zero because "duration" is zero. I suppose it should be like in the attached instead. Indeed, I clearly overlooked option -t (transactions) which I never use. Patch

Re: [HACKERS] extend pgbench expressions with functions

2016-03-29 Thread Fabien COELHO
Hello Robert, If we don't nuke it, it'll never die. Hearing no objections, BOOM. FIZZ! :-) Thanks for the commits, and apology for the portability bugs. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] extend pgbench expressions with functions

2016-03-28 Thread Fabien COELHO
I guess the question here is whether we want the part-c patch, which removes the historical \setrandom syntax. That's technically no longer needed since we now can use random() as a function directly inside \set commands, but we might want it anyway for backward compatibility. This patch is

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-28 Thread Fabien COELHO
- that it does work:-) I'm not sure what happens by the script selection process, it should be checked carefully because it was not designed with allowing a zero weight, and it may depend on its/their positions. It may already work, but it really needs checking. Hmmm, it seems ok.

Re: [HACKERS] extend pgbench expressions with functions

2016-03-27 Thread Fabien COELHO
v40 is yet another rebase. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index c6d1454..4ceddae 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -815,9 +815,10 @@ pgbench options dbname - Sets variable

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Fabien COELHO
To emphasize potential bad effects without having to build too large a host and involve too many table spaces, I would suggest to reduce significantly the "checkpoint_flush_after" setting while running these tests. Meh, that completely distorts the test. Yep, I agree. The point would be to

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Fabien COELHO
My impression is that we actually know what we need to know anyway? Sure, the overall summary is "it is much better with the patch" on this large SSD test, which is good news because the patch was really designed to help with HDDs. -- Fabien. -- Sent via pgsql-hackers mailing list

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Fabien COELHO
You took 5% of the tx on two 12 hours runs, totaling say 85M tx on one and 100M tx on the other, so you get 4.25M tx from the first and 5M from the second. OK I'm saying that the percentile should be computed on the largest one (5M), so that you get a curve like the following, with both

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Fabien COELHO
WRT tablespaces: What I'm planning to do, unless somebody has a better proposal, is to basically rent two big amazon instances, and run pgbench in parallel over N tablespaces. Once with local SSD and once with local HDD storage. Ok. Not sure how to control that table spaces are actually on

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Fabien COELHO
1) regular-latency.png I'm wondering whether it would be clearer if the percentiles where relative to the largest sample, not to itself, so that the figures from the largest one would still be between 0 and 1, but the other (unpatched) one would go between 0 and 0.85, that is would be cut

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Fabien COELHO
Hello Tomas, Thanks again for these interesting benches. Overall, this means ~300M transactions in total for the un-throttled case, so sample with ~15M transactions available when computing the following charts. Still a very sizable run! There results (including scripts for generating the

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-21 Thread Fabien COELHO
Ok, I added a reference to the commitfest entry from this wiki page, and a note about partial 9.5 backporting. Please split the patch into one part for backporting and one part for master-only and post both patches, clearly indicating which is which. Attached are the full patch for head and

Re: [HACKERS] extend pgbench expressions with functions

2016-03-21 Thread Fabien COELHO
Bonjour Michaël, v39 is yet another rebase: 42 is in sight! What's up with v42? Is that your personal record? It is just a (geek) joke, see: https://en.wikipedia.org/wiki/42_%28number%29#Hitchhiker.27s_Guide_to_the_Galaxy It is the answer to the "Ultimate Question of Life, The Universe

Re: [HACKERS] extend pgbench expressions with functions

2016-03-20 Thread Fabien COELHO
v38 is a simple rebase, trying to keep up-to-date with Tom's work. v39 is yet another rebase: 42 is in sight! -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index c6d1454..4ceddae 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-20 Thread Fabien COELHO
Hello Jeff, So I wanted to do something like: for f in `seq 0 5 100`; do pgbench -T 180 -c8 -j8 -b tpcb-like@$f -b select-only@100 done; But, I'm not allowed to specify a weight of zero. Indeed. I did not envision such a use case, but it is quite legitimate and interesting! I would hope

Re: [HACKERS] extend pgbench expressions with functions

2016-03-19 Thread Fabien COELHO
v38 is a simple rebase, trying to keep up-to-date with Tom's work. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index dd3fb1d..cf9c1cd 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -802,9 +802,10 @@ pgbench options

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-19 Thread Fabien COELHO
Thanks for the pointer. However, I do not have "editor priviledge" on this wiki, maybe Tomas has? I gave you editor privs now, but since it's in 9.5 I guess it needs to be on the bug tracker (Except, of course, we don't have one.) Ok, I added a reference to the commitfest entry from this

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-19 Thread Fabien COELHO
I've created an entry in the next commit fest so that it is not lost, but I hope it will be committed before then. If this problem was introduced by commits in 9.6, this issue should be listed in the open items page, https://wiki.postgresql.org/wiki/Open_Items Thanks for the pointer.

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Fabien COELHO
Hello Tomas, Thanks for these great measures. * 4 x CPU E5-4620 (2.2GHz) 4*8 = 32 cores / 64 threads. * 256GB of RAM Wow! * 24x SSD on LSI 2208 controller (with 1GB BBWC) Wow! RAID configuration ? The patch is designed to fix very big issues on HDD, but it is good to see that the

Re: [HACKERS] extend pgbench expressions with functions

2016-03-19 Thread Fabien COELHO
Here is a v36 which inspect very carefully the string to decide whether it is an int or a double. You may, or may not, find it to your taste, I can't say. Here is a v37 which is mostly a rebase after recent changes. Also I noticed that I was double counting errors in the previous version, so

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-19 Thread Fabien COELHO
I've created an entry in the next commit fest so that it is not lost, but I hope it will be committed before then. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-19 Thread Fabien COELHO
Hello Álvaro, I pushed your 25, with some additional minor tweaks. I hope I didn't break anything; please test. I've made a few tests and all looks well. I guess the build farm will say if it does not like it. Thanks, -- Fabien. -- Sent via pgsql-hackers mailing list

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Fabien COELHO
Hello Tomas, But I do think it's a very useful tool when it comes to measuring the consistency of behavior over time, assuming you're asking questions about the intervals and not the original transactions. For a throttled run, I think it is better to check whether or not the system could

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Fabien COELHO
Is it possible to run tests with distinct table spaces on those many disks? Nope, that'd require reconfiguring the system (and then back), and I don't have access to that system (just SSH). Ok. Also, I don't quite see what would that tell us? Currently the flushing context is shared

Re: [HACKERS] [BUGS] pgbench -C -M prepared gives an error

2016-03-19 Thread Fabien COELHO
Hello Tatsuo, We're not resetting the prepared[] array when we pull the plug on an existing connection. Is a connection per transaction really a sane case to consider? Yes, I would think. This case reveals the connection overhead. We already are able to handle the simple query cases. Why

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-19 Thread Fabien COELHO
Hello Tomas, while learning about format of the transaction log produced by pgbench, I've noticed this sentence in the section describing format of the per-transaction log: The last field skipped_transactions reports the number of transactions skipped because they were too far behind

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-16 Thread Fabien COELHO
Hello Alvaro, If somebody specifies thousands of -f switches, they will waste a few bytes with each, but I'm hardly concerned about a few dozen kilobytes there ... Ok, so you prefer a memory leak. I hate it on principle. I don't "prefer" memory leaks -- I prefer interfaces that make sense.

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2016-03-15 Thread Fabien COELHO
I intend to have a look at it, I had a look at a previous instance, but I'm ok if someone wants to proceed. There's not exactly a long line of reviewers at the moment so if you could do a followup review that would be great. Ok. It is in the queue, not for right know, though. -- Fabien.

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2016-03-14 Thread Fabien COELHO
Hello David, Any takers to review this updated patch? I intend to have a look at it, I had a look at a previous instance, but I'm ok if someone wants to proceed. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] checkpointer continuous flushing

2016-03-10 Thread Fabien COELHO
I just pushed the two major remaining patches in this thread. Hurray! Nine months the this baby out:-) -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Fabien COELHO
As you wish. I thought that understanding the underlying performance model with sequential writes written in chunks is important for the admin, and as this guc would have an impact on performance it should be hinted about, including the limits of its effect where large bases will converge to

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Fabien COELHO
[...] If the default is in pages, maybe you could state it and afterwards translate it in size. Hm, I think that's more complicated for users than it's worth. As you wish. I liked the number of pages you used initially because it really gives a hint of how much random IOs are avoided when

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Fabien COELHO
Hello Andres, I'm not sure I've seen these performance... If you have hard evidence, please feel free to share it. Man, are you intentionally trying to be hard to work with? Sorry, I do not understand this remark. You were refering to some latency measures in your answer, and I was just

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Fabien COELHO
Whenever more than bgwriter_flush_after bytes have been written by the bgwriter, attempt to force the OS to issue these writes to the underlying storage. Doing so will limit the amount of dirty data in the kernel's page cache, reducing the likelihood of

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Fabien COELHO
[...] I had originally kept it with one context per tablespace after refactoring this, but found that it gave worse results in rate limited loads even over only two tablespaces. That's on SSDs though. Might just mean that a smaller context size is better on SSD, and it could still be

Re: [HACKERS] pgbench small bug fix

2016-03-09 Thread Fabien COELHO
OK, I've committed the fix for the -T part. It didn't back-patch cleanly, and it is a minor bug, so I'm not inclined to worry about it further. I agree that it is a very minor bug and not necessary worth back-patching. I didn't commit the fix for the -P part, because Alvaro objected to the

Re: [HACKERS] extend pgbench expressions with functions

2016-03-09 Thread Fabien COELHO
Hello Robert, [...] With your patch, you get different behavior depending on exactly how the input is malformed. I understand that you require only one possible error message on malformed input, instead of failing when converting to double if the input looked like a double (there was a '.'

Re: [HACKERS] pgbench small bug fix

2016-03-09 Thread Fabien COELHO
On Wed, Jan 27, 2016 at 2:31 PM, Fabien COELHO <coe...@cri.ensmp.fr> wrote: - when a duration (-T) is specified, ensure that pgbench ends at that time (i.e. do not wait for a transaction beyond the end of the run). Every other place where doCustom() returns false is implemented as

Re: [HACKERS] extend pgbench expressions with functions

2016-03-09 Thread Fabien COELHO
Hello Robert, Here is a v35 b & c. This is not acceptable: + /* guess double type (n for "inf", "-inf" and "nan") */ + if (strchr(var, '.') != NULL || strchr(var, 'n') != NULL) + { +

Re: [HACKERS] extend pgbench expressions with functions

2016-03-08 Thread Fabien COELHO
Hello Robert. Here is a v34 b & c. // comments are not allowed. I'd just remove the two you have. Back to the eighties! It make no sense to exit(1) and then return 0, so don't do that. I might write this code as: This would get rid of the internal-error case here altogether in favor of

Re: [HACKERS] extend pgbench expressions with functions

2016-03-08 Thread Fabien COELHO
Hello Robert, Having a look at 33-b, this looks pretty good now, but: // comments are not allowed. I'd just remove the two you have. Oops, C89 did not make it everywhere yet! It make no sense to exit(1) and then return 0, so don't do that. [...] This would get rid of the internal-error

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-08 Thread Fabien COELHO
Now I cannot see how having one context per table space would have a significant negative performance impact. The 'dirty data' etc. limits are global, not per block device. By having several contexts with unflushed dirty data the total amount of dirty data in the kernel increases. Possibly,

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-07 Thread Fabien COELHO
Hello Andres, Now I cannot see how having one context per table space would have a significant negative performance impact. The 'dirty data' etc. limits are global, not per block device. By having several contexts with unflushed dirty data the total amount of dirty data in the kernel

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-07 Thread Fabien COELHO
Hello Andres, (1) with 16 tablespaces (1 per table) on 1 disk : 680.0 tps per second avg, stddev [ min q1 median d3 max ] <=300tps 679.6 ± 750.4 [0.0, 317.0, 371.0, 438.5, 2724.0] 19.5% (2) with 1 tablespace on 1 disk : 956.0 tps per second avg, stddev [ min q1 median d3 max ]

Re: [HACKERS] extend pgbench expressions with functions

2016-03-07 Thread Fabien COELHO
- 32-b: add double functions, including double variables - 32-c: remove \setrandom support (advice to use \set + random instead) Here is a rebased version after Tom's updates, 33-b & 33-c. I also extended the floating point syntax to signed accept signed exponents, and changed the regexpr

Re: [HACKERS] extend pgbench expressions with functions

2016-03-04 Thread Fabien COELHO
Attached is the fixed patch for the array method. Committed with a few tweaks, including running pgindent over some of it. Thanks. So the first set of functions is in, and the operators are executed as functions as well. Fabien, are you planning to send rebased versions of the rest? By that

Re: [HACKERS] extend pgbench expressions with functions

2016-03-04 Thread Fabien COELHO
Committed with a few tweaks, including running pgindent over some of it. Thanks! -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench small bug fix

2016-03-04 Thread Fabien COELHO
You're probably right, but TBH I'm pretty unsure about this whole thing. If the question is "is there a bug", then answer is yes. The progress report may disappear if thread 0 happens to stop, even of all other threads go on. Obviously it only concerns slow queries, but there is no reason why

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-04 Thread Fabien COELHO
That is why the "fs" variable in process_file is declared "static", and why I wrote "some hidden awkwarness". I did want to avoid a malloc because then who would free the struct? addScript cannot to it systematically because builtins are static. Or it would have to create an on purpose struct,

Re: [HACKERS] pgbench small bug fix

2016-03-04 Thread Fabien COELHO
Probably it is possible, but it will sure need more that one little condition to be achieved... I do not think that introducing a non trivial distributed election algorithm involving locks and so would be a good decision for this very little matter. My advice is "keep it simple". If this is a

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-04 Thread Fabien COELHO
*-21.patch does what you suggested above, some hidden awkwardness but much less that the previous one. Yeah, I think this is much nicer, don't you agree? Yep, I said "less awkwarness than previous", a pretty contrived way to say better:-) However, this is still a bit broken --

Re: [HACKERS] pgbench small bug fix

2016-03-04 Thread Fabien COELHO
Hello Alvaro, Attached is a v3 which test integers more logically. I'm a lazy programmer who tends to minimize the number of key strokes. Well. From what I can tell this patch is Ready for Committer. I'm not a fan of this approach either. Would it be too complicated if we had a global

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-04 Thread Fabien COELHO
Hello Alvaro, I looked at 19.d and I think the design has gotten pretty convoluted. I think we could simplify with the following changes: struct script_t gets a new member, of type Command **, which is initially null. function process_builtin receives the complete script_t (not individual

Re: [HACKERS] pgbench small bug fix

2016-03-03 Thread Fabien COELHO
Hello Aleksander, Thanks for the look at the patch. time pgbench -T 5 -R 0.1 -P 1 -c 2 -j 2 On my laptop this command executes 25 seconds instead of 5. I'm pretty sure it IS a bug. Probably a minor one though. Sure. [...] you should probably write: if(someint > 0) Ok. if(somebool

Re: [HACKERS] checkpointer continuous flushing - V16

2016-03-01 Thread Fabien COELHO
Hello Tomas, One of the goals of this thread (as I understand it) was to make the overall behavior smoother - eliminate sudden drops in transaction rate due to bursts of random I/O etc. One way to look at this is in terms of how much the tps fluctuates, so let's see some charts. I've

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-22 Thread Fabien COELHO
Random updates on 16 tables which total to 1.1GB of data, so this is in buffer, no significant "read" traffic. (1) with 16 tablespaces (1 per table) on 1 disk : 680.0 tps per second avg, stddev [ min q1 median d3 max ] <=300tps 679.6 ± 750.4 [0.0, 317.0, 371.0, 438.5, 2724.0] 19.5%

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-22 Thread Fabien COELHO
I did a quick & small test with random updates on 16 tables with checkpoint_flush_after=16 checkpoint_timeout=30 Another run with more "normal" settings and over 1000 seconds, so less "quick & small" that the previous one. checkpoint_flush_after = 16 checkpoint_timeout = 5min # default

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-22 Thread Fabien COELHO
Hallo Andres, AFAICR I used a "flush context" for each table space in some version I submitted, because I do think that this whole writeback logic really does make sense *per table space*, which suggest that there should be as many write backs contexts as table spaces, otherwise the positive

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-21 Thread Fabien COELHO
ISTM that "progress" and "progress_slice" only depend on num_scanned and per-tablespace num_to_scan and total num_to_scan, so they are somehow redundant and the progress could be recomputed from the initial figures when needed. They don't cause much space usage, and we access the values

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-21 Thread Fabien COELHO
[...] I do think that this whole writeback logic really does make sense *per table space*, Leads to less regular IO, because if your tablespaces are evenly sized (somewhat common) you'll sometimes end up issuing sync_file_range's shortly after each other. For latency outside checkpoints it's

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-21 Thread Fabien COELHO
Hallo Andres, Here is a review for the second patch. For 0002 I've recently changed: * Removed the sort timing information, we've proven sufficiently that it doesn't take a lot of time. I put it there initialy to demonstrate that there was no cache performance issue when sorting on just

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-21 Thread Fabien COELHO
Hallo Andres, [...] I do think that this whole writeback logic really does make sense *per table space*, Leads to less regular IO, because if your tablespaces are evenly sized (somewhat common) you'll sometimes end up issuing sync_file_range's shortly after each other. For latency outside

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-20 Thread Fabien COELHO
Hallo Andres, In some previous version I think a warning was shown if the feature was requested but not available. I think we should either silently ignore it, or error out. Warnings somewhere in the background aren't particularly meaningful. I like "ignoring with a warning" in the log

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-20 Thread Fabien COELHO
Hello Andres, For 0001 I've recently changed: * Don't schedule writeback after smgrextend() - that defeats linux delayed allocation mechanism, increasing fragmentation noticeably. * Add docs for the new GUC variables * comment polishing * BackendWritebackContext now isn't dynamically

Re: [HACKERS] checkpointer continuous flushing - V18

2016-02-19 Thread Fabien COELHO
Hello Andres, Here's the next two (the most important) patches of the series: 0001: Allow to trigger kernel writeback after a configurable number of writes. 0002: Checkpoint sorting and balancing. I will look into these two in depth. Note that I would have ordered them in reverse because

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-19 Thread Fabien COELHO
Hallo Patric, Kernel 3.2 is extremely bad for Postgresql, as the vm seems to amplify IO somehow. The difference to 3.13 (the latest LTS kernel for 12.04) is huge.

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-19 Thread Fabien COELHO
Hello. Based on these results I think 32 will be a good default for checkpoint_flush_after? There's a few cases where 64 showed to be beneficial, and some where 32 is better. I've seen 64 perform a bit better in some cases here, but the differences were not too big. Yes, these many runs show

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-19 Thread Fabien COELHO
Hello Andres, I don't want to post a full series right now, but my working state is available on http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/checkpoint-flush git://git.postgresql.org/git/users/andresfreund/postgres.git checkpoint-flush Below

Re: [HACKERS] checkpointer continuous flushing - V16

2016-02-18 Thread Fabien COELHO
Hello Andres, 0001: Make SetHintBit() a bit more aggressive, afaics that fixes all the potential regressions of 0002 0002: Fix the overaggressive flushing by the wal writer, by only flushing every wal_writer_delay ms or wal_writer_flush_after bytes. I've looked at these

Re: [HACKERS] extend pgbench expressions with functions

2016-02-17 Thread Fabien COELHO
Hello Michaël, \set aid 1 + 1 pgbench -f addition.sql -t 5000 I have the following: HEAD: 3.5~3.7M TPS list method: 3.6~3.7M TPS array method: 3.4~3.5M TPS So all approaches have a comparable performance. Yep, the execution trace is pretty similar in all cases, maybe with a little more

Re: [HACKERS] extend pgbench expressions with functions

2016-02-16 Thread Fabien COELHO
Hello Robert, However, for obvious reasons the committer opinion prevails:-) You're welcome to solicit other opinions. I'm not unwilling to give way if there's a chorus of support for the way you've done it. Hmmm. I do not expect much chorus on such a minor subject:-) But to me it sounds

Re: [HACKERS] extend pgbench expressions with functions

2016-02-16 Thread Fabien COELHO
Hello Robert, Good point. One simple idea here would be to use a custom pgbench script that has no SQL commands and just calculates the values of some parameters to measure the impact without depending on the backend, with a fixed number of transactions. Sure, we could do that. But whether

Re: [HACKERS] extend pgbench expressions with functions

2016-02-16 Thread Fabien COELHO
Hello Robert, [...] But we can't have things that are logically part of patch #2 just tossed in with patch #1. So you want integer functions without type in patch #1. I was in the middle of ripping that out of the patch when I realized that evalFunc() is pretty badly designed. Probably,

Re: [HACKERS] innocuous: pgbench does FD_ISSET on invalid socket

2016-02-15 Thread Fabien COELHO
Hello Alvaro, Any objections to changing it like this? I'd probably backpatch to 9.5, but no further (even though this pattern actually appears all the way back.) My 0.02€: if the pattern is repeated, maybe a function which incorporates the check would save lines and improve overall

<    2   3   4   5   6   7   8   9   10   11   >