Re: [HACKERS] Poorly-thought-out handling of double variables in pgbench

[Tom Lane]

Fabien COELHO  writes:
>> Neither way allows for trailing whitespace, though.  I don't see that
>> that's important here.

> No, this is not an issue here for variables specified from the command 
> line. Consider pushing this fix?

Done already.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Poorly-thought-out handling of double variables in pgbench

[Fabien COELHO]

While testing the patch I found a minor preexisting (mine...) bug: when
string-scanning doubles, whether the whole string is consumed or not is
not checked. This means that -D x=0one is interpreted as double 0.



I came up with the attached check, but maybe there is a cleaner way to do
that.


I like the way the zic code does it:



+   char xs;
!   if (sscanf(var->value, "%lf%c", , ) != 1)


Indeed... I cannot say that it is cleaner such, but at least it is short 
and it avoids the strlen call.



Neither way allows for trailing whitespace, though.  I don't see that
that's important here.


No, this is not an issue here for variables specified from the command 
line. Consider pushing this fix?


--
Fabien.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Poorly-thought-out handling of double variables in pgbench

[Tom Lane]

Fabien COELHO  writes:
> This is a definite improvement.

> I like the lazyness between string & numeric forms, and for sorting, that 
> is what was needed doing to have something clean.

> Applied on head, it works for me.

OK, pushed.

> While testing the patch I found a minor preexisting (mine...) bug: when 
> string-scanning doubles, whether the whole string is consumed or not is 
> not checked. This means that -D x=0one is interpreted as double 0.

> I came up with the attached check, but maybe there is a cleaner way to do 
> that.

I like the way the zic code does it:

else /* type should be double */
{
double dv;
  
!   if (sscanf(var->value, "%lf", ) != 1)
{
fprintf(stderr,
"malformed variable \"%s\" value: 
\"%s\"\n",
--- 928,936 
else /* type should be double */
{
double dv;
+   char xs;
  
!   if (sscanf(var->value, "%lf%c", , ) != 1)
{
fprintf(stderr,
"malformed variable \"%s\" value: 
\"%s\"\n",

This relies on the idea that if there is any trailing junk, one character
will get assigned into "xs" and then sscanf will say it filled two fields.
Otherwise, it'll report filling only one field (or none, if there's
no number either).  This requires only a minimal amount of code and no
extra strlen() calculations.

Neither way allows for trailing whitespace, though.  I don't see that
that's important here.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Poorly-thought-out handling of double variables in pgbench

[Fabien COELHO]

Hello Tom,


That's probably a bigger change than we want to be putting in right now,
though I'm a bit tempted to go try it.



I definitely agree that the text variable solution is pretty ugly, but it
was the minimum change solution, and I do not have much time available.


Well, I felt like doing some minor hacking, so I went and adjusted the
code to work this way.  I'm pretty happy with the result, what do you
think?


This is a definite improvement.

I like the lazyness between string & numeric forms, and for sorting, that 
is what was needed doing to have something clean.


Applied on head, it works for me.

While testing the patch I found a minor preexisting (mine...) bug: when 
string-scanning doubles, whether the whole string is consumed or not is 
not checked. This means that -D x=0one is interpreted as double 0.


I came up with the attached check, but maybe there is a cleaner way to do 
that.


--
Fabien.diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a484165..9673640 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -928,8 +928,10 @@ makeVariableNumeric(Variable *var)
 	else /* type should be double */
 	{
 		double dv;
+		int consumed;
 
-		if (sscanf(var->value, "%lf", ) != 1)
+		if (sscanf(var->value, "%lf%n", , ) != 1 ||
+			consumed != strlen(var->value))
 		{
 			fprintf(stderr,
 	"malformed variable \"%s\" value: \"%s\"\n",

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Poorly-thought-out handling of double variables in pgbench

[Tom Lane]

Fabien COELHO  writes:
>> A better answer, perhaps, would be to store double-valued variables in 
>> double format to begin with, coercing to text only when and if the value 
>> is interpolated into a string.

> Yep, but that was yet more changes for a limited benefit and would have 
> increase the probability that the patch would have been rejected.

>> That's probably a bigger change than we want to be putting in right now, 
>> though I'm a bit tempted to go try it.

> I definitely agree that the text variable solution is pretty ugly, but it 
> was the minimum change solution, and I do not have much time available.

Well, I felt like doing some minor hacking, so I went and adjusted the
code to work this way.  I'm pretty happy with the result, what do you
think?

regards, tom lane

diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 2a9063a..a484165 100644
*** a/src/bin/pgbench/pgbench.c
--- b/src/bin/pgbench/pgbench.c
***
*** 38,43 
--- 38,44 
  #include "portability/instr_time.h"
  
  #include 
+ #include 
  #include 
  #include 
  #include 
*** const char *progname;
*** 185,195 
  
  volatile bool timer_exceeded = false;	/* flag from signal handler */
  
! /* variable definitions */
  typedef struct
  {
! 	char	   *name;			/* variable name */
! 	char	   *value;			/* its value */
  } Variable;
  
  #define MAX_SCRIPTS		128		/* max number of SQL scripts allowed */
--- 186,205 
  
  volatile bool timer_exceeded = false;	/* flag from signal handler */
  
! /*
!  * Variable definitions.  If a variable has a string value, "value" is that
!  * value, is_numeric is false, and num_value is undefined.  If the value is
!  * known to be numeric, is_numeric is true and num_value contains the value
!  * (in any permitted numeric variant).  In this case "value" contains the
!  * string equivalent of the number, if we've had occasion to compute that,
!  * or NULL if we haven't.
!  */
  typedef struct
  {
! 	char	   *name;			/* variable's name */
! 	char	   *value;			/* its value in string form, if known */
! 	bool		is_numeric;		/* is numeric value known? */
! 	PgBenchValue num_value;		/* variable's value in numeric form */
  } Variable;
  
  #define MAX_SCRIPTS		128		/* max number of SQL scripts allowed */
*** typedef struct
*** 237,243 
  	bool		throttling;		/* whether nap is for throttling */
  	bool		is_throttled;	/* whether transaction throttling is done */
  	Variable   *variables;		/* array of variable definitions */
! 	int			nvariables;
  	int64		txn_scheduled;	/* scheduled start time of transaction (usec) */
  	instr_time	txn_begin;		/* used for measuring schedule lag times */
  	instr_time	stmt_begin;		/* used for measuring statement latencies */
--- 247,254 
  	bool		throttling;		/* whether nap is for throttling */
  	bool		is_throttled;	/* whether transaction throttling is done */
  	Variable   *variables;		/* array of variable definitions */
! 	int			nvariables;		/* number of variables */
! 	bool		vars_sorted;	/* are variables sorted by name? */
  	int64		txn_scheduled;	/* scheduled start time of transaction (usec) */
  	instr_time	txn_begin;		/* used for measuring schedule lag times */
  	instr_time	stmt_begin;		/* used for measuring statement latencies */
*** static const BuiltinScript builtin_scrip
*** 363,368 
--- 374,381 
  
  
  /* Function prototypes */
+ static void setIntValue(PgBenchValue *pv, int64 ival);
+ static void setDoubleValue(PgBenchValue *pv, double dval);
  static bool evaluateExpr(TState *, CState *, PgBenchExpr *, PgBenchValue *);
  static void doLog(TState *thread, CState *st, instr_time *now,
  	  StatsData *agg, bool skipped, double latency, double lag);
*** discard_response(CState *state)
*** 836,868 
  	} while (res);
  }
  
  static int
! compareVariables(const void *v1, const void *v2)
  {
  	return strcmp(((const Variable *) v1)->name,
    ((const Variable *) v2)->name);
  }
  
! static char *
! getVariable(CState *st, char *name)
  {
! 	Variable	key,
! 			   *var;
  
  	/* On some versions of Solaris, bsearch of zero items dumps core */
  	if (st->nvariables <= 0)
  		return NULL;
  
  	key.name = name;
! 	var = (Variable *) bsearch((void *) ,
! 			   (void *) st->variables,
! 			   st->nvariables,
! 			   sizeof(Variable),
! 			   compareVariables);
! 	if (var != NULL)
! 		return var->value;
  	else
! 		return NULL;
  }
  
  /* check whether the name consists of alphabets, numerals and underscores. */
--- 849,945 
  	} while (res);
  }
  
+ /* qsort comparator for Variable array */
  static int
! compareVariableNames(const void *v1, const void *v2)
  {
  	return strcmp(((const Variable *) v1)->name,
    ((const Variable *) v2)->name);
  }
  
! /* Locate a variable by name; returns NULL if unknown */
! static Variable *
! lookupVariable(CState *st, char *name)
  {
! 	Variable	key;
  
  	

Re: [HACKERS] Poorly-thought-out handling of double variables in pgbench

[Fabien COELHO]

Hello Tom,


While testing 7a622b273 I happened to notice this:

\set x greatest(3, 2, 4.9)
create table mytab (x numeric);
insert into mytab values(:x);
 x
--
4.900355

The reason for that is that the result of a "double" calculation
is coerced to text like this:

   sprintf(res, "%.18e", result.u.dval);

apparently on the theory that this will result in being able to convert it
back to double with no loss of low-order bits.


Yep. I did that for this very reason.

ISTM that the alternative was to reimplement the variable stuff to add & 
manage types, but that induces significant changes, and such change are 
likely to be rejected, or at least to raise long debates and questions, 
make the patch harder to check... so I just avoided them.


But of course the last two or three digits of such output are pretty 
much garbage to the naked eye.


Sure, it is binary to decimal conversion noise.

Then what gets interpolated as the variable value is something like 
'4.900355e+00'.


I think this is a very poor decision; it's going to cause surprises and 
probably bug reports.


Moreover, if we were testing this behavior in the buildfarm (which we 
are not, but only because the test coverage for pgbench is so shoddy),


I think that "none" is the right word?

we would be seeing failures, because those low-order digits are going to 
be platform dependent.


Possibly.


The only value of doing it like this would be if people chained multiple
floating-point calculations in successive pgbench \set commands and needed
full precision to be retained from one \set to the next ... but how likely
is that?


The constraint for me is that a stored double should not lose precision. 
Any solution which achieves this goal is fine with me.



A minimum-change fix would be to print only DBL_DIG digits here.


Probably 15, but there is some rounding implied I think (?). I'm not that 
sure of DBL_DIG+1, so I put 18 to be safer. I did not envision that 
someone would store that in a NUMERIC:-) Your example would work fine with 
a DOUBLE PRECISION attribute.


A better answer, perhaps, would be to store double-valued variables in 
double format to begin with, coercing to text only when and if the value 
is interpolated into a string.


Yep, but that was yet more changes for a limited benefit and would have 
increase the probability that the patch would have been rejected.


That's probably a bigger change than we want to be putting in right now, 
though I'm a bit tempted to go try it.



Thoughts?


My 0.02€:

I definitely agree that the text variable solution is pretty ugly, but it 
was the minimum change solution, and I do not have much time available.


I do not think that rounding values is desirable, on principle.

Now doubles are only really there because random_*() functions need one 
double argument, otherwise only integers make much sense as keys to select 
tuples in a performance bench. So this is not a key feature, just a side 
effect of having more realistic non uniform random generators and 
expressions. In an early submission I did not bother to include double 
variables because I cannot see much actual use to them, and there were 
implementation issues given how integer variables were managed.


Also I think that the above NUMERIC/DOUBLE case is a little contrived, so 
I would wait for someone to actually complain with a good use case before 
spending any significant time to change how variables are stored in 
pgbench.


I would expect a long review process for such a patch if I submitted it, 
because there are details and the benefit is quite limited.



BTW, just to add insult to injury, the debug() function prints double
values with "%.f", which evidently had even less thought put into it.


AFAICR the thought I put into it is that it is definitely useful to be 
able to get a simple trace for debugging purpose. The precision of this 
debug output is not very important, so I probably just put the simplest 
possible format.



That one should definitely be %g with DBL_DIG precision.


That would be fine as well.

--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers