Also, maybe per-command detailed stats should use the same common struct
to hold data as all other stats. I did not change it because it is
maintained in a different part of the code.
I played just once with the --report-latencies option and was astonished
that meta commands showed negative latencies...
This v5 also fixes this bug (on meta commands there is a goto loop in
doCustom, but as now was not reset the stmt_begin ended up being after
now, hence accumulating increasing negative times) and in passing uses the
same stats structure as the rest, which result in removing some more code.
The "report-latencies" option is made to imply per script stats, which
simplifies the final output code, and if you want per-command per-script
stats, probably providing the per-script stats, i.e. the sum of the
commands, make sense.
--
Fabien.
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2517a3a..dc36f5d 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -260,7 +260,22 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<application>pgbench</application> accepts the following command-line
benchmarking arguments:
- <variablelist>
+ <term><option>-b</> <replaceable>scriptname[@weight]</></term>
+ <term><option>--builtin</> <replaceable>scriptname[@weight]</></term>
+ <listitem>
+ <para>
+ Add the specified builtin script to the list of executed scripts.
+ An optional integer weight after <literal>@</> allows to adjust the
+ probability of drawing the test.
+ Available builtin scripts are: <literal>tpcb-like</>,
+ <literal>simple-update</> and <literal>select-only</>.
+ The provided <repleacable>scriptname</> needs only to be a prefix
+ of the builtin name, hence <literal>simp</> would be enough to select
+ <literal>simple-update</>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-c</option> <replaceable>clients</></term>
@@ -307,14 +322,15 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</varlistentry>
<varlistentry>
- <term><option>-f</option> <replaceable>filename</></term>
- <term><option>--file=</option><replaceable>filename</></term>
+ <term><option>-f</> <replaceable>filename[@weight]</></term>
+ <term><option>--file=</><replaceable>filename[@weight]</></term>
<listitem>
<para>
- Read transaction script from <replaceable>filename</>.
+ Add a transaction script read from <replaceable>filename</> to
+ the list of executed scripts.
+ An optional integer weight after <literal>@</> allows to adjust the
+ probability of drawing the test.
See below for details.
- <option>-N</option>, <option>-S</option>, and <option>-f</option>
- are mutually exclusive.
</para>
</listitem>
</varlistentry>
@@ -404,10 +420,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<term><option>--skip-some-updates</option></term>
<listitem>
<para>
- Do not update <structname>pgbench_tellers</> and
- <structname>pgbench_branches</>.
- This will avoid update contention on these tables, but
- it makes the test case even less like TPC-B.
+ Shorthand for <option>-b simple-update@1</>.
</para>
</listitem>
</varlistentry>
@@ -499,9 +512,9 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
Report the specified scale factor in <application>pgbench</>'s
output. With the built-in tests, this is not necessary; the
correct scale factor will be detected by counting the number of
- rows in the <structname>pgbench_branches</> table. However, when testing
- custom benchmarks (<option>-f</> option), the scale factor
- will be reported as 1 unless this option is used.
+ rows in the <structname>pgbench_branches</> table.
+ However, when testing only custom benchmarks (<option>-f</> option),
+ the scale factor will be reported as 1 unless this option is used.
</para>
</listitem>
</varlistentry>
@@ -511,7 +524,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<term><option>--select-only</option></term>
<listitem>
<para>
- Perform select-only transactions instead of TPC-B-like test.
+ Shorthand for <option>-b select-only@1</>.
</para>
</listitem>
</varlistentry>
@@ -567,6 +580,16 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</varlistentry>
<varlistentry>
+ <term><option>--per-script-stats</option></term>
+ <listitem>
+ <para>
+ Report some statistics per script run by pgbench.
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
<term><option>--sampling-rate=<replaceable>rate</></option></term>
<listitem>
<para>
@@ -661,7 +684,20 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<title>What is the <quote>Transaction</> Actually Performed in pgbench?</title>
<para>
- The default transaction script issues seven commands per transaction:
+ Pgbench executes test scripts chosen randomly from a specified list.
+ They include built-in scripts with <option>-b</> and
+ user-provided custom scripts with <option>-f</>.
+ Each script may be given a relative weight specified after a
+ <literal>@</> so as to change its drawing probability.
+ The default weight is <literal>1</>.
+ </para>
+
+ <para>
+ The default builtin transaction script (also invoked with <option>-b tpcb-like</>)
+ issues seven commands per transaction over randomly chosen <literal>aid</>,
+ <literal>tid</>, <literal>bid</> and <literal>balance</>.
+ The scenario is inspired by the TPC-B benchmark, but is not actually TPC-B,
+ hence the name.
</para>
<orderedlist>
@@ -675,9 +711,15 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</orderedlist>
<para>
- If you specify <option>-N</>, steps 4 and 5 aren't included in the
- transaction. If you specify <option>-S</>, only the <command>SELECT</> is
- issued.
+ If you select the <literal>simple-update</> builtin (also <option>-N</>),
+ steps 4 and 5 aren't included in the transaction.
+ This will avoid update contention on these tables, but
+ it makes the test case even less like TPC-B.
+ </para>
+
+ <para>
+ If you select the <literal>select-only</> builtin (also <option>-S</>),
+ only the <command>SELECT</> is issued.
</para>
</refsect2>
@@ -689,10 +731,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
benchmark scenarios by replacing the default transaction script
(described above) with a transaction script read from a file
(<option>-f</option> option). In this case a <quote>transaction</>
- counts as one execution of a script file. You can even specify
- multiple scripts (multiple <option>-f</option> options), in which
- case a random one of the scripts is chosen each time a client session
- starts a new transaction.
+ counts as one execution of a script file.
</para>
<para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e839fa3..98a88f9 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -162,11 +162,10 @@ bool use_log; /* log transaction latencies to a file */
bool use_quiet; /* quiet logging onto stderr */
int agg_interval; /* log aggregates instead of individual
* transactions */
+bool per_script_stats = false; /* whether to collect stats per script */
int progress = 0; /* thread progress report every this seconds */
-int progress_nclients = 0; /* number of clients for progress
- * report */
-int progress_nthreads = 0; /* number of threads for progress
- * report */
+int nclients = 1; /* number of clients */
+int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
int main_pid; /* main process id used in log filename */
@@ -177,6 +176,8 @@ char *login = NULL;
char *dbName;
const char *progname;
+#define WSEP '@' /* weight separator */
+
volatile bool timer_exceeded = false; /* flag from signal handler */
/* variable definitions */
@@ -186,13 +187,39 @@ typedef struct
char *value; /* its value */
} Variable;
-#define MAX_FILES 128 /* max number of SQL script files allowed */
+#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
/*
- * structures used in custom query mode
+ * simple data structure to keep stats about something.
+ * probably the first value should be kept and used as an offset for
+ * better numerical stability...
+ */
+typedef struct
+{
+ int64_t count; /* how many values where encountered */
+ double min; /* the minimum seen */
+ double max; /* the maximum seen */
+ double sum; /* sum of values */
+ double sum2; /* sum of squared values */
+} SimpleStats;
+
+/* data structure to hold various statistics.
+ * it is used for interval statistics as well as file statistics.
*/
+typedef struct
+{
+ long start_time; /* when the interval starts, for aggregates */
+ int64 cnt; /* number of transactions */
+ int64 skipped; /* number of transactions skipped under --rate
+ * and --latency-limit */
+ SimpleStats latency;
+ SimpleStats lag;
+} StatsData;
+/*
+ * structures used in custom query mode
+ */
typedef struct
{
PGconn *con; /* connection handle to DB */
@@ -200,22 +227,20 @@ typedef struct
int state; /* state No. */
int listen; /* 0 indicates that an async query has been
* sent */
- int sleeping; /* 1 indicates that the client is napping */
+ bool is_throttled; /* whether transaction throttling is done */
+ bool sleeping; /* whether the client is napping */
bool throttling; /* whether nap is for throttling */
Variable *variables; /* array of variable definitions */
int nvariables;
int64 txn_scheduled; /* scheduled start time of transaction (usec) */
instr_time txn_begin; /* used for measuring schedule lag times */
instr_time stmt_begin; /* used for measuring statement latencies */
- bool is_throttled; /* whether transaction throttling is done */
- int use_file; /* index in sql_files for this client */
- bool prepared[MAX_FILES];
+ int use_file; /* index in sql_scripts for this client */
+ bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
/* per client collected stats */
- int cnt; /* xacts count */
+ int64 cnt; /* transaction count */
int ecnt; /* error count */
- int64 txn_latencies; /* cumulated latencies */
- int64 txn_sqlats; /* cumulated square latencies */
} CState;
/*
@@ -227,19 +252,14 @@ typedef struct
pthread_t thread; /* thread handle */
CState *state; /* array of CState */
int nstate; /* length of state[] */
- instr_time start_time; /* thread start time */
- instr_time *exec_elapsed; /* time spent executing cmds (per Command) */
- int *exec_count; /* number of cmd executions (per Command) */
- unsigned short random_state[3]; /* separate randomness for each thread */
- int64 throttle_trigger; /* previous/next throttling (us) */
+ unsigned short random_state[3]; /* separate randomness for each thread */
+ int64 throttle_trigger; /* previous/next throttling (us) */
/* per thread collected stats */
+ instr_time start_time; /* thread start time */
instr_time conn_time;
- int64 throttle_lag; /* total transaction lag behind throttling */
- int64 throttle_lag_max; /* max transaction lag */
- int64 throttle_latency_skipped; /* lagging transactions
- * skipped */
- int64 latency_late; /* late transactions */
+ StatsData stats;
+ int64 latency_late; /* executed but late transactions */
} TState;
#define INVALID_THREAD ((pthread_t) 0)
@@ -271,35 +291,41 @@ typedef struct
char *argv[MAX_ARGS]; /* command word list */
int cols[MAX_ARGS]; /* corresponding column starting from 1 */
PgBenchExpr *expr; /* parsed expression */
+ SimpleStats stats; /* time spent in this command */
} Command;
+/* SQL script to be executed, either a file or an internal script
+ */
typedef struct
{
-
- long start_time; /* when does the interval start */
- int cnt; /* number of transactions */
- int skipped; /* number of transactions skipped under --rate
- * and --latency-limit */
-
- double min_latency; /* min/max latencies */
- double max_latency;
- double sum_latency; /* sum(latency), sum(latency^2) - for
- * estimates */
- double sum2_latency;
-
- double min_lag;
- double max_lag;
- double sum_lag; /* sum(lag) */
- double sum2_lag; /* sum(lag*lag) */
-} AggVals;
-
-static Command **sql_files[MAX_FILES]; /* SQL script files */
-static int num_files; /* number of script files */
+ const char *name;
+ int weight;
+ Command **commands;
+ StatsData stats;
+} SQLScript;
+
+static SQLScript sql_script[MAX_SCRIPTS];
+static int num_scripts; /* number of script in sql_script[] */
static int num_commands = 0; /* total number of Command structs */
+static int total_weight = 0;
+
static int debug = 0; /* debug flag */
-/* default scenario */
-static char *tpc_b = {
+/* Function prototypes */
+static void setalarm(int seconds);
+static void *threadRun(void *arg);
+static void doTxStats(TState*, CState*, instr_time*, bool, FILE*, StatsData*);
+
+/* Define builtin test scripts */
+#define N_BUILTIN 3
+static struct {
+ char *name; /* very short name for -b ...*/
+ char *desc; /* short description */
+ char *script; /* actual pgbench script */
+} builtin_script[] = {
+{
+ "tpcb-like",
+ "<builtin: TPC-B (sort of)>",
"\\set nbranches " CppAsString2(nbranches) " * :scale\n"
"\\set ntellers " CppAsString2(ntellers) " * :scale\n"
"\\set naccounts " CppAsString2(naccounts) " * :scale\n"
@@ -314,10 +340,10 @@ static char *tpc_b = {
"UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;\n"
"INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);\n"
"END;\n"
-};
-
-/* -N case */
-static char *simple_update = {
+},
+{
+ "simple-update",
+ "<builtin: simple update>",
"\\set nbranches " CppAsString2(nbranches) " * :scale\n"
"\\set ntellers " CppAsString2(ntellers) " * :scale\n"
"\\set naccounts " CppAsString2(naccounts) " * :scale\n"
@@ -330,21 +356,32 @@ static char *simple_update = {
"SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n"
"INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);\n"
"END;\n"
-};
-
-/* -S case */
-static char *select_only = {
+},
+{
+ "select-only",
+ "<builtin: select only>",
"\\set naccounts " CppAsString2(naccounts) " * :scale\n"
"\\setrandom aid 1 :naccounts\n"
"SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n"
-};
+} };
-/* Function prototypes */
-static void setalarm(int seconds);
-static void *threadRun(void *arg);
+static char *
+find_builtin(const char *name, char **desc)
+{
+ int len = strlen(name), i;
+
+ for (i = 0; i < N_BUILTIN; i++)
+ {
+ if (strncmp(builtin_script[i].name, name, len) == 0)
+ {
+ *desc = builtin_script[i].desc;
+ return builtin_script[i].script;
+ }
+ }
-static void doLog(TState *thread, CState *st, FILE *logfile, instr_time *now,
- AggVals *agg, bool skipped);
+ fprintf(stderr, "no builtin found for \"%s\"\n", name);
+ exit(1);
+}
static void
usage(void)
@@ -364,11 +401,13 @@ usage(void)
" --tablespace=TABLESPACE create tables in the specified tablespace\n"
" --unlogged-tables create tables as unlogged tables\n"
"\nBenchmarking options:\n"
+ " -b, --builtin=NAME@W add weighted buitin script among \"tpcb-like\"\n"
+ " \"simple-update\" and \"select-only\".\n"
" -c, --client=NUM number of concurrent database clients (default: 1)\n"
" -C, --connect establish new connection for each transaction\n"
" -D, --define=VARNAME=VALUE\n"
" define variable for use by custom script\n"
- " -f, --file=FILENAME read transaction script from FILENAME\n"
+ " -f, --file=FILENAME@W add weighted transaction script from FILENAME\n"
" -j, --jobs=NUM number of threads (default: 1)\n"
" -l, --log write transaction times to log file\n"
" -L, --latency-limit=NUM count transactions lasting more than NUM ms\n"
@@ -376,17 +415,18 @@ usage(void)
" -M, --protocol=simple|extended|prepared\n"
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
- " -N, --skip-some-updates skip updates of pgbench_tellers and pgbench_branches\n"
+ " -N, --skip-some-updates same as \"-b simple-update@1\"\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
" -r, --report-latencies report average latency per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
- " -S, --select-only perform SELECT-only transactions\n"
+ " -S, --select-only same as \"-b select-only@1\"\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
" --sampling-rate=NUM fraction of transactions to log (e.g. 0.01 for 1%%)\n"
+ " --per-script-stats report per-script statistics\n"
"\nCommon options:\n"
" -d, --debug print debugging output\n"
" -h, --host=HOSTNAME database server host or socket directory\n"
@@ -578,6 +618,49 @@ getPoissonRand(TState *thread, int64 center)
return (int64) (-log(uniform) * ((double) center) + 0.5);
}
+static void
+initSimpleStats(SimpleStats * ss)
+{
+ memset(ss, 0, sizeof(SimpleStats));
+}
+
+static void
+doSimpleStats(SimpleStats *ss, double val)
+{
+ if (ss->count == 0 || val < ss->min)
+ ss->min = val;
+ if (ss->count == 0 || val > ss->max)
+ ss->max = val;
+ ss->count ++;
+ ss->sum += val;
+ ss->sum2 += val * val;
+}
+
+static void
+appendSimpleStats(SimpleStats *acc, SimpleStats *ss)
+{
+ if (acc->count == 0 || ss->min < acc->min)
+ acc->min = ss->min;
+ if (acc->count == 0 || ss->max > acc->max)
+ acc->max = ss->max;
+ acc->count += ss->count;
+ acc->sum += ss->sum;
+ acc->sum2 += ss->sum2;
+}
+
+static void
+initStats(StatsData *sd, double start_time)
+{
+ sd->cnt = 0;
+ sd->skipped = 0;
+ initSimpleStats(& sd->latency);
+ initSimpleStats(& sd->lag);
+
+ /* not necessarily overriden? */
+ if (start_time)
+ sd->start_time = start_time;
+}
+
/* call PQexec() and exit() on failure */
static void
executeStatement(PGconn *con, const char *sql)
@@ -1097,50 +1180,41 @@ clientDone(CState *st, bool ok)
return false; /* always false */
}
-static void
-agg_vals_init(AggVals *aggs, instr_time start)
+static int
+chooseScript(TState *thread)
{
- /* basic counters */
- aggs->cnt = 0; /* number of transactions (includes skipped) */
- aggs->skipped = 0; /* xacts skipped under --rate --latency-limit */
+ int i = 0, w = 0, wc = (int) getrand(thread, 0, total_weight - 1);
- aggs->sum_latency = 0; /* SUM(latency) */
- aggs->sum2_latency = 0; /* SUM(latency*latency) */
+ do {
+ w += sql_script[i++].weight;
+ } while (w <= wc);
- /* min and max transaction duration */
- aggs->min_latency = 0;
- aggs->max_latency = 0;
-
- /* schedule lag counters */
- aggs->sum_lag = 0;
- aggs->sum2_lag = 0;
- aggs->min_lag = 0;
- aggs->max_lag = 0;
-
- /* start of the current interval */
- aggs->start_time = INSTR_TIME_GET_DOUBLE(start);
+ return i - 1;
}
/* return false iff client should be disconnected */
static bool
-doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, AggVals *agg)
+doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, StatsData *agg)
{
PGresult *res;
Command **commands;
bool trans_needs_throttle = false;
instr_time now;
+top:
/*
* gettimeofday() isn't free, so we get the current timestamp lazily the
* first time it's needed, and reuse the same value throughout this
* function after that. This also ensures that e.g. the calculated latency
* reported in the log file and in the totals are the same. Zero means
* "not set yet".
+ *
+ * on meta command we loop over and also reset now which may be used for
+ * per-command stats.
*/
INSTR_TIME_SET_ZERO(now);
-top:
- commands = sql_files[st->use_file];
+ commands = sql_script[st->use_file].commands;
/*
* Handle throttling once per transaction by sleeping. It is simpler to
@@ -1176,18 +1250,15 @@ top:
now_us = INSTR_TIME_GET_MICROSEC(now);
while (thread->throttle_trigger < now_us - latency_limit)
{
- thread->throttle_latency_skipped++;
-
- if (logfile)
- doLog(thread, st, logfile, &now, agg, true);
-
+ doTxStats(thread, st, &now, true, logfile, agg);
+ /* next rendez-vous */
wait = getPoissonRand(thread, throttle_delay);
thread->throttle_trigger += wait;
st->txn_scheduled = thread->throttle_trigger;
}
}
- st->sleeping = 1;
+ st->sleeping = true;
st->throttling = true;
st->is_throttled = true;
if (debug)
@@ -1197,27 +1268,13 @@ top:
if (st->sleeping)
{ /* are we sleeping? */
- int64 now_us;
-
if (INSTR_TIME_IS_ZERO(now))
INSTR_TIME_SET_CURRENT(now);
- now_us = INSTR_TIME_GET_MICROSEC(now);
- if (st->txn_scheduled <= now_us)
- {
- st->sleeping = 0; /* Done sleeping, go ahead with next command */
- if (st->throttling)
- {
- /* Measure lag of throttled transaction relative to target */
- int64 lag = now_us - st->txn_scheduled;
-
- thread->throttle_lag += lag;
- if (lag > thread->throttle_lag_max)
- thread->throttle_lag_max = lag;
- st->throttling = false;
- }
- }
- else
+ if (INSTR_TIME_GET_MICROSEC(now) < st->txn_scheduled)
return true; /* Still sleeping, nothing to do here */
+ /* Else done sleeping, go ahead with next command */
+ st->sleeping = false;
+ st->throttling = false;
}
if (st->listen)
@@ -1241,47 +1298,24 @@ top:
*/
if (is_latencies)
{
- int cnum = commands[st->state]->command_num;
-
if (INSTR_TIME_IS_ZERO(now))
INSTR_TIME_SET_CURRENT(now);
- INSTR_TIME_ACCUM_DIFF(thread->exec_elapsed[cnum],
- now, st->stmt_begin);
- thread->exec_count[cnum]++;
+
+ /* although a mutex would make sense, the likelyhood of an issue
+ * is small and these are only stats which may be slightly false
+ */
+ doSimpleStats(& commands[st->state]->stats,
+ INSTR_TIME_GET_DOUBLE(now) -
+ INSTR_TIME_GET_DOUBLE(st->stmt_begin));
}
/* transaction finished: calculate latency and log the transaction */
if (commands[st->state + 1] == NULL)
{
- /* only calculate latency if an option is used that needs it */
- if (progress || throttle_delay || latency_limit)
- {
- int64 latency;
-
- if (INSTR_TIME_IS_ZERO(now))
- INSTR_TIME_SET_CURRENT(now);
-
- latency = INSTR_TIME_GET_MICROSEC(now) - st->txn_scheduled;
-
- st->txn_latencies += latency;
-
- /*
- * XXX In a long benchmark run of high-latency transactions,
- * this int64 addition eventually overflows. For example, 100
- * threads running 10s transactions will overflow it in 2.56
- * hours. With a more-typical OLTP workload of .1s
- * transactions, overflow would take 256 hours.
- */
- st->txn_sqlats += latency * latency;
-
- /* record over the limit transactions if needed. */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
-
- /* record the time it took in the log */
- if (logfile)
- doLog(thread, st, logfile, &now, agg, false);
+ if (progress || throttle_delay || latency_limit || per_script_stats || logfile)
+ doTxStats(thread, st, &now, false, logfile, agg);
+ else
+ thread->stats.cnt ++;
}
if (commands[st->state]->type == SQL_COMMAND)
@@ -1324,8 +1358,9 @@ top:
if (commands[st->state] == NULL)
{
st->state = 0;
- st->use_file = (int) getrand(thread, 0, num_files - 1);
- commands = sql_files[st->use_file];
+ st->use_file = num_scripts==1? 0: chooseScript(thread);
+
+ commands = sql_script[st->use_file].commands;
st->is_throttled = false;
/*
@@ -1368,7 +1403,7 @@ top:
}
/* Record transaction start time under logging, progress or throttling */
- if ((logfile || progress || throttle_delay || latency_limit) && st->state == 0)
+ if ((logfile || progress || throttle_delay || latency_limit || per_script_stats) && st->state == 0)
{
INSTR_TIME_SET_CURRENT(st->txn_begin);
@@ -1656,7 +1691,7 @@ top:
INSTR_TIME_SET_CURRENT(now);
st->txn_scheduled = INSTR_TIME_GET_MICROSEC(now) + usec;
- st->sleeping = 1;
+ st->sleeping = true;
st->listen = 1;
}
@@ -1688,22 +1723,39 @@ top:
else /* succeeded */
st->listen = 1;
}
+
+ /* after a meta command, immediately proceed with next command */
goto top;
}
return true;
}
+static void
+doStats(StatsData *stats, bool skipped, double lat, double lag)
+{
+ stats->cnt ++;
+
+ if (skipped)
+ /* no latency to record on skipped transactions */
+ stats->skipped ++;
+ else
+ {
+ doSimpleStats(& stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ doSimpleStats(& stats->lag, lag);
+ }
+}
+
/*
* print log entry after completing one transaction.
*/
static void
-doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
- bool skipped)
+doLog(TState *thread, CState *st, FILE *logfile, instr_time *now,
+ StatsData *agg, bool skipped, double latency, double lag)
{
- double lag;
- double latency;
-
/*
* Skip the log entry if sampling is enabled and this row doesn't belong
* to the random sample.
@@ -1712,15 +1764,6 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
pg_erand48(thread->random_state) > sample_rate)
return;
- if (INSTR_TIME_IS_ZERO(*now))
- INSTR_TIME_SET_CURRENT(*now);
-
- latency = (double) (INSTR_TIME_GET_MICROSEC(*now) - st->txn_scheduled);
- if (skipped)
- lag = latency;
- else
- lag = (double) (INSTR_TIME_GET_MICROSEC(st->txn_begin) - st->txn_scheduled);
-
/* should we aggregate the results or not? */
if (agg_interval > 0)
{
@@ -1730,39 +1773,7 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
*/
if (agg->start_time + agg_interval >= INSTR_TIME_GET_DOUBLE(*now))
{
- agg->cnt += 1;
- if (skipped)
- {
- /*
- * there is no latency to record if the transaction was
- * skipped
- */
- agg->skipped += 1;
- }
- else
- {
- agg->sum_latency += latency;
- agg->sum2_latency += latency * latency;
-
- /* first in this aggregation interval */
- if ((agg->cnt == 1) || (latency < agg->min_latency))
- agg->min_latency = latency;
-
- if ((agg->cnt == 1) || (latency > agg->max_latency))
- agg->max_latency = latency;
-
- /* and the same for schedule lag */
- if (throttle_delay)
- {
- agg->sum_lag += lag;
- agg->sum2_lag += lag * lag;
-
- if ((agg->cnt == 1) || (lag < agg->min_lag))
- agg->min_lag = lag;
- if ((agg->cnt == 1) || (lag > agg->max_lag))
- agg->max_lag = lag;
- }
- }
+ doStats(agg, skipped, latency, lag);
}
else
{
@@ -1777,52 +1788,34 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
* usage), so we don't need to handle this in a special way
* (see below).
*/
- fprintf(logfile, "%ld %d %.0f %.0f %.0f %.0f",
+ fprintf(logfile, "%ld " INT64_FORMAT " %.0f %.0f %.0f %.0f",
agg->start_time,
agg->cnt,
- agg->sum_latency,
- agg->sum2_latency,
- agg->min_latency,
- agg->max_latency);
+ agg->latency.sum,
+ agg->latency.sum2,
+ agg->latency.min,
+ agg->latency.max);
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
- agg->sum_lag,
- agg->sum2_lag,
- agg->min_lag,
- agg->max_lag);
+ agg->lag.sum,
+ agg->lag.sum2,
+ agg->lag.min,
+ agg->lag.max);
if (latency_limit)
- fprintf(logfile, " %d", agg->skipped);
+ fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
fputc('\n', logfile);
- /* move to the next inteval */
- agg->start_time = agg->start_time + agg_interval;
+ /* move to the next interval */
+ agg->start_time += agg_interval;
/* reset for "no transaction" intervals */
- agg->cnt = 0;
- agg->skipped = 0;
- agg->min_latency = 0;
- agg->max_latency = 0;
- agg->sum_latency = 0;
- agg->sum2_latency = 0;
- agg->min_lag = 0;
- agg->max_lag = 0;
- agg->sum_lag = 0;
- agg->sum2_lag = 0;
+ initStats(agg, 0.0);
}
/* reset the values to include only the current transaction. */
- agg->cnt = 1;
- agg->skipped = skipped ? 1 : 0;
- agg->min_latency = latency;
- agg->max_latency = latency;
- agg->sum_latency = skipped ? 0.0 : latency;
- agg->sum2_latency = skipped ? 0.0 : latency * latency;
- agg->min_lag = lag;
- agg->max_lag = lag;
- agg->sum_lag = lag;
- agg->sum2_lag = lag * lag;
+ doStats(agg, skipped, latency, lag);
}
}
else
@@ -1832,21 +1825,21 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
/* This is more than we really ought to know about instr_time */
if (skipped)
- fprintf(logfile, "%d %d skipped %d %ld %ld",
+ fprintf(logfile, "%d " INT64_FORMAT " skipped %d %ld %ld",
st->id, st->cnt, st->use_file,
(long) now->tv_sec, (long) now->tv_usec);
else
- fprintf(logfile, "%d %d %.0f %d %ld %ld",
+ fprintf(logfile, "%d " INT64_FORMAT " %.0f %d %ld %ld",
st->id, st->cnt, latency, st->use_file,
(long) now->tv_sec, (long) now->tv_usec);
#else
/* On Windows, instr_time doesn't provide a timestamp anyway */
if (skipped)
- fprintf(logfile, "%d %d skipped %d 0 0",
+ fprintf(logfile, "%d "INT64_FORMAT" skipped %d 0 0",
st->id, st->cnt, st->use_file);
else
- fprintf(logfile, "%d %d %.0f %d 0 0",
+ fprintf(logfile, "%d "INT64_FORMAT" %.0f %d 0 0",
st->id, st->cnt, latency, st->use_file);
#endif
if (throttle_delay)
@@ -1855,6 +1848,44 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
}
}
+/*
+ * end of transaction statistics
+ */
+static void
+doTxStats(TState *thread, CState *st, instr_time *now,
+ bool skipped, FILE *logfile, StatsData *agg)
+{
+ double latency = 0.0, lag = 0.0;
+
+ if ((!skipped || agg_interval) && INSTR_TIME_IS_ZERO(*now))
+ INSTR_TIME_SET_CURRENT(*now);
+
+ if (!skipped)
+ {
+ /* compute latency & lag if needed */
+ latency = INSTR_TIME_GET_MICROSEC(*now) - st->txn_scheduled;
+ lag = INSTR_TIME_GET_MICROSEC(st->txn_begin) - st->txn_scheduled;
+ }
+
+ if (progress || throttle_delay || latency_limit)
+ {
+ doStats(& thread->stats, skipped, latency, lag);
+
+ /* record over the limit transactions if needed. */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
+ }
+ else
+ thread->stats.cnt ++;
+
+ if (use_log)
+ doLog(thread, st, logfile, now, agg, skipped, latency, lag);
+
+ if (per_script_stats) /* mutex? hmmm... these are only statistics */
+ doStats(& sql_script[st->use_file].stats, skipped, latency, lag);
+}
+
+
/* discard connections */
static void
disconnect_all(CState *state, int length)
@@ -2251,6 +2282,7 @@ process_commands(char *buf, const char *source, const int lineno)
my_commands->command_num = num_commands++;
my_commands->type = 0; /* until set */
my_commands->argc = 0;
+ initSimpleStats(& my_commands->stats);
if (*p == '\\')
{
@@ -2475,7 +2507,7 @@ read_line_from_file(FILE *fd)
return NULL;
}
-static int
+static Command **
process_file(char *filename)
{
#define COMMANDS_ALLOC_NUM 128
@@ -2483,15 +2515,9 @@ process_file(char *filename)
Command **my_commands;
FILE *fd;
int lineno,
- index;
+ index,
+ alloc_num;
char *buf;
- int alloc_num;
-
- if (num_files >= MAX_FILES)
- {
- fprintf(stderr, "at most %d SQL files are allowed\n", MAX_FILES);
- exit(1);
- }
alloc_num = COMMANDS_ALLOC_NUM;
my_commands = (Command **) pg_malloc(sizeof(Command *) * alloc_num);
@@ -2503,7 +2529,7 @@ process_file(char *filename)
fprintf(stderr, "could not open file \"%s\": %s\n",
filename, strerror(errno));
pg_free(my_commands);
- return false;
+ return NULL;
}
lineno = 0;
@@ -2535,13 +2561,11 @@ process_file(char *filename)
my_commands[index] = NULL;
- sql_files[num_files++] = my_commands;
-
- return true;
+ return my_commands;
}
static Command **
-process_builtin(char *tb, const char *source)
+process_builtin(const char *tb, const char *source)
{
#define COMMANDS_ALLOC_NUM 128
@@ -2595,35 +2619,77 @@ process_builtin(char *tb, const char *source)
return my_commands;
}
+/* Possiby truncate option and return weight */
+static int
+getWeight(char *option)
+{
+ char *sep;
+ int weight;
+
+ if ((sep = strrchr(option, WSEP)))
+ {
+ *sep++ = '\0';
+ weight = atoi(sep);
+ if (weight <= 0)
+ {
+ fprintf(stderr,
+ "weight must be strictly positive, got \"%s\"\n", sep);
+ exit(1);
+ }
+ }
+ else
+ weight = 1;
+ return weight;
+}
+
+static void
+addScript(const char *name, Command ** commands, int weight)
+{
+ if (commands == NULL)
+ {
+ fprintf(stderr, "empty commands for %s\n", name);
+ exit(1);
+ }
+
+ if (num_scripts >= MAX_SCRIPTS)
+ {
+ fprintf(stderr, "at most %d SQL scripts are allowed\n", MAX_SCRIPTS);
+ exit(1);
+ }
+
+ sql_script[num_scripts].name = name;
+ sql_script[num_scripts].weight = weight;
+ sql_script[num_scripts].commands = commands;
+ initStats(& sql_script[num_scripts].stats, 0.0);
+ num_scripts++;
+}
+
+static void
+printSimpleStats(char *prefix, SimpleStats *ss)
+{
+ /* print NaN if no transactions where executed */
+ double latency = ss->sum / ss->count;
+ double stddev = sqrt(ss->sum2 / ss->count - latency*latency);
+ printf("%s average = %.3f ms\n", prefix, 0.001 * latency);
+ printf("%s stddev = %.3f ms\n", prefix, 0.001 * stddev);
+}
+
/* print out results */
static void
-printResults(int ttype, int64 normal_xacts, int nclients,
- TState *threads, int nthreads,
- instr_time total_time, instr_time conn_total_time,
- int64 total_latencies, int64 total_sqlats,
- int64 throttle_lag, int64 throttle_lag_max,
- int64 throttle_latency_skipped, int64 latency_late)
+printResults(TState *threads, StatsData *total, instr_time total_time,
+ instr_time conn_total_time, int latency_late)
{
double time_include,
tps_include,
tps_exclude;
- char *s;
time_include = INSTR_TIME_GET_DOUBLE(total_time);
- tps_include = normal_xacts / time_include;
- tps_exclude = normal_xacts / (time_include -
+ tps_include = total->cnt / time_include;
+ tps_exclude = total->cnt / (time_include -
(INSTR_TIME_GET_DOUBLE(conn_total_time) / nthreads));
- if (ttype == 0)
- s = "TPC-B (sort of)";
- else if (ttype == 2)
- s = "Update only pgbench_accounts";
- else if (ttype == 1)
- s = "SELECT only";
- else
- s = "Custom query";
-
- printf("transaction type: %s\n", s);
+ printf("transaction type: %s\n",
+ num_scripts == 1? sql_script[0].name: "multiple scripts");
printf("scaling factor: %d\n", scale);
printf("query mode: %s\n", QUERYMODE[querymode]);
printf("number of clients: %d\n", nclients);
@@ -2631,49 +2697,38 @@ printResults(int ttype, int64 normal_xacts, int nclients,
if (duration <= 0)
{
printf("number of transactions per client: %d\n", nxacts);
- printf("number of transactions actually processed: " INT64_FORMAT "/" INT64_FORMAT "\n",
- normal_xacts, (int64) nxacts * nclients);
+ printf("number of transactions actually processed: "INT64_FORMAT"/%d\n",
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
- printf("number of transactions actually processed: " INT64_FORMAT "\n",
- normal_xacts);
+ printf("number of transactions actually processed: "INT64_FORMAT"\n",
+ total->cnt);
}
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (normal_xacts <= 0)
+ if (total->cnt <= 0)
return;
if (throttle_delay && latency_limit)
- printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- throttle_latency_skipped,
- 100.0 * throttle_latency_skipped / (throttle_latency_skipped + normal_xacts));
+ printf("number of transactions skipped: "INT64_FORMAT" (%.3f %%)\n",
+ total->skipped,
+ 100.0 * total->skipped / (total->skipped + total->cnt));
if (latency_limit)
- printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT " (%.3f %%)\n",
+ printf("number of transactions above the %.1f ms latency limit: %d (%.3f %%)\n",
latency_limit / 1000.0, latency_late,
- 100.0 * latency_late / (throttle_latency_skipped + normal_xacts));
+ 100.0 * latency_late / (total->skipped + total->cnt));
if (throttle_delay || progress || latency_limit)
- {
- /* compute and show latency average and standard deviation */
- double latency = 0.001 * total_latencies / normal_xacts;
- double sqlat = (double) total_sqlats / normal_xacts;
-
- printf("latency average: %.3f ms\n"
- "latency stddev: %.3f ms\n",
- latency, 0.001 * sqrt(sqlat - 1000000.0 * latency * latency));
- }
+ printSimpleStats("latency", & total->latency);
else
- {
/* only an average latency computed from the duration is available */
printf("latency average: %.3f ms\n",
- 1000.0 * duration * nclients / normal_xacts);
- }
+ 1000.0 * duration * nclients / total->cnt);
if (throttle_delay)
- {
/*
* Report average transaction lag under rate limit throttling. This
* is the delay between scheduled and actual start times for the
@@ -2681,53 +2736,44 @@ printResults(int ttype, int64 normal_xacts, int nclients,
* the database load, or the Poisson throttling process.
*/
printf("rate limit schedule lag: avg %.3f (max %.3f) ms\n",
- 0.001 * throttle_lag / normal_xacts, 0.001 * throttle_lag_max);
- }
+ 0.001 * total->lag.sum / total->cnt, 0.001 * total->lag.max);
printf("tps = %f (including connections establishing)\n", tps_include);
printf("tps = %f (excluding connections establishing)\n", tps_exclude);
- /* Report per-command latencies */
- if (is_latencies)
+ /* Report per-file data */
+ if (per_script_stats)
{
- int i;
+ int i;
- for (i = 0; i < num_files; i++)
+ for (i = 0; i < num_scripts; i++)
{
- Command **commands;
+ printf("SQL script %d, weight %d: %s\n"
+ " - "INT64_FORMAT" transactions (%.1f%% of total, tps = %f)\n",
+ i, sql_script[i].weight, sql_script[i].name,
+ sql_script[i].stats.cnt,
+ 100.0 * sql_script[i].stats.cnt / total->cnt,
+ sql_script[i].stats.cnt / time_include);
- if (num_files > 1)
- printf("statement latencies in milliseconds, file %d:\n", i + 1);
- else
- printf("statement latencies in milliseconds:\n");
+ if (latency_limit)
+ printf(" - number of transactions skipped: "INT64_FORMAT" (%.3f%%)\n",
+ sql_script[i].stats.skipped,
+ 100.0 * sql_script[i].stats.skipped /
+ (sql_script[i].stats.skipped + sql_script[i].stats.cnt));
- for (commands = sql_files[i]; *commands != NULL; commands++)
- {
- Command *command = *commands;
- int cnum = command->command_num;
- double total_time;
- instr_time total_exec_elapsed;
- int total_exec_count;
- int t;
-
- /* Accumulate per-thread data for command */
- INSTR_TIME_SET_ZERO(total_exec_elapsed);
- total_exec_count = 0;
- for (t = 0; t < nthreads; t++)
- {
- TState *thread = &threads[t];
+ printSimpleStats(" - latency", & sql_script[i].stats.latency);
- INSTR_TIME_ADD(total_exec_elapsed,
- thread->exec_elapsed[cnum]);
- total_exec_count += thread->exec_count[cnum];
- }
+ /* Report per-command latencies */
+ if (is_latencies)
+ {
+ Command ** com;
- if (total_exec_count > 0)
- total_time = INSTR_TIME_GET_MILLISEC(total_exec_elapsed) / (double) total_exec_count;
- else
- total_time = 0.0;
+ printf(" - per command latencies in ms:\n");
- printf("\t%f\t%s\n", total_time, command->line);
+ for (com = sql_script[i].commands; *com != NULL; com++)
+ printf(" %11.3f %s\n",
+ 1000.0 * (*com)->stats.sum / (*com)->stats.count,
+ (*com)->line);
}
}
}
@@ -2739,6 +2785,7 @@ main(int argc, char **argv)
{
static struct option long_options[] = {
/* systematic long/short named options */
+ {"tpc-b", no_argument, NULL, 'B'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
{"debug", no_argument, NULL, 'd'},
@@ -2749,12 +2796,14 @@ main(int argc, char **argv)
{"initialize", no_argument, NULL, 'i'},
{"jobs", required_argument, NULL, 'j'},
{"log", no_argument, NULL, 'l'},
+ {"latency-limit", required_argument, NULL, 'L'},
{"no-vacuum", no_argument, NULL, 'n'},
{"port", required_argument, NULL, 'p'},
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
{"report-latencies", no_argument, NULL, 'r'},
+ {"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
{"skip-some-updates", no_argument, NULL, 'N'},
@@ -2769,25 +2818,20 @@ main(int argc, char **argv)
{"unlogged-tables", no_argument, &unlogged_tables, 1},
{"sampling-rate", required_argument, NULL, 4},
{"aggregate-interval", required_argument, NULL, 5},
- {"rate", required_argument, NULL, 'R'},
- {"latency-limit", required_argument, NULL, 'L'},
+ {"per-script-stats", no_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
int c;
- int nclients = 1; /* default number of simulated clients */
- int nthreads = 1; /* default number of threads */
int is_init_mode = 0; /* initialize mode? */
int is_no_vacuum = 0; /* no vacuum at all before testing? */
int do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
- int ttype = 0; /* transaction type. 0: TPC-B, 1: SELECT only,
- * 2: skip update of branches and tellers */
int optindex;
- char *filename = NULL;
bool scale_given = false;
bool benchmarking_option_set = false;
bool initialization_option_set = false;
+ bool internal_script_used = false;
CState *state; /* status of clients */
TState *threads; /* array of thread */
@@ -2795,13 +2839,10 @@ main(int argc, char **argv)
instr_time start_time; /* start up time */
instr_time total_time;
instr_time conn_total_time;
- int64 total_xacts = 0;
- int64 total_latencies = 0;
- int64 total_sqlats = 0;
- int64 throttle_lag = 0;
- int64 throttle_lag_max = 0;
- int64 throttle_latency_skipped = 0;
int64 latency_late = 0;
+ StatsData stats;
+ int weight;
+ char *desc;
int i;
int nclients_dealt;
@@ -2847,7 +2888,7 @@ main(int argc, char **argv)
state = (CState *) pg_malloc(sizeof(CState));
memset(state, 0, sizeof(CState));
- while ((c = getopt_long(argc, argv, "ih:nvp:dqSNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -2869,14 +2910,6 @@ main(int argc, char **argv)
case 'd':
debug++;
break;
- case 'S':
- ttype = 1;
- benchmarking_option_set = true;
- break;
- case 'N':
- ttype = 2;
- benchmarking_option_set = true;
- break;
case 'c':
benchmarking_option_set = true;
nclients = atoi(optarg);
@@ -2928,6 +2961,7 @@ main(int argc, char **argv)
break;
case 'r':
benchmarking_option_set = true;
+ per_script_stats = true;
is_latencies = true;
break;
case 's':
@@ -2979,12 +3013,30 @@ main(int argc, char **argv)
initialization_option_set = true;
use_quiet = true;
break;
+ /* what to run*/
+ case 'b':
+ weight = getWeight(optarg);
+ addScript(desc, process_builtin(
+ find_builtin(optarg, &desc), desc), weight);
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ break;
+ case 'S':
+ addScript(desc, process_builtin(
+ find_builtin("select-only", &desc), desc), 1);
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ break;
+ case 'N':
+ addScript(desc, process_builtin(
+ find_builtin("simple-update", &desc), desc), 1);
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ break;
case 'f':
+ weight = getWeight(optarg);
+ addScript(optarg, process_file(optarg), weight);
benchmarking_option_set = true;
- ttype = 3;
- filename = pg_strdup(optarg);
- if (process_file(filename) == false || *sql_files[num_files - 1] == NULL)
- exit(1);
break;
case 'D':
{
@@ -3015,9 +3067,9 @@ main(int argc, char **argv)
break;
case 'M':
benchmarking_option_set = true;
- if (num_files > 0)
+ if (num_scripts > 0)
{
- fprintf(stderr, "query mode (-M) should be specified before any transaction scripts (-f)\n");
+ fprintf(stderr, "query mode (-M) should be specified before any transaction scripts (-f or -b)\n");
exit(1);
}
for (querymode = 0; querymode < NUM_QUERYMODE; querymode++)
@@ -3107,6 +3159,9 @@ main(int argc, char **argv)
}
#endif
break;
+ case 6:
+ per_script_stats = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -3114,6 +3169,19 @@ main(int argc, char **argv)
}
}
+ /* set default script if none */
+ if (num_scripts == 0 && !initialization_option_set)
+ {
+ addScript(desc, process_builtin(
+ find_builtin("tpcb-like", &desc), desc), 1);
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ }
+
+ /* compute total_weight */
+ for (i = 0; i < num_scripts; i++)
+ total_weight += sql_script[i].weight;
+
/*
* Don't need more threads than there are clients. (This is not merely an
* optimization; throttle_delay is calculated incorrectly below if some
@@ -3198,8 +3266,6 @@ main(int argc, char **argv)
* changed after fork.
*/
main_pid = (int) getpid();
- progress_nclients = nclients;
- progress_nthreads = nthreads;
if (nclients > 1)
{
@@ -3242,7 +3308,7 @@ main(int argc, char **argv)
exit(1);
}
- if (ttype != 3)
+ if (internal_script_used)
{
/*
* get the scaling factor that should be same as count(*) from
@@ -3319,31 +3385,6 @@ main(int argc, char **argv)
INSTR_TIME_SET_CURRENT(start_time);
srandom((unsigned int) INSTR_TIME_GET_MICROSEC(start_time));
- /* process builtin SQL scripts */
- switch (ttype)
- {
- case 0:
- sql_files[0] = process_builtin(tpc_b,
- "<builtin: TPC-B (sort of)>");
- num_files = 1;
- break;
-
- case 1:
- sql_files[0] = process_builtin(select_only,
- "<builtin: select only>");
- num_files = 1;
- break;
-
- case 2:
- sql_files[0] = process_builtin(simple_update,
- "<builtin: simple update>");
- num_files = 1;
- break;
-
- default:
- break;
- }
-
/* set up thread data structures */
threads = (TState *) pg_malloc(sizeof(TState) * nthreads);
nclients_dealt = 0;
@@ -3359,32 +3400,10 @@ main(int argc, char **argv)
thread->random_state[0] = random();
thread->random_state[1] = random();
thread->random_state[2] = random();
- thread->throttle_latency_skipped = 0;
thread->latency_late = 0;
+ initStats(& thread->stats, 0.0);
nclients_dealt += thread->nstate;
-
- if (is_latencies)
- {
- /* Reserve memory for the thread to store per-command latencies */
- int t;
-
- thread->exec_elapsed = (instr_time *)
- pg_malloc(sizeof(instr_time) * num_commands);
- thread->exec_count = (int *)
- pg_malloc(sizeof(int) * num_commands);
-
- for (t = 0; t < num_commands; t++)
- {
- INSTR_TIME_SET_ZERO(thread->exec_elapsed[t]);
- thread->exec_count[t] = 0;
- }
- }
- else
- {
- thread->exec_elapsed = NULL;
- thread->exec_count = NULL;
- }
}
/* all clients must be assigned to a thread */
@@ -3427,11 +3446,11 @@ main(int argc, char **argv)
#endif /* ENABLE_THREAD_SAFETY */
/* wait for threads and accumulate results */
+ initStats(& stats, 0.0);
INSTR_TIME_SET_ZERO(conn_total_time);
for (i = 0; i < nthreads; i++)
{
TState *thread = &threads[i];
- int j;
#ifdef ENABLE_THREAD_SAFETY
if (threads[i].thread == INVALID_THREAD)
@@ -3444,21 +3463,13 @@ main(int argc, char **argv)
(void) threadRun(thread);
#endif /* ENABLE_THREAD_SAFETY */
- /* thread level stats */
- throttle_lag += thread->throttle_lag;
- throttle_latency_skipped = threads->throttle_latency_skipped;
- latency_late = thread->latency_late;
- if (throttle_lag_max > thread->throttle_lag_max)
- throttle_lag_max = thread->throttle_lag_max;
+ /* aggregate thread level stats */
+ appendSimpleStats(& stats.latency, & thread->stats.latency);
+ appendSimpleStats(& stats.lag, & thread->stats.lag);
+ stats.cnt += thread->stats.cnt;
+ stats.skipped += thread->stats.skipped;
+ latency_late += thread->latency_late;
INSTR_TIME_ADD(conn_total_time, thread->conn_time);
-
- /* client-level stats */
- for (j = 0; j < thread->nstate; j++)
- {
- total_xacts += thread->state[j].cnt;
- total_latencies += thread->state[i].txn_latencies;
- total_sqlats += thread->state[i].txn_sqlats;
- }
}
disconnect_all(state, nclients);
@@ -3474,10 +3485,7 @@ main(int argc, char **argv)
*/
INSTR_TIME_SET_CURRENT(total_time);
INSTR_TIME_SUBTRACT(total_time, start_time);
- printResults(ttype, total_xacts, nclients, threads, nthreads,
- total_time, conn_total_time, total_latencies, total_sqlats,
- throttle_lag, throttle_lag_max, throttle_latency_skipped,
- latency_late);
+ printResults(threads, &stats, total_time, conn_total_time, latency_late);
return 0;
}
@@ -3498,13 +3506,7 @@ threadRun(void *arg)
int64 thread_start = INSTR_TIME_GET_MICROSEC(thread->start_time);
int64 last_report = thread_start;
int64 next_report = last_report + (int64) progress * 1000000;
- int64 last_count = 0,
- last_lats = 0,
- last_sqlats = 0,
- last_lags = 0,
- last_skipped = 0;
-
- AggVals aggs;
+ StatsData last, aggs;
/*
* Initialize throttling rate target for all of the thread's clients. It
@@ -3514,8 +3516,6 @@ threadRun(void *arg)
*/
INSTR_TIME_SET_CURRENT(start);
thread->throttle_trigger = INSTR_TIME_GET_MICROSEC(start);
- thread->throttle_lag = 0;
- thread->throttle_lag_max = 0;
INSTR_TIME_SET_ZERO(thread->conn_time);
@@ -3552,16 +3552,17 @@ threadRun(void *arg)
INSTR_TIME_SET_CURRENT(thread->conn_time);
INSTR_TIME_SUBTRACT(thread->conn_time, thread->start_time);
- agg_vals_init(&aggs, thread->start_time);
+ initStats(&aggs, INSTR_TIME_GET_DOUBLE(thread->start_time));
+ last = aggs;
/* send start up queries in async manner */
for (i = 0; i < nstate; i++)
{
CState *st = &state[i];
- Command **commands = sql_files[st->use_file];
+ Command **commands = sql_script[st->use_file].commands;
int prev_ecnt = st->ecnt;
- st->use_file = getrand(thread, 0, num_files - 1);
+ st->use_file = num_scripts==1? 0: chooseScript(thread);
if (!doCustom(thread, st, &thread->conn_time, logfile, &aggs))
remains--; /* I've aborted */
@@ -3589,7 +3590,7 @@ threadRun(void *arg)
for (i = 0; i < nstate; i++)
{
CState *st = &state[i];
- Command **commands = sql_files[st->use_file];
+ Command **commands = sql_script[st->use_file].commands;
int sock;
if (st->con == NULL)
@@ -3602,7 +3603,7 @@ threadRun(void *arg)
{
/* interrupt client which has not started a transaction */
remains--;
- st->sleeping = 0;
+ st->sleeping = false;
st->throttling = false;
PQfinish(st->con);
st->con = NULL;
@@ -3695,7 +3696,7 @@ threadRun(void *arg)
for (i = 0; i < nstate; i++)
{
CState *st = &state[i];
- Command **commands = sql_files[st->use_file];
+ Command **commands = sql_script[st->use_file].commands;
int prev_ecnt = st->ecnt;
if (st->con && (FD_ISSET(PQsocket(st->con), &input_mask)
@@ -3726,11 +3727,7 @@ threadRun(void *arg)
if (now >= next_report)
{
/* generate and show report */
- int64 count = 0,
- lats = 0,
- sqlats = 0,
- lags = 0,
- skipped = 0;
+ StatsData cur;
int64 run = now - last_report;
double tps,
total_run,
@@ -3750,23 +3747,24 @@ threadRun(void *arg)
* (If a read from a 64-bit integer is not atomic, you might
* get a "torn" read and completely bogus latencies though!)
*/
- for (i = 0; i < progress_nclients; i++)
+ initStats(& cur, 0.0);
+ for (i = 0; i < nthreads; i++)
{
- count += state[i].cnt;
- lats += state[i].txn_latencies;
- sqlats += state[i].txn_sqlats;
+ appendSimpleStats(& cur.latency, & thread[i].stats.latency);
+ appendSimpleStats(& cur.lag, & thread[i].stats.lag);
+ cur.cnt += thread[i].stats.cnt;
+ cur.skipped += thread[i].stats.skipped;
}
- for (i = 0; i < progress_nthreads; i++)
- lags += thread[i].throttle_lag;
-
total_run = (now - thread_start) / 1000000.0;
- tps = 1000000.0 * (count - last_count) / run;
- latency = 0.001 * (lats - last_lats) / (count - last_count);
- sqlat = 1.0 * (sqlats - last_sqlats) / (count - last_count);
+ tps = 1000000.0 * (cur.cnt - last.cnt) / run;
+ latency = 0.001 * (cur.latency.sum - last.latency.sum) /
+ (cur.cnt - last.cnt);
+ sqlat = 1.0 * (cur.latency.sum2 - last.latency.sum2)
+ / (cur.cnt - last.cnt);
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (lags - last_lags) / (count - last_count);
- skipped = thread->throttle_latency_skipped - last_skipped;
+ lag = 0.001 * (cur.lag.sum - last.lag.sum) /
+ (cur.cnt - last.cnt);
fprintf(stderr,
"progress: %.1f s, %.1f tps, "
@@ -3776,16 +3774,13 @@ threadRun(void *arg)
{
fprintf(stderr, ", lag %.3f ms", lag);
if (latency_limit)
- fprintf(stderr, ", " INT64_FORMAT " skipped", skipped);
+ fprintf(stderr, ", "INT64_FORMAT" skipped",
+ cur.skipped - last.skipped);
}
fprintf(stderr, "\n");
- last_count = count;
- last_lats = lats;
- last_sqlats = sqlats;
- last_lags = lags;
+ last = cur;
last_report = now;
- last_skipped = thread->throttle_latency_skipped;
/*
* Ensure that the next report is in the future, in case
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers