On 5.9.2012 06:17, Robert Haas wrote:
> On Tue, Sep 4, 2012 at 11:31 PM, Peter Eisentraut <[email protected]> wrote:
>> On Tue, 2012-09-04 at 23:14 -0400, Robert Haas wrote:
>>> Actually, this whole things seems like a solution in search of a
>>> problem to me. We just reduced the verbosity of pgbench -i tenfold in
>>> the very recent past - I would have thought that enough to address
>>> this problem. But maybe not.
>>
>> The problem is that
>>
>> a) It blasts out too much output and everything scrolls off the screen,
>> and
>>
>> b) There is no indication of where the end is.
>>
>> These are independent problems, and I'd be happy to address them
>> separately if there are such specific concerns attached to this.
>>
>> Speaking of tenfold, we could reduce the output frequency tenfold to
>> once every 1000000, which would alleviate this problem for a while
>> longer.
>
> Well, I wouldn't object to displaying a percentage on each output
> line. But I don't really like the idea of having them less frequent
> than they already are, because if you run into a situation that makes
> pgbench -i run slowly, as I occasionally do, it's marginal to tell the
> difference between "slow" and "completely hung" even with the current
> level of verbosity.
>
> However, we could add a -q flag to run more quietly, or something like
> that. Actually, I'd even be fine with making the default quieter,
> though we can't use -v for verbose since that's already taken. But
> I'd like to preserve the option of getting the current amount of
> output because sometimes I need that to troubleshoot problems.
> Actually it'd be nice to even get a bit more output: say, a timestamp
> on each line, and a completion percentage... but now I'm getting
> greedy.
Hi,
I've been thinking about this a bit more, and do propose to use an
option that determines "logging step" i.e. number of items (either
directly or as a percentage) between log lines.
The attached patch defines a new option "--logging-step" that accepts
either integers or percents. For example if you want to print a line
each 1000 lines, you can to this
$ pgbench -i -s 1000 --logging-step 1000 testdb
and if you want to print a line each 5%, you can do this
$ pgbench -i -s 1000 --logging-step 5% testdb
and that's it.
Moreover the patch adds a record of elapsed an estimate of remaining
time. So for example with 21% you may get this:
creating tables...
21000 of 100000 tuples (21%) done (elapsed 1.56 s, remaining 5.85 s).
42000 of 100000 tuples (42%) done (elapsed 3.15 s, remaining 4.35 s).
63000 of 100000 tuples (63%) done (elapsed 4.73 s, remaining 2.78 s).
84000 of 100000 tuples (84%) done (elapsed 6.30 s, remaining 1.20 s).
100000 of 100000 tuples (100%) done (elapsed 8.17 s, remaining 0.00 s).
vacuum...
set primary keys...
Now, I've had a hard time with the patch - no matter what I do, I do get
"invalid option" error whenever I try to run that from command line for
some reason. But when I run it from gdb, it works just fine.
kind regards
Tomas
diff --git a/contrib/pgbench/pgbench.c b/contrib/pgbench/pgbench.c
index f5ac3b1..ce7e240 100644
--- a/contrib/pgbench/pgbench.c
+++ b/contrib/pgbench/pgbench.c
@@ -130,6 +130,11 @@ int foreign_keys = 0;
int unlogged_tables = 0;
/*
+ * logging step (inserts)
+ */
+int log_step = 100000;
+
+/*
* tablespace selection
*/
char *tablespace = NULL;
@@ -356,6 +361,8 @@ usage(void)
" create tables in the specified tablespace\n"
" --unlogged-tables\n"
" create tables as unlogged tables\n"
+ " --logging-step NUM\n"
+ " how often to print info about init
progress\n"
"\nBenchmarking options:\n"
" -c NUM number of concurrent database clients (default:
1)\n"
" -C establish new connection for each
transaction\n"
@@ -1340,6 +1347,10 @@ init(bool is_no_vacuum)
char sql[256];
int i;
+ /* used to track elapsed time and estimate of the remaining time */
+ instr_time start, diff;
+ double elapsed_sec, remaining_sec;
+
if ((con = doConnect()) == NULL)
exit(1);
@@ -1408,6 +1419,8 @@ init(bool is_no_vacuum)
}
PQclear(res);
+ INSTR_TIME_SET_CURRENT(start);
+
for (i = 0; i < naccounts * scale; i++)
{
int j = i + 1;
@@ -1419,10 +1432,18 @@ init(bool is_no_vacuum)
exit(1);
}
- if (j % 100000 == 0)
- fprintf(stderr, "%d of %d tuples (%d%%) done.\n",
+ if (j % log_step == 0 || j == scale * naccounts)
+ {
+ INSTR_TIME_SET_CURRENT(diff);
+ INSTR_TIME_SUBTRACT(diff, start);
+
+ elapsed_sec = INSTR_TIME_GET_DOUBLE(diff);
+ remaining_sec = (scale * naccounts - j) * elapsed_sec /
j;
+
+ fprintf(stderr, "%d of %d tuples (%d%%) done (elapsed
%.2f s, remaining %.2f s).\n",
j, naccounts * scale,
- j * 100 / (naccounts * scale));
+ j * 100 / (naccounts * scale),
elapsed_sec, remaining_sec);
+ }
}
if (PQputline(con, "\\.\n"))
{
@@ -1901,6 +1922,7 @@ main(int argc, char **argv)
int do_vacuum_accounts = 0; /* do vacuum accounts
before testing? */
int ttype = 0; /* transaction type. 0:
TPC-B, 1: SELECT only,
* 2: skip
update of branches and tellers */
+ float log_step_pct = 0; /* logging step in percent */
int optindex;
char *filename = NULL;
bool scale_given = false;
@@ -1920,6 +1942,7 @@ main(int argc, char **argv)
{"index-tablespace", required_argument, NULL, 3},
{"tablespace", required_argument, NULL, 2},
{"unlogged-tables", no_argument, &unlogged_tables, 1},
+ {"logging-step", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -2125,6 +2148,14 @@ main(int argc, char **argv)
case 3: /* index-tablespace */
index_tablespace = optarg;
break;
+ case 6:
+ if (optarg[strlen(optarg)-1] == '%') {
+ optarg[strlen(optarg)-1] = '\0';
+ log_step_pct = atof(optarg);
+ } else {
+ log_step = atol(optarg);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more
information.\n"), progname);
exit(1);
@@ -2144,6 +2175,11 @@ main(int argc, char **argv)
dbName = "";
}
+ /* compute the log_step from total number of accounts and log_step_pct
*/
+ if (log_step_pct != 0) {
+ log_step = log_step_pct * naccounts * scale / 100;
+ }
+
if (is_init_mode)
{
init(is_no_vacuum);
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
