This patch makes fio --client JSON output work better when many server threads
are used by removing interleaved output from fio servers with
--output-format=json .
Background: Red Hat is testing large numbers of OpenStack guests with Cinder
block devices backed by Ceph storage using fio. Originally we were using ssh
to start independent fio processes on the guests, but we ran into problems with
threads not starting and stopping at the same time, despite using options to
use fixed elapsed time for all threads. But when we started using --client,
we noticed that thread time skew is greatly reduced, because fio is *already
running* on the server and we only have to connect to it and start the test.
This is great, but for large numbers of fio servers, it becomes impossible to
parse the output.
For example, here's a sample of per-VM JSON output:
Disk stats (read/write):
<vm4> {
<vm4> "fio version" : "fio-2.2.4",
<vm4> "timestamp" : 1421677173,
<vm4> "time" : "Mon Jan 19 09:19:33 2015",
<vm4> "jobs" : [
<vm4>
<vm4> ],
<vm4> "disk_util" : [
...
<vm4> ]
<vm4> }
https://s3.amazonaws.com/ben.england/client.c.diff
https://s3.amazonaws.com/ben.england/fio-results.2015-01-20-11-50.json.gz
The above output works fine for a few threads and is not hard to filter out,
but the various guests' output sections get mixed together for large thread
counts such as 256 client threads. if you look at the previous URL
fio-results*json.gz containing garbled output, the cause of this output is the
handle_text() routine, which is called by fio whenever a FIO_NET_CMD_TEXT PDU
is received by the fio client (in fio_handle_client()). If you use --debug=all
you can see that these PDUs are processed in whatever order they are received
from the fio server processes, essentially in random order. But almost all of
this info is in the final json object output at the end of the file (except for
the timestamp field which I wish was there!) so there is no reason to output
twice, except as a debugging tool perhaps. The patch is to V2.2.4.
[ben@localhost fio-master]$ diff -u client.{c.sav2,c}
--- client.c.sav2 2015-01-28 11:30:02.657073505 -0500
+++ client.c 2015-01-28 11:33:02.248373685 -0500
@@ -968,12 +968,6 @@
const char *name;
int fio_unused ret;
- name = client->name ? client->name : client->hostname;
-
- if (!client->skip_newline)
- fprintf(f_out, "<%s> ", name);
- ret = fwrite(buf, pdu->buf_len, 1, f_out);
- fflush(f_out);
client->skip_newline = strchr(buf, '\n') == NULL;
}
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
