Hello community, here is the log from the commit of package fio for openSUSE:Factory checked in at 2017-06-29 15:21:57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/fio (Old) and /work/SRC/openSUSE:Factory/.fio.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "fio" Thu Jun 29 15:21:57 2017 rev:38 rq:507004 version:2.21 Changes: -------- --- /work/SRC/openSUSE:Factory/fio/fio.changes 2017-05-29 22:19:54.721767291 +0200 +++ /work/SRC/openSUSE:Factory/.fio.new/fio.changes 2017-06-29 15:22:14.324636646 +0200 @@ -1,0 +2,7 @@ +Thu Jun 29 09:22:09 UTC 2017 - [email protected] + +- Update to version 2.21: + * See http://brick.kernel.dk/snaps/fio-2.21.shortlog.txt +- Enabled tests + +------------------------------------------------------------------- Old: ---- fio-2.20.tar.bz2 fio-2.20.tar.bz2.asc New: ---- fio-2.21.tar.bz2 fio-2.21.tar.bz2.asc ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ fio.spec ++++++ --- /var/tmp/diff_new_pack.qfwDcX/_old 2017-06-29 15:22:16.452335891 +0200 +++ /var/tmp/diff_new_pack.qfwDcX/_new 2017-06-29 15:22:16.456335326 +0200 @@ -18,7 +18,7 @@ Name: fio -Version: 2.20 +Version: 2.21 Release: 0 Summary: Flexible I/O Tester/benchmarker License: GPL-2.0 @@ -90,6 +90,9 @@ mandir="%{_mandir}" \ install +%check +make %{?_smp_mflags} test + %files %defattr(-,root,root) %doc COPYING README examples ++++++ fio-2.20.tar.bz2 -> fio-2.21.tar.bz2 ++++++ ++++ 2178 lines of diff (skipped) ++++ retrying with extended exclude list diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/FIO-VERSION-GEN new/fio-2.21/FIO-VERSION-GEN --- old/fio-2.20/FIO-VERSION-GEN 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/FIO-VERSION-GEN 2017-06-15 20:25:03.000000000 +0200 @@ -1,7 +1,7 @@ #!/bin/sh GVF=FIO-VERSION-FILE -DEF_VER=fio-2.20 +DEF_VER=fio-2.21 LF=' ' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/HOWTO new/fio-2.21/HOWTO --- old/fio-2.20/HOWTO 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/HOWTO 2017-06-15 20:25:03.000000000 +0200 @@ -1072,10 +1072,11 @@ .. option:: offset=int - Start I/O at the given offset in the file. The data before the given offset - will not be touched. This effectively caps the file size at `real_size - - offset`. Can be combined with :option:`size` to constrain the start and - end range that I/O will be done within. + Start I/O at the provided offset in the file, given as either a fixed size or + a percentage. If a percentage is given, the next ``blockalign``-ed offset + will be used. Data before the given offset will not be touched. This + effectively caps the file size at `real_size - offset`. Can be combined with + :option:`size` to constrain the start and end range of the I/O workload. .. option:: offset_increment=int @@ -1290,7 +1291,7 @@ means default for reads, 8k for writes and trims. **bs=,8k,** - means default for reads, 8k for writes, and default for writes. + means default for reads, 8k for writes, and default for trims. .. option:: blocksize_range=irange[,irange][,irange], bsrange=irange[,irange][,irange] @@ -1400,11 +1401,18 @@ .. option:: buffer_pattern=str - If set, fio will fill the I/O buffers with this pattern. If not set, the - contents of I/O buffers is defined by the other options related to buffer - contents. The setting can be any pattern of bytes, and can be prefixed with - 0x for hex values. It may also be a string, where the string must then be - wrapped with ``""``, e.g.:: + If set, fio will fill the I/O buffers with this pattern or with the contents + of a file. If not set, the contents of I/O buffers are defined by the other + options related to buffer contents. The setting can be any pattern of bytes, + and can be prefixed with 0x for hex values. It may also be a string, where + the string must then be wrapped with ``""``. Or it may also be a filename, + where the filename must be wrapped with ``''`` in which case the file is + opened and read. Note that not all the file contents will be read if that + would cause the buffers to overflow. So, for example:: + + buffer_pattern='filename' + + or:: buffer_pattern="abcd" @@ -1418,7 +1426,7 @@ Also you can combine everything together in any order:: - buffer_pattern=0xdeadface"abcd"-12 + buffer_pattern=0xdeadface"abcd"-12'filename' .. option:: dedupe_percentage=int @@ -2467,6 +2475,7 @@ contents to one or more separate threads. If using this offload option, even sync I/O engines can benefit from using an :option:`iodepth` setting higher than 1, as it allows them to have I/O in flight while verifies are running. + Defaults to 0 async threads, i.e. verification is not asynchronous. .. option:: verify_async_cpus=str @@ -2503,7 +2512,7 @@ <type> is "local" for a local run, "sock" for a client/server socket connection, and "ip" (192.168.0.1, for instance) for a networked - client/server connection. + client/server connection. Defaults to true. .. option:: verify_state_load=bool @@ -2834,7 +2843,8 @@ .. option:: ignore_error=str Sometimes you want to ignore some errors during test in that case you can - specify error list for each error type. + specify error list for each error type, instead of only being able to + ignore the default 'non-fatal error' using :option:`continue_on_error`. ``ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST`` errors for given error type is separated with ':'. Error may be symbol ('ENOSPC', 'ENOMEM') or integer. Example:: @@ -2842,7 +2852,8 @@ ignore_error=EAGAIN,ENOSPC:122 This option will ignore EAGAIN from READ, and ENOSPC and 122(EDQUOT) from - WRITE. + WRITE. This option works by overriding :option:`continue_on_error` with + the list of errors for each error type if any. .. option:: error_dump=bool @@ -3224,6 +3235,11 @@ For disk utilization, all disks used by fio are shown. So for each disk there will be a disk utilization section. +Below is a single line containing short names for each of the fields in the +minimal output v3, separated by semicolons: + +terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_max;read_clat_min;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_max;write_clat_min;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;pu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util + Trace file format ----------------- diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/Makefile new/fio-2.21/Makefile --- old/fio-2.20/Makefile 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/Makefile 2017-06-15 20:25:03.000000000 +0200 @@ -107,6 +107,9 @@ ifndef CONFIG_STRLCAT SOURCE += oslib/strlcat.c endif +ifndef CONFIG_HAVE_STRNDUP + SOURCE += oslib/strndup.c +endif ifndef CONFIG_GETOPT_LONG_ONLY SOURCE += oslib/getopt_long.c endif @@ -140,7 +143,7 @@ LDFLAGS += -rdynamic endif ifeq ($(CONFIG_TARGET_OS), Android) - SOURCE += diskutil.c fifo.c blktrace.c trim.c profiles/tiobench.c \ + SOURCE += diskutil.c fifo.c blktrace.c cgroup.c trim.c profiles/tiobench.c \ oslib/linux-dev-lookup.c LIBS += -ldl LDFLAGS += -rdynamic @@ -209,7 +212,8 @@ T_ZIPF_OBS = t/genzipf.o T_ZIPF_OBJS += t/log.o lib/ieee754.o lib/rand.o lib/pattern.o lib/zipf.o \ - lib/strntol.o lib/gauss.o t/genzipf.o oslib/strcasestr.o + lib/strntol.o lib/gauss.o t/genzipf.o oslib/strcasestr.o \ + oslib/strndup.o T_ZIPF_PROGS = t/fio-genzipf T_AXMAP_OBJS = t/axmap.o @@ -222,7 +226,7 @@ T_GEN_RAND_OBJS = t/gen-rand.o T_GEN_RAND_OBJS += t/log.o t/debug.o lib/rand.o lib/pattern.o lib/strntol.o \ - oslib/strcasestr.o + oslib/strcasestr.o oslib/strndup.o T_GEN_RAND_PROGS = t/gen-rand ifeq ($(CONFIG_TARGET_OS), Linux) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/blktrace.c new/fio-2.21/blktrace.c --- old/fio-2.20/blktrace.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/blktrace.c 2017-06-15 20:25:03.000000000 +0200 @@ -10,6 +10,7 @@ #include "flist.h" #include "fio.h" +#include "blktrace.h" #include "blktrace_api.h" #include "oslib/linux-dev-lookup.h" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/blktrace.h new/fio-2.21/blktrace.h --- old/fio-2.20/blktrace.h 1970-01-01 01:00:00.000000000 +0100 +++ new/fio-2.21/blktrace.h 2017-06-15 20:25:03.000000000 +0200 @@ -0,0 +1,23 @@ +#ifndef FIO_BLKTRACE_H +#define FIO_BLKTRACE_H + +#ifdef FIO_HAVE_BLKTRACE + +int is_blktrace(const char *, int *); +int load_blktrace(struct thread_data *, const char *, int); + +#else + +static inline int is_blktrace(const char *fname, int *need_swap) +{ + return 0; +} + +static inline int load_blktrace(struct thread_data *td, const char *fname, + int need_swap) +{ + return 1; +} + +#endif +#endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/cconv.c new/fio-2.21/cconv.c --- old/fio-2.20/cconv.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/cconv.c 2017-06-15 20:25:03.000000000 +0200 @@ -104,6 +104,7 @@ o->file_size_low = le64_to_cpu(top->file_size_low); o->file_size_high = le64_to_cpu(top->file_size_high); o->start_offset = le64_to_cpu(top->start_offset); + o->start_offset_percent = le32_to_cpu(top->start_offset_percent); for (i = 0; i < DDIR_RWDIR_CNT; i++) { o->bs[i] = le32_to_cpu(top->bs[i]); @@ -543,6 +544,7 @@ top->file_size_low = __cpu_to_le64(o->file_size_low); top->file_size_high = __cpu_to_le64(o->file_size_high); top->start_offset = __cpu_to_le64(o->start_offset); + top->start_offset_percent = __cpu_to_le32(o->start_offset_percent); top->trim_backlog = __cpu_to_le64(o->trim_backlog); top->offset_increment = __cpu_to_le64(o->offset_increment); top->number_ios = __cpu_to_le64(o->number_ios); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/diskutil.c new/fio-2.21/diskutil.c --- old/fio-2.20/diskutil.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/diskutil.c 2017-06-15 20:25:03.000000000 +0200 @@ -363,7 +363,7 @@ return 0; while ((dir = readdir(D)) != NULL) { - char full_path[256]; + char full_path[257]; if (!strcmp(dir->d_name, ".") || !strcmp(dir->d_name, "..")) continue; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/engines/binject.c new/fio-2.21/engines/binject.c --- old/fio-2.20/engines/binject.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/engines/binject.c 2017-06-15 20:25:03.000000000 +0200 @@ -59,11 +59,12 @@ return 0; } -static unsigned int binject_read_commands(struct thread_data *td, void *p, +static unsigned int binject_read_commands(struct thread_data *td, void *buf, int left, int *err) { struct fio_file *f; int i, ret, events; + char *p = buf; one_more: events = 0; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/engines/sg.c new/fio-2.21/engines/sg.c --- old/fio-2.20/engines/sg.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/engines/sg.c 2017-06-15 20:25:03.000000000 +0200 @@ -124,7 +124,7 @@ } while (left) { - void *p; + char *p; dprint(FD_IO, "sgio_getevents: sd %p: left=%d\n", sd, left); @@ -184,7 +184,7 @@ if (hdr->info & SG_INFO_CHECK) { struct io_u *io_u; io_u = (struct io_u *)(hdr->usr_ptr); - memcpy((void*)&(io_u->hdr), (void*)hdr, sizeof(struct sg_io_hdr)); + memcpy(&io_u->hdr, hdr, sizeof(struct sg_io_hdr)); sd->events[i]->error = EIO; } } @@ -572,17 +572,17 @@ struct sg_io_hdr *hdr = &io_u->hdr; #define MAXERRDETAIL 1024 #define MAXMSGCHUNK 128 - char *msg, msgchunk[MAXMSGCHUNK], *ret = NULL; + char *msg, msgchunk[MAXMSGCHUNK]; int i; msg = calloc(1, MAXERRDETAIL); + strcpy(msg, ""); /* * can't seem to find sg_err.h, so I'll just echo the define values * so others can search on internet to find clearer clues of meaning. */ if (hdr->info & SG_INFO_CHECK) { - ret = msg; if (hdr->host_status) { snprintf(msgchunk, MAXMSGCHUNK, "SG Host Status: 0x%02x; ", hdr->host_status); strlcat(msg, msgchunk, MAXERRDETAIL); @@ -755,14 +755,14 @@ if (hdr->resid != 0) { snprintf(msgchunk, MAXMSGCHUNK, "SG Driver: %d bytes out of %d not transferred. ", hdr->resid, hdr->dxfer_len); strlcat(msg, msgchunk, MAXERRDETAIL); - ret = msg; } } - if (!ret) - ret = strdup("SG Driver did not report a Host, Driver or Device check"); + if (!(hdr->info & SG_INFO_CHECK) && !strlen(msg)) + strncpy(msg, "SG Driver did not report a Host, Driver or Device check", + MAXERRDETAIL - 1); - return ret; + return msg; } /* diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/engines/splice.c new/fio-2.21/engines/splice.c --- old/fio-2.20/engines/splice.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/engines/splice.c 2017-06-15 20:25:03.000000000 +0200 @@ -32,7 +32,7 @@ struct fio_file *f = io_u->file; int ret, ret2, buflen; off_t offset; - void *p; + char *p; offset = io_u->offset; buflen = io_u->xfer_buflen; @@ -77,7 +77,8 @@ struct iovec iov; int ret , buflen, mmap_len; off_t offset; - void *p, *map; + void *map; + char *p; ret = 0; offset = io_u->offset; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/engines/sync.c new/fio-2.21/engines/sync.c --- old/fio-2.20/engines/sync.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/engines/sync.c 2017-06-15 20:25:03.000000000 +0200 @@ -36,6 +36,7 @@ struct psyncv2_options { void *pad; unsigned int hipri; + unsigned int stream; }; static struct fio_option options[] = { @@ -49,6 +50,29 @@ .group = FIO_OPT_G_INVALID, }, { + .name = "stream", + .lname = "Stream ID", + .type = FIO_OPT_STR, + .off1 = offsetof(struct psyncv2_options, stream), + .help = "Set expected write life time", + .category = FIO_OPT_C_ENGINE, + .group = FIO_OPT_G_INVALID, + .posval = { + { .ival = "short", + .oval = RWF_WRITE_LIFE_SHORT, + }, + { .ival = "medium", + .oval = RWF_WRITE_LIFE_MEDIUM, + }, + { .ival = "long", + .oval = RWF_WRITE_LIFE_LONG, + }, + { .ival = "extreme", + .oval = RWF_WRITE_LIFE_EXTREME, + }, + }, + }, + { .name = NULL, }, }; @@ -134,6 +158,8 @@ if (o->hipri) flags |= RWF_HIPRI; + if (o->stream) + flags |= o->stream; iov->iov_base = io_u->xfer_buf; iov->iov_len = io_u->xfer_buflen; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/filesetup.c new/fio-2.21/filesetup.c --- old/fio-2.20/filesetup.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/filesetup.c 2017-06-15 20:25:03.000000000 +0200 @@ -833,12 +833,42 @@ uint64_t get_start_offset(struct thread_data *td, struct fio_file *f) { struct thread_options *o = &td->o; + unsigned long long align_bs; + unsigned long long offset; if (o->file_append && f->filetype == FIO_TYPE_FILE) return f->real_file_size; - return td->o.start_offset + - td->subjob_number * td->o.offset_increment; + if (o->start_offset_percent > 0) { + /* + * if blockalign is provided, find the min across read, write, + * and trim + */ + if (fio_option_is_set(o, ba)) { + align_bs = (unsigned long long) min(o->ba[DDIR_READ], o->ba[DDIR_WRITE]); + align_bs = min((unsigned long long) o->ba[DDIR_TRIM], align_bs); + } else { + /* else take the minimum block size */ + align_bs = td_min_bs(td); + } + + /* calculate the raw offset */ + offset = (f->real_file_size * o->start_offset_percent / 100) + + (td->subjob_number * o->offset_increment); + + /* + * block align the offset at the next available boundary at + * ceiling(offset / align_bs) * align_bs + */ + offset = (offset / align_bs + (offset % align_bs != 0)) * align_bs; + + } else { + /* start_offset_percent not set */ + offset = o->start_offset + + td->subjob_number * o->offset_increment; + } + + return offset; } /* @@ -986,7 +1016,14 @@ total_size = -1ULL; else { if (o->size_percent) { - f->io_size = (f->io_size * o->size_percent) / 100; + uint64_t file_size; + + file_size = f->io_size + f->file_offset; + f->io_size = (file_size * + o->size_percent) / 100; + if (f->io_size > (file_size - f->file_offset)) + f->io_size = file_size - f->file_offset; + f->io_size -= (f->io_size % td_min_bs(td)); } total_size += f->io_size; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/fio.1 new/fio-2.21/fio.1 --- old/fio-2.20/fio.1 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/fio.1 2017-06-15 20:25:03.000000000 +0200 @@ -1,4 +1,4 @@ -.TH fio 1 "March 2017" "User Manual" +.TH fio 1 "May 2017" "User Manual" .SH NAME fio \- flexible I/O tester .SH SYNOPSIS @@ -533,7 +533,7 @@ bs=8k,32k means 8k for reads, 32k for writes and trims bs=8k,32k, means 8k for reads, 32k for writes, and default for trims bs=,8k means default for reads, 8k for writes and trims -bs=,8k, means default for reads, 8k for writes, and default for writes +bs=,8k, means default for reads, 8k for writes, and default for trims .fi .TP .BI blocksize_range \fR=\fPirange[,irange][,irange] "\fR,\fB bsrange" \fR=\fPirange[,irange][,irange] @@ -611,13 +611,20 @@ size, fio can alternate random and zeroed data throughout the IO buffer. .TP .BI buffer_pattern \fR=\fPstr -If set, fio will fill the IO buffers with this pattern. If not set, the contents -of IO buffers is defined by the other options related to buffer contents. The -setting can be any pattern of bytes, and can be prefixed with 0x for hex -values. It may also be a string, where the string must then be wrapped with -"", e.g.: +If set, fio will fill the I/O buffers with this pattern or with the contents +of a file. If not set, the contents of I/O buffers are defined by the other +options related to buffer contents. The setting can be any pattern of bytes, +and can be prefixed with 0x for hex values. It may also be a string, where +the string must then be wrapped with ``""``. Or it may also be a filename, +where the filename must be wrapped with ``''`` in which case the file is +opened and read. Note that not all the file contents will be read if that +would cause the buffers to overflow. So, for example: .RS .RS +\fBbuffer_pattern\fR='filename' +.RS +or +.RE \fBbuffer_pattern\fR="abcd" .RS or @@ -632,7 +639,7 @@ Also you can combine everything together in any order: .LP .RS -\fBbuffer_pattern\fR=0xdeadface"abcd"-12 +\fBbuffer_pattern\fR=0xdeadface"abcd"-12'filename' .RE .RE .TP @@ -904,7 +911,11 @@ Default: true. .TP .BI offset \fR=\fPint -Offset in the file to start I/O. Data before the offset will not be touched. +Start I/O at the provided offset in the file, given as either a fixed size or a +percentage. If a percentage is given, the next \fBblockalign\fR-ed offset will +be used. Data before the given offset will not be touched. This effectively +caps the file size at (real_size - offset). Can be combined with \fBsize\fR to +constrain the start and end range of the I/O workload. .TP .BI offset_increment \fR=\fPint If this is provided, then the real offset becomes the @@ -2241,6 +2252,15 @@ .P .B text description (if provided in config - appears on newline) .RE +.P +Below is a single line containing short names for each of the fields in +the minimal output v3, separated by semicolons: +.RS +.P +.nf +terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_max;read_clat_min;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_max;write_clat_min;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;pu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util +.fi +.RE .SH TRACE FILE FORMAT There are two trace file format that you can encounter. The older (v1) format is unsupported since version 1.20-rc3 (March 2008). It will still be described @@ -2594,7 +2614,7 @@ .br These are typically located under /usr/share/doc/fio. -\fBHOWTO\fR: http://git.kernel.dk/?p=fio.git;a=blob_plain;f=HOWTO +\fBHOWTO\fR: http://git.kernel.dk/cgit/fio/plain/HOWTO .br -\fBREADME\fR: http://git.kernel.dk/?p=fio.git;a=blob_plain;f=README +\fBREADME\fR: http://git.kernel.dk/cgit/fio/plain/README .br diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/fio.h new/fio-2.21/fio.h --- old/fio-2.20/fio.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/fio.h 2017-06-15 20:25:03.000000000 +0200 @@ -149,7 +149,7 @@ unsigned int thread_number; unsigned int subjob_number; unsigned int groupid; - struct thread_stat ts; + struct thread_stat ts __attribute__ ((aligned)); int client_type; @@ -640,14 +640,6 @@ */ extern void reset_all_stats(struct thread_data *); -/* - * blktrace support - */ -#ifdef FIO_HAVE_BLKTRACE -extern int is_blktrace(const char *, int *); -extern int load_blktrace(struct thread_data *, const char *, int); -#endif - extern int io_queue_event(struct thread_data *td, struct io_u *io_u, int *ret, enum fio_ddir ddir, uint64_t *bytes_issued, int from_verify, struct timeval *comp_time); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/flist.h new/fio-2.21/flist.h --- old/fio-2.20/flist.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/flist.h 2017-06-15 20:25:03.000000000 +0200 @@ -2,13 +2,7 @@ #define _LINUX_FLIST_H #include <stdlib.h> - -#undef offsetof -#ifdef __compiler_offsetof -#define offsetof(TYPE,MEMBER) __compiler_offsetof(TYPE,MEMBER) -#else -#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER) -#endif +#include <stddef.h> #define container_of(ptr, type, member) ({ \ const typeof( ((type *)0)->member ) *__mptr = (ptr); \ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/init.c new/fio-2.21/init.c --- old/fio-2.20/init.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/init.c 2017-06-15 20:25:03.000000000 +0200 @@ -909,9 +909,9 @@ bool use64; if (td->o.random_generator == FIO_RAND_GEN_TAUSWORTHE64) - use64 = 1; + use64 = true; else - use64 = 0; + use64 = false; init_rand_seed(&td->verify_state, td->rand_seeds[FIO_RAND_VER_OFF], use64); @@ -967,9 +967,9 @@ } if (td->o.random_generator == FIO_RAND_GEN_TAUSWORTHE64) - use64 = 1; + use64 = true; else - use64 = 0; + use64 = false; td_fill_rand_seeds_internal(td, use64); @@ -1080,8 +1080,12 @@ unsigned long seed; unsigned int i; - if (!td->o.rand_repeatable && !fio_option_is_set(&td->o, rand_seed)) - return init_random_state(td, td->rand_seeds, sizeof(td->rand_seeds)); + if (!td->o.rand_repeatable && !fio_option_is_set(&td->o, rand_seed)) { + int ret = init_random_seeds(td->rand_seeds, sizeof(td->rand_seeds)); + if (!ret) + td_fill_rand_seeds(td); + return ret; + } seed = td->o.rand_seed; for (i = 0; i < 4; i++) @@ -1376,7 +1380,7 @@ prev_group_jobs++; if (setup_random_seeds(td)) { - td_verror(td, errno, "init_random_state"); + td_verror(td, errno, "setup_random_seeds"); goto err; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/iolog.c new/fio-2.21/iolog.c --- old/fio-2.20/iolog.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/iolog.c 2017-06-15 20:25:03.000000000 +0200 @@ -19,6 +19,7 @@ #include "trim.h" #include "filelock.h" #include "smalloc.h" +#include "blktrace.h" static int iolog_flush(struct io_log *log); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/lib/output_buffer.c new/fio-2.21/lib/output_buffer.c --- old/fio-2.20/lib/output_buffer.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/lib/output_buffer.c 2017-06-15 20:25:03.000000000 +0200 @@ -3,7 +3,6 @@ #include <stdlib.h> #include "output_buffer.h" -#include "../log.h" #include "../minmax.h" #define BUF_INC 1024 @@ -18,6 +17,7 @@ void buf_output_free(struct buf_output *out) { free(out->buf); + buf_output_init(out); } size_t buf_output_add(struct buf_output *out, const char *buf, size_t len) @@ -40,16 +40,3 @@ out->buflen += len; return len; } - -size_t buf_output_flush(struct buf_output *out) -{ - size_t ret = 0; - - if (out->buflen) { - ret = log_info_buf(out->buf, out->buflen); - memset(out->buf, 0, out->max_buflen); - out->buflen = 0; - } - - return ret; -} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/lib/output_buffer.h new/fio-2.21/lib/output_buffer.h --- old/fio-2.20/lib/output_buffer.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/lib/output_buffer.h 2017-06-15 20:25:03.000000000 +0200 @@ -12,6 +12,5 @@ void buf_output_init(struct buf_output *out); void buf_output_free(struct buf_output *out); size_t buf_output_add(struct buf_output *out, const char *buf, size_t len); -size_t buf_output_flush(struct buf_output *out); #endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/lib/pattern.c new/fio-2.21/lib/pattern.c --- old/fio-2.20/lib/pattern.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/lib/pattern.c 2017-06-15 20:25:03.000000000 +0200 @@ -4,11 +4,74 @@ #include <limits.h> #include <errno.h> #include <assert.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <unistd.h> #include "strntol.h" #include "pattern.h" #include "../minmax.h" #include "../oslib/strcasestr.h" +#include "../oslib/strndup.h" + +/** + * parse_file() - parses binary file to fill buffer + * @beg - string input, extract filename from this + * @out - output buffer where parsed number should be put + * @out_len - length of the output buffer + * @filled - pointer where number of bytes successfully + * parsed will be put + * + * Returns the end pointer where parsing has been stopped. + * In case of parsing error or lack of bytes in output buffer + * NULL will be returned. + */ +static const char *parse_file(const char *beg, char *out, + unsigned int out_len, + unsigned int *filled) +{ + const char *end; + char *file; + int fd; + ssize_t count; + + if (!out_len) + goto err_out; + + assert(*beg == '\''); + beg++; + end = strchr(beg, '\''); + if (!end) + goto err_out; + + file = strndup(beg, end - beg); + if (file == NULL) + goto err_out; + + fd = open(file, O_RDONLY); + if (fd < 0) + goto err_free_out; + + count = read(fd, out, out_len); + if (count == -1) + goto err_free_close_out; + + *filled = count; + close(fd); + free(file); + + /* Catch up quote */ + return end + 1; + +err_free_close_out: + close(fd); +err_free_out: + free(file); +err_out: + return NULL; + +} /** * parse_string() - parses string in double quotes, like "abc" @@ -271,6 +334,9 @@ parsed_fmt = 0; switch (*beg) { + case '\'': + end = parse_file(beg, out, out_len, &filled); + break; case '"': end = parse_string(beg, out, out_len, &filled); break; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/libfio.c new/fio-2.21/libfio.c --- old/fio-2.20/libfio.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/libfio.c 2017-06-15 20:25:03.000000000 +0200 @@ -353,14 +353,17 @@ * can run into problems on archs that fault on unaligned fp * access (ARM). */ + compiletime_assert((offsetof(struct thread_data, ts) % sizeof(void *)) == 0, "ts"); compiletime_assert((offsetof(struct thread_stat, percentile_list) % 8) == 0, "stat percentile_list"); compiletime_assert((offsetof(struct thread_stat, total_run_time) % 8) == 0, "total_run_time"); compiletime_assert((offsetof(struct thread_stat, total_err_count) % 8) == 0, "total_err_count"); compiletime_assert((offsetof(struct thread_stat, latency_percentile) % 8) == 0, "stat latency_percentile"); + compiletime_assert((offsetof(struct thread_data, ts.clat_stat) % 8) == 0, "ts.clat_stat"); compiletime_assert((offsetof(struct thread_options_pack, zipf_theta) % 8) == 0, "zipf_theta"); compiletime_assert((offsetof(struct thread_options_pack, pareto_h) % 8) == 0, "pareto_h"); compiletime_assert((offsetof(struct thread_options_pack, percentile_list) % 8) == 0, "percentile_list"); compiletime_assert((offsetof(struct thread_options_pack, latency_percentile) % 8) == 0, "latency_percentile"); + compiletime_assert((offsetof(struct jobs_eta, m_rate) % 8) == 0, "m_rate"); err = endian_check(); if (err) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/log.c new/fio-2.21/log.c --- old/fio-2.20/log.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/log.c 2017-06-15 20:25:03.000000000 +0200 @@ -6,8 +6,16 @@ #include "fio.h" +#define LOG_START_SZ 512 + size_t log_info_buf(const char *buf, size_t len) { + /* + * buf could be NULL (not just ""). + */ + if (!buf) + return 0; + if (is_backend) { size_t ret = fio_server_text_output(FIO_LOG_INFO, buf, len); if (ret != -1) @@ -21,40 +29,66 @@ return fwrite(buf, len, 1, f_out); } -size_t log_valist(const char *str, va_list args) +static size_t valist_to_buf(char **buffer, const char *fmt, va_list src_args) { - char buffer[1024]; + size_t len, cur = LOG_START_SZ; + va_list args; + + do { + *buffer = calloc(1, cur); + + va_copy(args, src_args); + len = vsnprintf(*buffer, cur, fmt, args); + va_end(args); + + if (len < cur) + break; + + cur = len + 1; + free(*buffer); + } while (1); + + return len; +} + +size_t log_valist(const char *fmt, va_list args) +{ + char *buffer; size_t len; - len = vsnprintf(buffer, sizeof(buffer), str, args); + len = valist_to_buf(&buffer, fmt, args); + len = log_info_buf(buffer, len); + free(buffer); - return log_info_buf(buffer, min(len, sizeof(buffer) - 1)); + return len; } size_t log_info(const char *format, ...) { - char buffer[1024]; va_list args; - size_t len; + size_t ret; va_start(args, format); - len = vsnprintf(buffer, sizeof(buffer), format, args); + ret = log_valist(format, args); va_end(args); - return log_info_buf(buffer, min(len, sizeof(buffer) - 1)); + return ret; } size_t __log_buf(struct buf_output *buf, const char *format, ...) { - char buffer[1024]; + char *buffer; va_list args; size_t len; va_start(args, format); - len = vsnprintf(buffer, sizeof(buffer), format, args); + len = valist_to_buf(&buffer, format, args); va_end(args); - return buf_output_add(buf, buffer, min(len, sizeof(buffer) - 1)); + len = buf_output_add(buf, buffer, len); + free(buffer); + + return len; } int log_info_flush(void) @@ -67,33 +101,33 @@ size_t log_err(const char *format, ...) { - char buffer[1024]; + size_t ret, len; + char *buffer; va_list args; - size_t len; va_start(args, format); - len = vsnprintf(buffer, sizeof(buffer), format, args); + len = valist_to_buf(&buffer, format, args); va_end(args); - len = min(len, sizeof(buffer) - 1); if (is_backend) { - size_t ret = fio_server_text_output(FIO_LOG_ERR, buffer, len); + ret = fio_server_text_output(FIO_LOG_ERR, buffer, len); if (ret != -1) - return ret; + goto done; } if (log_syslog) { syslog(LOG_INFO, "%s", buffer); - return len; + ret = len; } else { - if (f_err != stderr) { - int fio_unused ret; - + if (f_err != stderr) ret = fwrite(buffer, len, 1, stderr); - } - return fwrite(buffer, len, 1, f_err); + ret = fwrite(buffer, len, 1, f_err); } + +done: + free(buffer); + return ret; } const char *log_get_level(int level) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/log.h new/fio-2.21/log.h --- old/fio-2.20/log.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/log.h 2017-06-15 20:25:03.000000000 +0200 @@ -16,13 +16,15 @@ extern size_t log_info_buf(const char *buf, size_t len); extern int log_info_flush(void); -#define log_buf(buf, format, args...) \ -do { \ - if ((buf) != NULL) \ - __log_buf(buf, format, ##args); \ - else \ - log_info(format, ##args); \ -} while (0) +#define log_buf(buf, format, args...) \ +({ \ + size_t __ret; \ + if ((buf) != NULL) \ + __ret = __log_buf(buf, format, ##args); \ + else \ + __ret = log_info(format, ##args); \ + __ret; \ +}) enum { FIO_LOG_DEBUG = 1, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/options.c new/fio-2.21/options.c --- old/fio-2.20/options.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/options.c 2017-06-15 20:25:03.000000000 +0200 @@ -270,7 +270,8 @@ return 0; } -static int ignore_error_type(struct thread_data *td, int etype, char *str) +static int ignore_error_type(struct thread_data *td, enum error_type_bit etype, + char *str) { unsigned int i; int *error; @@ -282,7 +283,7 @@ } td->o.ignore_error_nr[etype] = 4; - error = malloc(4 * sizeof(struct bssplit)); + error = calloc(4, sizeof(int)); i = 0; while ((fname = strsep(&str, ":")) != NULL) { @@ -306,8 +307,9 @@ error[i] = -error[i]; } if (!error[i]) { - log_err("Unknown error %s, please use number value \n", + log_err("Unknown error %s, please use number value\n", fname); + td->o.ignore_error_nr[etype] = 0; free(error); return 1; } @@ -317,8 +319,10 @@ td->o.continue_on_error |= 1 << etype; td->o.ignore_error_nr[etype] = i; td->o.ignore_error[etype] = error; - } else + } else { + td->o.ignore_error_nr[etype] = 0; free(error); + } return 0; @@ -328,7 +332,8 @@ { struct thread_data *td = cb_data_to_td(data); char *str, *p, *n; - int type = 0, ret = 1; + int ret = 1; + enum error_type_bit type = 0; if (parse_dryrun()) return 0; @@ -1381,6 +1386,22 @@ return 0; } +static int str_offset_cb(void *data, unsigned long long *__val) +{ + struct thread_data *td = cb_data_to_td(data); + unsigned long long v = *__val; + + if (parse_is_percent(v)) { + td->o.start_offset = 0; + td->o.start_offset_percent = -1ULL - v; + dprint(FD_PARSE, "SET start_offset_percent %d\n", + td->o.start_offset_percent); + } else + td->o.start_offset = v; + + return 0; +} + static int str_size_cb(void *data, unsigned long long *__val) { struct thread_data *td = cb_data_to_td(data); @@ -1938,6 +1959,7 @@ .lname = "IO offset", .alias = "fileoffset", .type = FIO_OPT_STR_VAL, + .cb = str_offset_cb, .off1 = offsetof(struct thread_options, start_offset), .help = "Start IO from this offset", .def = "0", diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/os/os-android.h new/fio-2.21/os/os-android.h --- old/fio-2.20/os/os-android.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/os/os-android.h 2017-06-15 20:25:03.000000000 +0200 @@ -32,6 +32,7 @@ #define FIO_HAVE_HUGETLB #define FIO_HAVE_BLKTRACE #define FIO_HAVE_CL_SIZE +#define FIO_HAVE_CGROUPS #define FIO_HAVE_FS_STAT #define FIO_HAVE_TRIM #define FIO_HAVE_GETTID @@ -59,19 +60,17 @@ #ifndef CONFIG_NO_SHM /* - * The Android NDK doesn't currently export <sys/shm.h>, so define the - * necessary stuff here. + * Bionic doesn't support SysV shared memeory, so implement it using ashmem */ - -#include <sys/shm.h> -#define SHM_HUGETLB 04000 - #include <stdio.h> #include <linux/ashmem.h> +#include <linux/shm.h> +#define shmid_ds shmid64_ds +#define SHM_HUGETLB 04000 #define ASHMEM_DEVICE "/dev/ashmem" -static inline int shmctl (int __shmid, int __cmd, struct shmid_ds *__buf) +static inline int shmctl(int __shmid, int __cmd, struct shmid_ds *__buf) { int ret=0; if (__cmd == IPC_RMID) @@ -84,7 +83,7 @@ return ret; } -static inline int shmget (key_t __key, size_t __size, int __shmflg) +static inline int shmget(key_t __key, size_t __size, int __shmflg) { int fd,ret; char keybuf[11]; @@ -98,7 +97,8 @@ if (ret < 0) goto error; - ret = ioctl(fd, ASHMEM_SET_SIZE, __size); + /* Stores size in first 8 bytes, allocate extra space */ + ret = ioctl(fd, ASHMEM_SET_SIZE, __size + sizeof(uint64_t)); if (ret < 0) goto error; @@ -109,21 +109,22 @@ return ret; } -static inline void *shmat (int __shmid, const void *__shmaddr, int __shmflg) +static inline void *shmat(int __shmid, const void *__shmaddr, int __shmflg) { - size_t *ptr, size = ioctl(__shmid, ASHMEM_GET_SIZE, NULL); - ptr = mmap(NULL, size + sizeof(size_t), PROT_READ | PROT_WRITE, MAP_SHARED, __shmid, 0); - *ptr = size; //save size at beginning of buffer, for use with munmap - return &ptr[1]; + size_t size = ioctl(__shmid, ASHMEM_GET_SIZE, NULL); + /* Needs to be 8-byte aligned to prevent SIGBUS on 32-bit ARM */ + uint64_t *ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, __shmid, 0); + /* Save size at beginning of buffer, for use with munmap */ + *ptr = size; + return ptr + 1; } static inline int shmdt (const void *__shmaddr) { - size_t *ptr, size; - ptr = (size_t *)__shmaddr; - ptr--; - size = *ptr; //find mmap size which we stored at the beginning of the buffer - return munmap((void *)ptr, size + sizeof(size_t)); + /* Find mmap size which we stored at the beginning of the buffer */ + uint64_t *ptr = (uint64_t *)__shmaddr - 1; + size_t size = *ptr; + return munmap(ptr, size); } #endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/os/os-linux.h new/fio-2.21/os/os-linux.h --- old/fio-2.20/os/os-linux.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/os/os-linux.h 2017-06-15 20:25:03.000000000 +0200 @@ -16,6 +16,8 @@ #include <linux/unistd.h> #include <linux/raw.h> #include <linux/major.h> +#include <linux/fs.h> +#include <scsi/sg.h> #include "./os-linux-syscall.h" #include "binject.h" @@ -258,6 +260,14 @@ return atoi(size); } +#ifdef __powerpc64__ +#define FIO_HAVE_CPU_ONLINE_SYSCONF +static inline unsigned int cpus_online(void) +{ + return sysconf(_SC_NPROCESSORS_CONF); +} +#endif + static inline unsigned long long get_fs_free_size(const char *path) { unsigned long long ret; @@ -309,6 +319,14 @@ #define RWF_SYNC 0x00000004 #endif +#ifndef RWF_WRITE_LIFE_SHIFT +#define RWF_WRITE_LIFE_SHIFT 4 +#define RWF_WRITE_LIFE_SHORT (1 << RWF_WRITE_LIFE_SHIFT) +#define RWF_WRITE_LIFE_MEDIUM (2 << RWF_WRITE_LIFE_SHIFT) +#define RWF_WRITE_LIFE_LONG (3 << RWF_WRITE_LIFE_SHIFT) +#define RWF_WRITE_LIFE_EXTREME (4 << RWF_WRITE_LIFE_SHIFT) +#endif + #ifndef CONFIG_PWRITEV2 #ifdef __NR_preadv2 static inline void make_pos_h_l(unsigned long *pos_h, unsigned long *pos_l, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/os/os-windows.h new/fio-2.21/os/os-windows.h --- old/fio-2.20/os/os-windows.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/os/os-windows.h 2017-06-15 20:25:03.000000000 +0200 @@ -116,7 +116,6 @@ ssize_t pread(int fildes, void *buf, size_t nbyte, off_t offset); ssize_t pwrite(int fildes, const void *buf, size_t nbyte, off_t offset); -extern void td_fill_rand_seeds(struct thread_data *); static inline int blockdev_size(struct fio_file *f, unsigned long long *bytes) { @@ -239,7 +238,7 @@ return 0; } -static inline int init_random_state(struct thread_data *td, unsigned long *rand_seeds, int size) +static inline int init_random_seeds(unsigned long *rand_seeds, int size) { HCRYPTPROV hCryptProv; @@ -258,7 +257,6 @@ } CryptReleaseContext(hCryptProv, 0); - td_fill_rand_seeds(td); return 0; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/os/os.h new/fio-2.21/os/os.h --- old/fio-2.20/os/os.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/os/os.h 2017-06-15 20:25:03.000000000 +0200 @@ -60,11 +60,6 @@ #endif #endif -#ifdef FIO_HAVE_SGIO -#include <linux/fs.h> -#include <scsi/sg.h> -#endif - #ifndef CONFIG_STRSEP #include "../oslib/strsep.h" #endif @@ -253,19 +248,6 @@ __cpu_to_le64(val); \ }) -#ifndef FIO_HAVE_BLKTRACE -static inline int is_blktrace(const char *fname, int *need_swap) -{ - return 0; -} -struct thread_data; -static inline int load_blktrace(struct thread_data *td, const char *fname, - int need_swap) -{ - return 1; -} -#endif - #define FIO_DEF_CL_SIZE 128 static inline int os_cache_line_size(void) @@ -316,12 +298,7 @@ #endif #ifdef FIO_USE_GENERIC_INIT_RANDOM_STATE -extern void td_fill_rand_seeds(struct thread_data *td); -/* - * Initialize the various random states we need (random io, block size ranges, - * read/write mix, etc). - */ -static inline int init_random_state(struct thread_data *td, unsigned long *rand_seeds, int size) +static inline int init_random_seeds(unsigned long *rand_seeds, int size) { int fd; @@ -336,7 +313,6 @@ } close(fd); - td_fill_rand_seeds(td); return 0; } #endif @@ -348,14 +324,6 @@ } #endif -#ifdef __powerpc64__ -#define FIO_HAVE_CPU_ONLINE_SYSCONF -static inline unsigned int cpus_online(void) -{ - return sysconf(_SC_NPROCESSORS_CONF); -} -#endif - #ifndef FIO_HAVE_CPU_ONLINE_SYSCONF static inline unsigned int cpus_online(void) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/os/windows/install.wxs new/fio-2.21/os/windows/install.wxs --- old/fio-2.20/os/windows/install.wxs 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/os/windows/install.wxs 2017-06-15 20:25:03.000000000 +0200 @@ -10,7 +10,7 @@ <Product Id="*" Codepage="1252" Language="1033" Manufacturer="fio" Name="fio" - UpgradeCode="2338A332-5511-43CF-B9BD-5C60496CCFCC" Version="2.20"> + UpgradeCode="2338A332-5511-43CF-B9BD-5C60496CCFCC" Version="2.21"> <Package Description="Flexible IO Tester" InstallerVersion="301" Keywords="Installer,MSI,Database" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/oslib/linux-dev-lookup.c new/fio-2.21/oslib/linux-dev-lookup.c --- old/fio-2.20/oslib/linux-dev-lookup.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/oslib/linux-dev-lookup.c 2017-06-15 20:25:03.000000000 +0200 @@ -20,7 +20,7 @@ return 0; while ((dir = readdir(D)) != NULL) { - char full_path[256]; + char full_path[257]; if (!strcmp(dir->d_name, ".") || !strcmp(dir->d_name, "..")) continue; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/oslib/strndup.c new/fio-2.21/oslib/strndup.c --- old/fio-2.20/oslib/strndup.c 1970-01-01 01:00:00.000000000 +0100 +++ new/fio-2.21/oslib/strndup.c 2017-06-15 20:25:03.000000000 +0200 @@ -0,0 +1,18 @@ +#include <stdlib.h> +#include "strndup.h" + +#ifndef CONFIG_HAVE_STRNDUP + +char *strndup(const char *s, size_t n) +{ + char *str = malloc(n + 1); + + if (str) { + strncpy(str, s, n); + str[n] = '\0'; + } + + return str; +} + +#endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/oslib/strndup.h new/fio-2.21/oslib/strndup.h --- old/fio-2.20/oslib/strndup.h 1970-01-01 01:00:00.000000000 +0100 +++ new/fio-2.21/oslib/strndup.h 2017-06-15 20:25:03.000000000 +0200 @@ -0,0 +1,7 @@ +#include <string.h> + +#ifndef CONFIG_HAVE_STRNDUP + +char *strndup(const char *s, size_t n); + +#endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/server.c new/fio-2.21/server.c --- old/fio-2.20/server.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/server.c 2017-06-15 20:25:03.000000000 +0200 @@ -1279,7 +1279,7 @@ ret = getsockname(sk, sockaddr_p, &len); if (ret) { - log_err("fio: getsockaddr: %s\n", strerror(errno)); + log_err("fio: getsockname: %s\n", strerror(errno)); return -1; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/server.h new/fio-2.21/server.h --- old/fio-2.20/server.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/server.h 2017-06-15 20:25:03.000000000 +0200 @@ -49,7 +49,7 @@ }; enum { - FIO_SERVER_VER = 61, + FIO_SERVER_VER = 62, FIO_SERVER_MAX_FRAGMENT_PDU = 1024, FIO_SERVER_MAX_CMD_MB = 2048, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/stat.c new/fio-2.21/stat.c --- old/fio-2.20/stat.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/stat.c 2017-06-15 20:25:03.000000000 +0200 @@ -919,7 +919,8 @@ iops = (1000.0 * (uint64_t) ts->total_io_u[ddir]) / runt; } - json_object_add_value_int(dir_object, "io_bytes", ts->io_bytes[ddir] >> 10); + json_object_add_value_int(dir_object, "io_bytes", ts->io_bytes[ddir]); + json_object_add_value_int(dir_object, "io_kbytes", ts->io_bytes[ddir] >> 10); json_object_add_value_int(dir_object, "bw", bw); json_object_add_value_float(dir_object, "iops", iops); json_object_add_value_int(dir_object, "runtime", ts->runtime[ddir]); @@ -1825,8 +1826,10 @@ } for (i = 0; i < FIO_OUTPUT_NR; i++) { - buf_output_flush(&output[i]); - buf_output_free(&output[i]); + struct buf_output *out = &output[i]; + + log_info_buf(out->buf, out->buflen); + buf_output_free(out); } log_info_flush(); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/stat.h new/fio-2.21/stat.h --- old/fio-2.20/stat.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/stat.h 2017-06-15 20:25:03.000000000 +0200 @@ -242,17 +242,17 @@ uint32_t nr_pending; uint32_t nr_setting_up; - uint32_t files_open; - uint64_t m_rate[DDIR_RWDIR_CNT], t_rate[DDIR_RWDIR_CNT]; - uint32_t m_iops[DDIR_RWDIR_CNT], t_iops[DDIR_RWDIR_CNT]; uint64_t rate[DDIR_RWDIR_CNT]; + uint32_t m_iops[DDIR_RWDIR_CNT], t_iops[DDIR_RWDIR_CNT]; uint32_t iops[DDIR_RWDIR_CNT]; uint64_t elapsed_sec; uint64_t eta_sec; uint32_t is_pow2; uint32_t unit_base; + uint32_t files_open; + /* * Network 'copy' of run_str[] */ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/td_error.c new/fio-2.21/td_error.c --- old/fio-2.20/td_error.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/td_error.c 2017-06-15 20:25:03.000000000 +0200 @@ -20,8 +20,7 @@ if (!td->o.ignore_error[etype]) { td->o.ignore_error[etype] = __NON_FATAL_ERR; - td->o.ignore_error_nr[etype] = sizeof(__NON_FATAL_ERR) - / sizeof(int); + td->o.ignore_error_nr[etype] = ARRAY_SIZE(__NON_FATAL_ERR); } if (!(td->o.continue_on_error & (1 << etype))) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/td_error.h new/fio-2.21/td_error.h --- old/fio-2.20/td_error.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/td_error.h 2017-06-15 20:25:03.000000000 +0200 @@ -2,7 +2,8 @@ #define FIO_TD_ERROR_H /* - * What type of errors to continue on when continue_on_error is used + * What type of errors to continue on when continue_on_error is used, + * and what type of errors to ignore when ignore_error is used. */ enum error_type_bit { ERROR_TYPE_READ_BIT = 0, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/thread_options.h new/fio-2.21/thread_options.h --- old/fio-2.20/thread_options.h 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/thread_options.h 2017-06-15 20:25:03.000000000 +0200 @@ -200,6 +200,7 @@ unsigned int numa_mem_prefer_node; char *numa_memnodes; unsigned int gpu_dev_id; + unsigned int start_offset_percent; unsigned int iolog; unsigned int rwmixcycle; @@ -469,7 +470,7 @@ uint8_t log_gz_cpumask[FIO_TOP_STR_MAX]; #endif uint32_t gpu_dev_id; - uint32_t pad; + uint32_t start_offset_percent; uint32_t cpus_allowed_policy; uint32_t iolog; uint32_t rwmixcycle; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' --exclude Makefile.in --exclude configure --exclude config.guess --exclude '*.pot' --exclude mkinstalldirs --exclude aclocal.m4 --exclude config.sub --exclude depcomp --exclude install-sh --exclude ltmain.sh old/fio-2.20/verify.c new/fio-2.21/verify.c --- old/fio-2.20/verify.c 2017-05-19 16:25:27.000000000 +0200 +++ new/fio-2.21/verify.c 2017-06-15 20:25:03.000000000 +0200 @@ -271,6 +271,7 @@ fd = open(fname, O_CREAT | O_TRUNC | O_WRONLY, 0644); if (fd < 0) { perror("open verify buf file"); + free(ptr); return; }
