Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
Em Fri, Jul 31, 2020 at 06:46:21PM -0600, David Ahern escreveu: > On 7/31/20 12:05 PM, pet...@infradead.org wrote: > > On Fri, Jul 31, 2020 at 08:36:12AM -0700, Andi Kleen wrote: > >>> yep, we have a customer that needs to compare data from multiple servers > >> It's also needed to correlate over different guests on the same machine. > >> This is an important use case. > > Both these cases you want to sync up CLOCK_MONOTONIC, using walltime is > > just utterly misguided. > Every userspace component logs in walltime. You can say that is > misguided, but that is the way it is. The missing piece is the ability > to correlate kernel events to userspace logs. > > What happens if the servers have (per accident or otherwise) different > > DST settings, or someone does a clock_setttime() for giggles. > Yes, someone *could* change the time. Someone *could* start ntpd or > other time server in the middle of a session. While technically such > things can happen, that is not real life in most environments (e.g., > Data center servers). ntpd (or other) is started at boot, and it is just > the little misc adjustments that happen over time. > We could add tracepoints and detect the changes and invalidate the > reference time. We could add tracepoints to track the adjustments and > update the reference time. In my experience over 9+ years using this > tool (out of tree patches) that has never been the problem. > > All you really want is a clock that runs at the same rate but is not > > subject to random jumps and user foibles. > All I want is to compare user logs to a kernel event via timestamps. Can we have both possibilities and leave the decision on which one to use in the hands of those who have a gun to shoot wherever they want, maybe in the foot? - Arnaldo
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
On 7/31/20 12:05 PM, pet...@infradead.org wrote: > On Fri, Jul 31, 2020 at 08:36:12AM -0700, Andi Kleen wrote: >>> yep, we have a customer that needs to compare data from multiple servers >> >> It's also needed to correlate over different guests on the same machine. >> This is an important use case. > > Both these cases you want to sync up CLOCK_MONOTONIC, using walltime is > just utterly misguided. Every userspace component logs in walltime. You can say that is misguided, but that is the way it is. The missing piece is the ability to correlate kernel events to userspace logs. > > What happens if the servers have (per accident or otherwise) different > DST settings, or someone does a clock_setttime() for giggles. Yes, someone *could* change the time. Someone *could* start ntpd or other time server in the middle of a session. While technically such things can happen, that is not real life in most environments (e.g., Data center servers). ntpd (or other) is started at boot, and it is just the little misc adjustments that happen over time. We could add tracepoints and detect the changes and invalidate the reference time. We could add tracepoints to track the adjustments and update the reference time. In my experience over 9+ years using this tool (out of tree patches) that has never been the problem. > > All you really want is a clock that runs at the same rate but is not > subject to random jumps and user foibles. > All I want is to compare user logs to a kernel event via timestamps.
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
On Fri, Jul 31, 2020 at 08:36:12AM -0700, Andi Kleen wrote: > > yep, we have a customer that needs to compare data from multiple servers > > It's also needed to correlate over different guests on the same machine. > This is an important use case. Both these cases you want to sync up CLOCK_MONOTONIC, using walltime is just utterly misguided. What happens if the servers have (per accident or otherwise) different DST settings, or someone does a clock_setttime() for giggles. All you really want is a clock that runs at the same rate but is not subject to random jumps and user foibles.
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
On Thu, Jul 30, 2020 at 07:21:54PM -0600, David Ahern wrote: > On 7/30/20 4:14 PM, pet...@infradead.org wrote: > > On Thu, Jul 30, 2020 at 11:39:44PM +0200, Jiri Olsa wrote: > > > >> The patchset is adding the ability to display TOD/wallclock timestamp > >> in 'perf script' output and in 'perf data convert --to-ctf' subcommand, > >> so the converted CTF data contain TOD/wallclock timestamps. > > > > But why? Wallclock is a horrible piece of crap. Why would you want to do > > this? > > > > Same reason I brought this up 9+ years ago: userspace lives on > time-of-day, and troubleshooting is based on correlating timestamps from > multiple sources. To correlate a perf event to syslog or an application > log, we need time-of-day. You need a sync'ed CLOCK_MONOTONIC for that, CLOCK_REALTIME 'aka' wallclock is a trainwreck, you don't ever want that.
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
> yep, we have a customer that needs to compare data from multiple servers It's also needed to correlate over different guests on the same machine. This is an important use case. It would be nice if we could find a way to export the per guest TSC offset from KVM, then we could use the more reliable TSC for this, at least as long as no TSC scaling is used. But failing that wall clock is probably the best you can do. -Andi
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
On Thu, Jul 30, 2020 at 07:21:54PM -0600, David Ahern wrote: > On 7/30/20 4:14 PM, pet...@infradead.org wrote: > > On Thu, Jul 30, 2020 at 11:39:44PM +0200, Jiri Olsa wrote: > > > >> The patchset is adding the ability to display TOD/wallclock timestamp > >> in 'perf script' output and in 'perf data convert --to-ctf' subcommand, > >> so the converted CTF data contain TOD/wallclock timestamps. > > > > But why? Wallclock is a horrible piece of crap. Why would you want to do > > this? > > > > Same reason I brought this up 9+ years ago: userspace lives on > time-of-day, and troubleshooting is based on correlating timestamps from > multiple sources. To correlate a perf event to syslog or an application > log, we need time-of-day. yep, we have a customer that needs to compare data from multiple servers jirka
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
On 7/30/20 4:14 PM, pet...@infradead.org wrote: > On Thu, Jul 30, 2020 at 11:39:44PM +0200, Jiri Olsa wrote: > >> The patchset is adding the ability to display TOD/wallclock timestamp >> in 'perf script' output and in 'perf data convert --to-ctf' subcommand, >> so the converted CTF data contain TOD/wallclock timestamps. > > But why? Wallclock is a horrible piece of crap. Why would you want to do > this? > Same reason I brought this up 9+ years ago: userspace lives on time-of-day, and troubleshooting is based on correlating timestamps from multiple sources. To correlate a perf event to syslog or an application log, we need time-of-day.
Re: [PATCH 0/6] perf tools: Add wallclock time conversion support
On Thu, Jul 30, 2020 at 11:39:44PM +0200, Jiri Olsa wrote: > The patchset is adding the ability to display TOD/wallclock timestamp > in 'perf script' output and in 'perf data convert --to-ctf' subcommand, > so the converted CTF data contain TOD/wallclock timestamps. But why? Wallclock is a horrible piece of crap. Why would you want to do this?
[PATCH 0/6] perf tools: Add wallclock time conversion support
hi, this patchset is based on changes made by David Ahern long time ago. The perf code moved a lot since then, but the idea is the same. The patchset is adding the ability to display TOD/wallclock timestamp in 'perf script' output and in 'perf data convert --to-ctf' subcommand, so the converted CTF data contain TOD/wallclock timestamps. It's done by adding new header FEATURE 'CLOCK_DATA' to perf.data, that stores reference times for both TOD/wallclock time and for the clock that perf record is configured to use. These reference times are then used to convert sample's timestamps to TOD/wallclock timestamps. The feature is available only for recording with clockid specified, because it's the only case where we can get reference time to TOD/wallclock time. We can't do that with perf clock yet. Also available in here: git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git perf/abs_time thanks, jirka --- Jiri Olsa (6): perf tools: Add clockid_name function perf tools: Store clock references for -k/--clockid option perf tools: Move clockid_res_ns under clock struct perf tools: Add support to store time of day in CTF data conversion perf script: Change enum perf_output_field values to be 64 bits perf script: Add tod field to display time of day tools/perf/Documentation/perf-data.txt | 3 ++ tools/perf/Documentation/perf.data-file-format.txt | 13 +++ tools/perf/builtin-data.c | 1 + tools/perf/builtin-record.c| 58 -- tools/perf/builtin-script.c| 195 ++-- tools/perf/util/data-convert-bt.c | 56 ++--- tools/perf/util/data-convert.h | 1 + tools/perf/util/env.h | 14 +++- tools/perf/util/header.c | 120 ++--- tools/perf/util/header.h | 1 + tools/perf/util/util.h | 2 ++ 11 files changed, 371 insertions(+), 93 deletions(-)
