[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[
https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357642#comment-15357642
]
Colin Patrick McCabe commented on HTRACE-376:
-
Hi [~adriancole],
Very interesting discussion as always!
This brings to mind the famous distinction between accuracy and precision that
tends to come up in statistics. We might be able to calculate the expected
value of some random variable to 5 digits right of the decimal point, but if
there is a wide variance in the probability distribution, we can say that our
calculated expected value isn't very precise.
I agree that our software's time measurements certainly becomes imprecise when
we get down to nanoseconds. And displaying that value with high accuracy could
be misleading. However, I think nanoseconds is still the way to go, for a few
reasons:
* Some systems are more precise than others. For example, some PC systems are
configured to use the high-resolution timer (the HPET hardware) rather than the
TSC hardware. Those systems will have more accuracy. If we artificially limit
accuracy, we won't serve the more precise systems well. We don't really know
what the limits of accuracy are, and we especially don't know what the accuracy
limits of future systems might be.
* A lot of times, people want to know about durations rather than absolute
times. For example, [~pranavanc5e] is probably hoping to get some insight into
things happening on a single node in Phoenix. For this purpose, it doesn't
matter if the call to get the time has a fixed overhead. Since both the start
and end time are shifted forward, the duration will still be in the ballpark.
* People should be collecting a lot of trace data which should allow them to
look at the variance in different values for the same span. People should also
be exercising common sense and not reading too much into the duration
difference between a 500 ns span and a 501 ns span.
* People are not used to thinking in terms of microseconds because APIs tend to
be in milliseconds or nanoseconds.
bq. Moreover, eventhough unix can report time in microseconds using the
presumably cheaper gettimeofday, it cannot in nanos.
Hmm. I haven't seen anything suggesting {{gettimeofday}} is cheaper than
{{clock_gettime}}. I thought they were both implemented via the VDSO hack on
Linux, so their performance should be the same? At least on my system, the man
page for {{gettimeofday}} says that's obsolete and {{clock_gettime}} should be
used.
> HTrace should support nanosecond time granularities
> ---
>
> Key: HTRACE-376
> URL: https://issues.apache.org/jira/browse/HTRACE-376
> Project: HTrace
> Issue Type: Improvement
> Components: core
>Affects Versions: 3.1.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HTRACE-376.001.patch, HTRACE-376.002.patch,
> HTRACE-376.003.patch
>
>
> HTrace should support nanosecond time granularities. Currently only a
> granularity of milliseconds is supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356360#comment-15356360 ] Adrian Cole commented on HTRACE-376: PS in case it helps, I just recently implemented something like this in another library that previously used millis. https://github.com/spring-cloud/spring-cloud-sleuth/issues/312 Here's a footnote that may or may not be distracting from that: I'd suggest not advertising duration in nanoseconds because we cannot accurately measure at that granularity due to the overhead and change frequency of System.nanoTime(). For example, if calling nanoTime has to happen twice, and we should expect 30nanos overhead per call, we can't reliably report duration in orders of nanos. Moreover, eventhough unix can report time in microseconds using the presumably cheaper gettimeofday, it cannot in nanos. In short, micros ends up a pretty decent choice for various reasons even if we didn't consider zipkin which uses micros consistently for everything. > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HTRACE-376.001.patch, HTRACE-376.002.patch, > HTRACE-376.003.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349841#comment-15349841 ] Colin Patrick McCabe commented on HTRACE-376: - Thanks [~pranavanc5e]. And [~fabbri], [~saint@gmail.com], [~abec], take a look at my patch if you get a chance. We talked about doing nanotime in the past-- I think this is a great way to do it > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HTRACE-376.001.patch, HTRACE-376.002.patch, > HTRACE-376.003.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344308#comment-15344308 ] Pranavan commented on HTRACE-376: - Thanks for all the background information. Great work. I will have a look again > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HTRACE-376.001.patch, HTRACE-376.002.patch, > HTRACE-376.003.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[
https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342898#comment-15342898
]
Colin Patrick McCabe commented on HTRACE-376:
-
So, there's a bunch of fairly complicated issues here. I posted a patch which
I think addresses them all.
First of all, like I said, we can't "cheat" and use times that are only
meaningful on a single node. If node1 says an event happened at time 1, and
node2 says another event happened at time 1000, we should be able to assume
that there were 999 time units in between them. This rules out using
{{System.nanoTime}} directly, as I've mentioned several times. It is a
system-local time, and we need a global time. Unfortunately, Java doesn't
offer any API to get a wall-clock time with nanosecond granularity.
There will be better time APIs coming out in future JVM versions. Until then,
as a workaround, we can try to correlate the monotonic clock with the wall
clock, and get a wall-clock nanosecond estimate out of that. The code I posted
uses {{NanoTimeTimeSource}} to do that.
Secondly, there shouldn't be any need to store times in both milliseconds and
nanoseconds. If we put nanoseconds since the epoch into a long, we still have
(2^63) / (10^9 * 60 * 24 * 365) = 17548 years of post-1970 time which we can
cover. I'm OK with the HTrace project lasting for "only" 17,000 years.
Overwriting the time in milliseconds should also set the time in nanoseconds,
and vice versa.
We need to change the Span JSON to support nanoseconds. I have done this by
adding the 'E' and 'B' (i.e. capital letter) fields in addition to the 'b' and
'e' fields. Also, we need nanosecond granularity for span time annotations as
well as spans.
Finally, we need some way of choosing to use millisecond granularity versus
nanosecond granularity. The code does this by looking at the configuration key
{{time.granularity}}, which can be either {{ns}} for nanoseconds or {{ms} for
milliseconds. This determines which {{TimeSource}} we use when creating spans.
bq. Nano time call is pretty expensive because it will take like 100+ cpu clock
cycles whereas System.currentMillis() will just take 5 or 6 cpu clock cycles.
Nanotime is an expensive operation. (In my opinion calling nanotime is a
barrier for performance)
That is true only on Windows, not on Linux. There are some relevant answers on
Stack Overflow you could look at. In any case, this will be configurable and
the default will be milliseconds.
> HTrace should support nanosecond time granularities
> ---
>
> Key: HTRACE-376
> URL: https://issues.apache.org/jira/browse/HTRACE-376
> Project: HTrace
> Issue Type: Improvement
> Components: core
>Affects Versions: 3.1.0
>Reporter: Colin Patrick McCabe
>Assignee: Pranavan
> Attachments: HTRACE-376.001.patch, HTRACE-376.002.patch,
> HTRACE-376.003.patch
>
>
> HTrace should support nanosecond time granularities. Currently only a
> granularity of milliseconds is supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338511#comment-15338511 ] Pranavan commented on HTRACE-376: - Hi Colin, I have attached the patch file. Please review it, I will squash all the commits > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > Attachments: HTRACE-376.001.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338293#comment-15338293 ] Colin Patrick McCabe commented on HTRACE-376: - Hi [~pranavanc5e], Thanks for working on this. Can you please combine these changes into a single patch that I can review? (Assuming you want to make the changes as a single commit rather than multiple). Unfortunately, it is not going to be possible for people to follow the discussion here on JIRA if the change history gets erased each time you re-push the branch. That's one reason I still prefer attaching numbered patch files. It's easy and we know exactly what is being reviewed. If you want to keep using github only, then kindly provide links to specific commits so that people can understand what patch each set of comments is referencing. Thanks! > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > Attachments: HTRACE-376.001.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337495#comment-15337495 ] Pranavan commented on HTRACE-376: - Hi Colin. I have updated the PR. I did not change the API method. I added new methods. But I need to add some changes inside MilliSpan. I added the changes to SpanSerializer. Update the thread if you have any issues. > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > Attachments: HTRACE-376.001.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334334#comment-15334334 ] Pranavan commented on HTRACE-376: - Thanks Collin for the update. I will look at this > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > Attachments: HTRACE-376.001.patch > > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[
https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334321#comment-15334321
]
Colin Patrick McCabe commented on HTRACE-376:
-
Thanks for looking at this.
{code}
+ begin(System.currentTimeMillis(),System.nanoTime()).
{code}
If you read the documentation of {{System#nanoTime}} here
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime() ,
it says that "This method can only be used to measure elapsed time and is not
related to any other notion of system or wall-clock time." Since HTrace uses
wall-clock time in its span times, nanotime is not suitable for us. We also do
not want to be fetching the time twice each time we create a span. There is a
cost associated with this. Once is enough.
{code}
-public Builder begin(long begin) {
+public Builder begin(long begin, long beginNano) {
{code}
{{MilliSpan#Builder#begin}} is a public function which is part of the API. It
can't be changed in the htrace 4.x line. You will need to add a new function
to preserve backwards compatibility.
We also need to add the nanoTimes to the Span JSON, both on the JSON parsing
and JSON emitting side.
> HTrace should support nanosecond time granularities
> ---
>
> Key: HTRACE-376
> URL: https://issues.apache.org/jira/browse/HTRACE-376
> Project: HTrace
> Issue Type: Improvement
> Components: core
>Affects Versions: 3.1.0
>Reporter: Colin Patrick McCabe
>Assignee: Pranavan
> Attachments: HTRACE-376.001.patch
>
>
> HTrace should support nanosecond time granularities. Currently only a
> granularity of milliseconds is supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333999#comment-15333999 ] Pranavan commented on HTRACE-376: - Hi Colin PR - Please check and review https://github.com/apache/incubator-htrace/pull/10/files > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333998#comment-15333998 ] Pranavan commented on HTRACE-376: - Hi Colin PR - Please check and review https://github.com/apache/incubator-htrace/pull/10/files > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HTRACE-376) HTrace should support nanosecond time granularities
[ https://issues.apache.org/jira/browse/HTRACE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332991#comment-15332991 ] Pranavan commented on HTRACE-376: - Hi Colin We can add new field rather than changing the old time fields. That would be good, we can use whatever we want. > HTrace should support nanosecond time granularities > --- > > Key: HTRACE-376 > URL: https://issues.apache.org/jira/browse/HTRACE-376 > Project: HTrace > Issue Type: Improvement > Components: core >Affects Versions: 3.1.0 >Reporter: Colin Patrick McCabe >Assignee: Pranavan > > HTrace should support nanosecond time granularities. Currently only a > granularity of milliseconds is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
