[
https://issues.apache.org/jira/browse/STORM-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350455#comment-15350455
]
ASF GitHub Bot commented on STORM-1742:
---------------------------------------
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1523
STORM-1742 More accurate 'complete latency'
PR for 1.x: #1379
* Acker computes 'complete latency' and sends back to Spout
* start time: Acker receiving ACK_INIT from Spout
* end time: Acker receiving ACK message which make the tuple tree
completed
* When Acker provides complete latency, Spout just uses this value to
record that latency
* exception case: tuple timed out - Spout doesn't receive failed
information from Acker
#1379 contains the result of functional test and performance test. Please
refer that pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-1742
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1523.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1523
----
commit 63fed3becadb418b224c323b970b65de42551c98
Author: Jungtaek Lim <[email protected]>
Date: 2016-06-27T06:01:29Z
STORM-1742 More accurate 'complete latency'
* Acker computes 'complete latency' and sends back to Spout
* start time: Acker receiving ACK_INIT from Spout
* end time: Acker receiving ACK message which make the tuple tree
completed
* When Acker provides complete latency, Spout just uses this value to
record that latency
* exception case: tuple timed out - Spout doesn't receive failed
information from Acker
----
> More accurate 'complete latency'
> --------------------------------
>
> Key: STORM-1742
> URL: https://issues.apache.org/jira/browse/STORM-1742
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
>
> I already initiated talking thread on dev@ list. Below is copy of the content
> in my mail.
> http://mail-archives.apache.org/mod_mbox/storm-dev/201604.mbox/%3CCAF5108gn=rskundfs7-sgy_pd-_prgj2hf2t5e5zppp-knd...@mail.gmail.com%3E
> While thinking about metrics improvements, I doubt how many users know that
> what 'exactly' is complete latency. In fact, it's somewhat complicated
> because additional waiting time could be added to complete latency because
> of single-thread model event loop of spout.
> Long running nextTuple() / ack() / fail() can affect complete latency but
> it's behind the scene. No latency information provided, and someone even
> didn't know about this characteristic. Moreover, calling nextTuple() could
> be skipped due to max spout waiting, which will make us harder to guess
> when avg. latency of nextTuple() will be provided.
> I think separation of threads (tuple handler to separate thread, as JStorm
> provides) would resolve the gap, but it requires our spout logic to be
> thread-safe, so I'd like to find workaround first.
> My sketched idea is let Acker decides end time for root tuple.
> There're two subsequent ways to decide start time for root tuple,
> 1. when Spout about to emit ACK_INIT to Acker (in other words, keep it as
> it is)
> - Acker sends ack / fail message to Spout with timestamp, and Spout
> calculates time delta
> - pros. : It's most accurate way since it respects the definition of
> 'complete latency'.
> - cons. : The sync of machine time between machines are very important.
> Sub-millisecond of precision would be required.
> 2. when Acker receives ACK_INIT from Spout
> - Acker calculates time delta itself, and sends ack / fail message to
> Spout with time delta
> - pros. : No requirement to sync the time between servers so strictly.
> - cons. : It doesn't contain the latency to send / receive ACK_INIT
> between Spout and Acker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
