Sudarshan Vasudevan created GOBBLIN-1087:
--------------------------------------------
Summary: Track and report histogram of observed lag from Gobblin
Kafka pipeline
Key: GOBBLIN-1087
URL: https://issues.apache.org/jira/browse/GOBBLIN-1087
Project: Apache Gobblin
Issue Type: Improvement
Components: gobblin-kafka
Affects Versions: 0.15.0
Reporter: Sudarshan Vasudevan
Assignee: Shirshanka Das
Fix For: 0.15.0
In this PR, we instrument the KafkaExtractor to track the observed latency of
Kafka consumer records processed by the pipeline. Here, observed latency is
measured as the time difference between processing time of the record and the
original creation time. The latency distribution is tracked in an HdrHistogram,
which is serialized into a string when emitted as part of a
GobblinTrackingEvent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
