[
https://issues.apache.org/jira/browse/IMPALA-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803300#comment-17803300
]
ASF subversion and git services commented on IMPALA-11553:
----------------------------------------------------------
Commit 324a1aa37e4abecc73e8ccd39cca75cfcc54791e in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=324a1aa37 ]
IMPALA-11553: Add event specific metrics in the table metrics
This patch adds an event specific metric "avg-events-process-duration"
at the table level metrics. This metric is also extended to last 1min,
5mins, 15mins duration. This metric is useful to identify the average
events processed duration on the table. This is helpful to identify if
a particular table is causing event procssor lagging and as a temporary
workaround, event processing can be disabled on that table.
Another metric is also added in the event processor summary page,
"events-consuming-delay-ms", is the time difference in milliseconds of
the event created in the metastore and event processed by event
processor. This is another useful metric to gauge how the event
processor is lagging.
Tests:
- Manually verified the metrics on catalogD UI page when running some
hive workloads.
Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd
Reviewed-on: http://gerrit.cloudera.org:8080/20473
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add events specific metrics on table level
> ------------------------------------------
>
> Key: IMPALA-11553
> URL: https://issues.apache.org/jira/browse/IMPALA-11553
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Major
> Attachments: Events_delay.png, events_duration.png
>
>
> Currently, we have global metrics for event-processor in catalogd's web UI:
> ||Name||Value||Description||
> |{{events-processor.avg-events-fetch-duration}}|989ms|Average time taken to
> fetch a batch of metastore events|
> |{{events-processor.avg-events-process-duration}}|0|Average time taken to
> process a batch of events received from metastore|
> |{{events-processor.events-received}}|0|Total number of metastore events
> received|
> |{{events-processor.events-received-15min-rate}}|0.000000|Exponentially
> weighted moving average (EWMA) of number of events received in last 15 min|
> |{{events-processor.events-received-1min-rate}}|0.000000|Exponentially
> weighted moving average (EWMA) of number of events received in last 1 min|
> |{{events-processor.events-received-5min-rate}}|0.000000|Exponentially
> weighted moving average (EWMA) of number of events received in last 5 min|
> |{{events-processor.events-skipped}}|0|Total number of metastore events
> skipped|
> |{{events-processor.last-synced-event-id}}|734979|Last metastore event id
> that the catalog server processed and synced to|
> |{{events-processor.status}}|ACTIVE|Metastore event processor status|
> Some metrics can be added for table level, e.g. {{avg-events-process-duration
> (also extend it to measure the last 30min, 1h, 24h)}}
> This helps users to find which tables are causing the event-processor lagging
> behind. So they can disable event-processor on them as a workaround.
> This Jira also tracks a metric at events-processor level
> 'events-consuming-delay' to gauge how much is taken by events-processor to
> consume the generated events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
