[
https://issues.apache.org/jira/browse/IMPALA-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala updated IMPALA-11553:
-------------------------------------------
Description:
Currently, we have global metrics for event-processor in catalogd's web UI:
||Name||Value||Description||
|{{events-processor.avg-events-fetch-duration}}|989ms|Average time taken to
fetch a batch of metastore events|
|{{events-processor.avg-events-process-duration}}|0|Average time taken to
process a batch of events received from metastore|
|{{events-processor.events-received}}|0|Total number of metastore events
received|
|{{events-processor.events-received-15min-rate}}|0.000000|Exponentially
weighted moving average (EWMA) of number of events received in last 15 min|
|{{events-processor.events-received-1min-rate}}|0.000000|Exponentially weighted
moving average (EWMA) of number of events received in last 1 min|
|{{events-processor.events-received-5min-rate}}|0.000000|Exponentially weighted
moving average (EWMA) of number of events received in last 5 min|
|{{events-processor.events-skipped}}|0|Total number of metastore events skipped|
|{{events-processor.last-synced-event-id}}|734979|Last metastore event id that
the catalog server processed and synced to|
|{{events-processor.status}}|ACTIVE|Metastore event processor status|
Some metrics can be added for table level, e.g. {{avg-events-process-duration
(also extend it to measure the last 30min, 1h, 24h)}}
This helps users to find which tables are causing the event-processor lagging
behind. So they can disable event-processor on them as a workaround.
This Jira also tracks a metric at events-processor level
'events-consuming-delay' to gauge how much is taken by events-processor to
consume the generated events.
was:
Currently, we have global metrics for event-processor in catalogd's web UI:
||Name||Value||Description||
|{{events-processor.avg-events-fetch-duration}}|989ms|Average time taken to
fetch a batch of metastore events|
|{{events-processor.avg-events-process-duration}}|0|Average time taken to
process a batch of events received from metastore|
|{{events-processor.events-received}}|0|Total number of metastore events
received|
|{{events-processor.events-received-15min-rate}}|0.000000|Exponentially
weighted moving average (EWMA) of number of events received in last 15 min|
|{{events-processor.events-received-1min-rate}}|0.000000|Exponentially weighted
moving average (EWMA) of number of events received in last 1 min|
|{{events-processor.events-received-5min-rate}}|0.000000|Exponentially weighted
moving average (EWMA) of number of events received in last 5 min|
|{{events-processor.events-skipped}}|0|Total number of metastore events skipped|
|{{events-processor.last-synced-event-id}}|734979|Last metastore event id that
the catalog server processed and synced to|
|{{events-processor.status}}|ACTIVE|Metastore event processor status|
Some metrics can be added for table level, e.g. {{avg-events-process-duration
(also extend it to measure the last 30min, 1h, 24h)}}
This helps users to find which tables are causing the event-processor lagging
behind. So they can disable event-processor on them as a workaround.
> Add events specific metrics on table level
> ------------------------------------------
>
> Key: IMPALA-11553
> URL: https://issues.apache.org/jira/browse/IMPALA-11553
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Major
> Attachments: Events_delay.png, events_duration.png
>
>
> Currently, we have global metrics for event-processor in catalogd's web UI:
> ||Name||Value||Description||
> |{{events-processor.avg-events-fetch-duration}}|989ms|Average time taken to
> fetch a batch of metastore events|
> |{{events-processor.avg-events-process-duration}}|0|Average time taken to
> process a batch of events received from metastore|
> |{{events-processor.events-received}}|0|Total number of metastore events
> received|
> |{{events-processor.events-received-15min-rate}}|0.000000|Exponentially
> weighted moving average (EWMA) of number of events received in last 15 min|
> |{{events-processor.events-received-1min-rate}}|0.000000|Exponentially
> weighted moving average (EWMA) of number of events received in last 1 min|
> |{{events-processor.events-received-5min-rate}}|0.000000|Exponentially
> weighted moving average (EWMA) of number of events received in last 5 min|
> |{{events-processor.events-skipped}}|0|Total number of metastore events
> skipped|
> |{{events-processor.last-synced-event-id}}|734979|Last metastore event id
> that the catalog server processed and synced to|
> |{{events-processor.status}}|ACTIVE|Metastore event processor status|
> Some metrics can be added for table level, e.g. {{avg-events-process-duration
> (also extend it to measure the last 30min, 1h, 24h)}}
> This helps users to find which tables are causing the event-processor lagging
> behind. So they can disable event-processor on them as a workaround.
> This Jira also tracks a metric at events-processor level
> 'events-consuming-delay' to gauge how much is taken by events-processor to
> consume the generated events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
