[
https://issues.apache.org/jira/browse/IMPALA-12460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768330#comment-17768330
]
Quanlong Huang commented on IMPALA-12460:
-----------------------------------------
We should also add some event info to the thread annotation. So in the jstack,
we know what event it is processing.
> Add lag and histogram of event processing in the log
> ----------------------------------------------------
>
> Key: IMPALA-12460
> URL: https://issues.apache.org/jira/browse/IMPALA-12460
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> At the end of processing each event batch, we log the processing time of that
> batch and the latest event info inside HMS, e.g.
> {code:java}
> I0923 04:02:05.530905 12886 MetastoreEventsProcessor.java:859] Time elapsed
> in processing event batch: 28m1s
> I0923 04:02:05.539295 12886 MetastoreEventsProcessor.java:730] Latest event
> in HMS: id=56918528, time=1695459725{code}
> We should also log the lag of the event processing which is (latestEventTime
> - lastSyncedEventTime). For each event batch that is slow to synced, we
> should also print a histogram to show what are the top10 expensive events to
> sync, and what are the top10 tables that take longest time to sync. So admins
> can decide which tables to disable event processing on them.
> CC [~hemanth619] , [~csringhofer] , [~wzhou] , [~kdeschle]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
