Quanlong Huang created IMPALA-12460:
---------------------------------------
Summary: Add lag and histogram of event processing in the log
Key: IMPALA-12460
URL: https://issues.apache.org/jira/browse/IMPALA-12460
Project: IMPALA
Issue Type: Improvement
Reporter: Quanlong Huang
Assignee: Quanlong Huang
At the end of processing each event batch, we log the processing time of that
batch and the latest event info inside HMS, e.g.
{code:java}
I0923 04:02:05.530905 12886 MetastoreEventsProcessor.java:859] Time elapsed in
processing event batch: 28m1s
I0923 04:02:05.539295 12886 MetastoreEventsProcessor.java:730] Latest event in
HMS: id=56918528, time=1695459725{code}
We should also log the lag of the event processing which is (latestEventTime -
lastSyncedEventTime). For each event batch that is slow to synced, we should
also print a histogram to show what are the top10 expensive events to sync, and
what are the top10 tables that take longest time to sync. So admins can decide
which tables to disable event processing on them.
CC [~hemanth619] , [~csringhofer] , [~wzhou] , [~kdeschle]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
