[
https://issues.apache.org/jira/browse/HIVE-26225?focusedWorklogId=770687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-770687
]
ASF GitHub Bot logged work on HIVE-26225:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 16/May/22 07:48
Start Date: 16/May/22 07:48
Worklog Time Spent: 10m
Work Description: hmangla98 commented on code in PR #3282:
URL: https://github.com/apache/hive/pull/3282#discussion_r873422196
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -11766,71 +11720,77 @@ public void cleanNotificationEvents(int olderThan) {
final Optional<Integer> batchSize = (eventBatchSize > 0) ?
Optional.of(eventBatchSize) : Optional.empty();
final long start = System.nanoTime();
- int deleteCount = doCleanNotificationEvents(tooOld, batchSize);
+ int deleteCount = doCleanOlderEvents(tooOld, batchSize, table, tableName);
if (deleteCount == 0) {
- LOG.info("No Notification events found to be cleaned with eventTime <
{}", tooOld);
+ LOG.info("No {} events found to be cleaned with eventTime < {}",
tableName, tooOld);
} else {
int batchCount = 0;
do {
- batchCount = doCleanNotificationEvents(tooOld, batchSize);
+ batchCount = doCleanOlderEvents(tooOld, batchSize, table, tableName);
deleteCount += batchCount;
} while (batchCount > 0);
}
final long finish = System.nanoTime();
- LOG.info("Deleted {} notification events older than epoch:{} in {}ms",
deleteCount, tooOld,
- TimeUnit.NANOSECONDS.toMillis(finish - start));
+ LOG.info("Deleted {} {} events older than epoch:{} in {}ms", deleteCount,
tableName, tooOld,
+ TimeUnit.NANOSECONDS.toMillis(finish - start));
}
- private int doCleanNotificationEvents(final int ageSec, final
Optional<Integer> batchSize) {
+ private <T> int doCleanOlderEvents(final int ageSec, final Optional<Integer>
batchSize, Class<T> tableClass, String tableName) {
final Transaction tx = pm.currentTransaction();
int eventsCount = 0;
try {
+ String key = null;
tx.begin();
- try (Query query = pm.newQuery(MNotificationLog.class, "eventTime <=
tooOld")) {
+ try (Query query = pm.newQuery(tableClass, "eventTime <= tooOld")) {
query.declareParameters("java.lang.Integer tooOld");
- query.setOrdering("eventId ascending");
+ if (MNotificationLog.class.equals(tableClass)) {
+ key = "eventId";
+ } else if (MTxnWriteNotificationLog.class.equals(tableClass)) {
+ key = "txnId";
+ }
+ query.setOrdering(key + " ascending");
if (batchSize.isPresent()) {
query.setRange(0, batchSize.get());
}
- List<MNotificationLog> events = (List) query.execute(ageSec);
+ List<T> events = (List) query.execute(ageSec);
if (CollectionUtils.isNotEmpty(events)) {
eventsCount = events.size();
-
- if (LOG.isDebugEnabled()) {
- int minEventTime, maxEventTime;
- long minEventId, maxEventId;
- Iterator<MNotificationLog> iter = events.iterator();
- MNotificationLog firstNotification = iter.next();
-
- minEventTime = maxEventTime = firstNotification.getEventTime();
- minEventId = maxEventId = firstNotification.getEventId();
-
- while (iter.hasNext()) {
- MNotificationLog notification = iter.next();
- minEventTime = Math.min(minEventTime,
notification.getEventTime());
- maxEventTime = Math.max(maxEventTime,
notification.getEventTime());
- minEventId = Math.min(minEventId, notification.getEventId());
- maxEventId = Math.max(maxEventId, notification.getEventId());
- }
-
- LOG.debug(
- "Remove notification batch of {} events with eventTime < {},
min eventId {}, max eventId {}, min eventTime {}, max eventTime {}",
- eventsCount, ageSec, minEventId, maxEventId, minEventTime,
maxEventTime);
+ int minEventTime, maxEventTime;
+ long minEventId, maxEventId;
+ T firstNotification = events.get(0);
+ T lastNotification = events.get(eventsCount - 1);
+ if (MNotificationLog.class.equals(tableClass)) {
+ minEventTime =
((MNotificationLog)firstNotification).getEventTime();
+ minEventId = ((MNotificationLog)firstNotification).getEventId();
+ maxEventTime = ((MNotificationLog)lastNotification).getEventTime();
+ maxEventId = ((MNotificationLog)lastNotification).getEventId();
Review Comment:
Since, query output will be sorted wrt txnId or EventId. So, min/maxEventId
can be found directly by looking at first and last entries in the list without
iterating over whole list.
Issue Time Tracking
-------------------
Worklog Id: (was: 770687)
Time Spent: 0.5h (was: 20m)
> Delete operations in ObjectStore.cleanWriteNotificationEvents should be
> performed in different transactions
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-26225
> URL: https://issues.apache.org/jira/browse/HIVE-26225
> Project: Hive
> Issue Type: Improvement
> Reporter: Haymant Mangla
> Assignee: Haymant Mangla
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We need to improve the ObjectStore.cleanWriteNotificationEvents in the same
> way as it was done for notification log table in:
> https://issues.apache.org/jira/browse/HIVE-24432
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
