belugabehr commented on a change in pull request #1710:
URL: https://github.com/apache/hive/pull/1710#discussion_r540981739
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -10800,53 +10801,89 @@ public void addNotificationEvent(NotificationEvent
entry) throws MetaException {
@Override
public void cleanNotificationEvents(int olderThan) {
- boolean commited = false;
- Query query = null;
+ final int eventBatchSize = MetastoreConf.getIntVar(conf,
MetastoreConf.ConfVars.EVENT_CLEAN_MAX_EVENTS);
+
+ final long ageSec = olderThan;
+ final Instant now = Instant.now();
+
+ final int tooOld = Math.toIntExact(now.getEpochSecond() - ageSec);
+
+ final Optional<Integer> batchSize = (eventBatchSize > 0) ?
Optional.of(eventBatchSize) : Optional.empty();
+
+ final long start = System.nanoTime();
+ int deleteCount = doCleanNotificationEvents(tooOld, batchSize);
+
+ if (deleteCount == 0) {
+ LOG.info("No Notification events found to be cleaned with eventTime <
{}", tooOld);
+ } else {
+ int batchCount = 0;
+ do {
+ batchCount = doCleanNotificationEvents(tooOld, batchSize);
+ deleteCount += batchCount;
+ } while (batchCount > 0);
+ }
+
+ final long finish = System.nanoTime();
+
+ LOG.info("Deleted {} notification events older than epoch:{} in {}ms",
deleteCount, tooOld,
+ TimeUnit.NANOSECONDS.toMillis(finish - start));
+ }
+
+ private int doCleanNotificationEvents(final int ageSec, final
Optional<Integer> batchSize) {
+ final Transaction tx = pm.currentTransaction();
+ int eventsCount = 0;
+
try {
- openTransaction();
- long tmp = System.currentTimeMillis() / 1000 - olderThan;
- int tooOld = (tmp > Integer.MAX_VALUE) ? 0 : (int) tmp;
- query = pm.newQuery(MNotificationLog.class, "eventTime < tooOld");
- query.declareParameters("java.lang.Integer tooOld");
+ tx.begin();
- int max_events = MetastoreConf.getIntVar(conf,
MetastoreConf.ConfVars.EVENT_CLEAN_MAX_EVENTS);
- max_events = max_events > 0 ? max_events : Integer.MAX_VALUE;
- query.setRange(0, max_events);
- query.setOrdering("eventId ascending");
+ try (Query query = pm.newQuery(MNotificationLog.class, "eventTime <
tooOld")) {
+ query.declareParameters("java.lang.Integer tooOld");
+ query.setOrdering("eventId ascending");
+ if (batchSize.isPresent()) {
+ query.setRange(0, batchSize.get());
+ }
- List<MNotificationLog> toBeRemoved = (List) query.execute(tooOld);
- int iteration = 0;
- int eventCount = 0;
- long minEventId = 0;
- long minEventTime = 0;
- long maxEventId = 0;
- long maxEventTime = 0;
- while (CollectionUtils.isNotEmpty(toBeRemoved)) {
- int listSize = toBeRemoved.size();
- if (iteration == 0) {
- MNotificationLog firstNotification = toBeRemoved.get(0);
- minEventId = firstNotification.getEventId();
- minEventTime = firstNotification.getEventTime();
+ List<MNotificationLog> events = (List) query.execute(ageSec);
+ if (CollectionUtils.isNotEmpty(events)) {
+ eventsCount = events.size();
+
+ if (LOG.isDebugEnabled()) {
+ int minEventTime, maxEventTime;
+ long minEventId, maxEventId;
+ Iterator<MNotificationLog> iter = events.iterator();
+ MNotificationLog firstNotification = iter.next();
+
+ minEventTime = maxEventTime = firstNotification.getEventTime();
+ minEventId = maxEventId = firstNotification.getEventId();
+
+ while (iter.hasNext()) {
+ MNotificationLog notification = iter.next();
Review comment:
Hey, so this goes back to the discussion I had over here... PR #1728
There is really no guarantee that the eventTime is always increasing. So,
the ID is always increasing, but if the min/max eventTime are to be discovered
for each batch, then it needs to iterate. I'm happy to remove the logging and
remove the iteration, but that was already in place and I didn't want to remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
