aajisaka commented on code in PR #4797:
URL: https://github.com/apache/hadoop/pull/4797#discussion_r1027616899
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/loghandler/NonAggregatingLogHandler.java:
##########
@@ -149,47 +156,52 @@ private void recover() throws IOException {
@Override
public void handle(LogHandlerEvent event) {
switch (event.getType()) {
- case APPLICATION_STARTED:
- LogHandlerAppStartedEvent appStartedEvent =
- (LogHandlerAppStartedEvent) event;
- this.appOwners.put(appStartedEvent.getApplicationId(),
- appStartedEvent.getUser());
- this.dispatcher.getEventHandler().handle(
- new ApplicationEvent(appStartedEvent.getApplicationId(),
- ApplicationEventType.APPLICATION_LOG_HANDLING_INITED));
+ case APPLICATION_STARTED:
+ LogHandlerAppStartedEvent appStartedEvent =
+ (LogHandlerAppStartedEvent) event;
+ this.appOwners.put(appStartedEvent.getApplicationId(),
+ appStartedEvent.getUser());
+ this.dispatcher.getEventHandler().handle(
+ new ApplicationEvent(appStartedEvent.getApplicationId(),
+ ApplicationEventType.APPLICATION_LOG_HANDLING_INITED));
+ break;
+ case CONTAINER_FINISHED:
+ // Ignore
+ break;
+ case APPLICATION_FINISHED:
+ LogHandlerAppFinishedEvent appFinishedEvent =
+ (LogHandlerAppFinishedEvent) event;
+ ApplicationId appId = appFinishedEvent.getApplicationId();
+ String user = appOwners.remove(appId);
+ if (user == null) {
+ LOG.error("Unable to locate user for " + appId);
+ // send LOG_HANDLING_FAILED out
+ NonAggregatingLogHandler.this.dispatcher.getEventHandler().handle(
+ new ApplicationEvent(appId,
+ ApplicationEventType.APPLICATION_LOG_HANDLING_FAILED));
break;
- case CONTAINER_FINISHED:
- // Ignore
- break;
- case APPLICATION_FINISHED:
- LogHandlerAppFinishedEvent appFinishedEvent =
- (LogHandlerAppFinishedEvent) event;
- ApplicationId appId = appFinishedEvent.getApplicationId();
+ }
+ LogDeleterRunnable logDeleter = new LogDeleterRunnable(user, appId);
+ long appLogSize = calculateSizeOfAppLogs(user, appId);
+ long deletionTimestamp = System.currentTimeMillis()
+ + this.deleteDelaySeconds * 1000;
+ LogDeleterProto deleterProto = LogDeleterProto.newBuilder()
+ .setUser(user)
+ .setDeletionTime(deletionTimestamp)
+ .build();
+ try {
+ stateStore.storeLogDeleter(appId, deleterProto);
+ } catch (IOException e) {
+ LOG.error("Unable to record log deleter state", e);
+ }
+ // delete no delay if log size exceed deleteThreshold
+ if (enableTriggerDeleteBySize && appLogSize >= deleteThreshold) {
Review Comment:
Hi @leixm thank you for your update.
1. Can we calculate the size of the application log directory only if the
feature is enabled?
2. Can we use `sched.schedule(logDeleter, 0, TimeUnit.SECONDS);` to delete
the files in background?
The code will be like
```java
try {
boolean logDeleterStarted = false;
if (enableTriggerDeleteBySize) {
final long appLogSize = calculateSizeOfAppLogs(user, appId);
if (appLogSize >= threshold) {
...
sched.schedule(logDeleter, 0, TimeUnit.SECONDS);
logDeleterStarted = true;
}
}
if (!logDeleterStarted) {
sched.schedule(logDeleter, this.deleteDelaySeconds, TimeUnit.SECONDS);
}
} catch (RejectedExecutionException e) {
logDeleter.run();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
