[ 
https://issues.apache.org/jira/browse/YARN-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152599#comment-14152599
 ] 

Xuan Gong commented on YARN-2468:
---------------------------------

bq. Why is the test in TestAggregatedLogsBlock ignored?

We will have YARN-2583 for web UI related changes. This test will be failed 
right now. So, I add @ignored

bq. pendingUploadFiles is really not neded to be a class field. Rename 
getNumOfLogFilesToUpload() to be getPendingLogFilesToUploadForThisContainer() 
and return the set of pending files. LogValue.write() can then take Set<File> 
pendingLogFilesToUpload as one of the arguments.

I would like to check how many log files we can upload this time. If the number 
is 0, we can skip this time. And this check is also happened before 
LogKey.write(), otherwise, we will write key, but without value.

bq. If deletion of previously uploaded file takes a while and the file remains 
by the time of the next cycle, we will upload it again? It seems to be, let's 
validate this via a test-case.

No, it will not. That is why I saved many information, such as 
allExistingFiles, alreadyUploadedFiles and etc. We will those to check whether 
the logs have been uploaded before.

bq. testLogAggregationServiceWithInterval: doLogAggregationOutOfBand + 
Thread.sleep() is unreliable. Use a clock and refactor AppLogAggregatorImpl to 
have the cyclic aggregation directly callable via a method.

The Thread.sleep() is not used to trigger the logAggregation. It is used to 
make sure the logs has been uploaded into the remote directory. But, deleted 
those Thread.sleep() from the testcases.

> Log handling for LRS
> --------------------
>
>                 Key: YARN-2468
>                 URL: https://issues.apache.org/jira/browse/YARN-2468
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-2468.1.patch, YARN-2468.2.patch, YARN-2468.3.patch, 
> YARN-2468.3.rebase.2.patch, YARN-2468.3.rebase.patch, YARN-2468.4.1.patch, 
> YARN-2468.4.patch, YARN-2468.5.1.patch, YARN-2468.5.1.patch, 
> YARN-2468.5.2.patch, YARN-2468.5.3.patch, YARN-2468.5.4.patch, 
> YARN-2468.5.patch, YARN-2468.6.1.patch, YARN-2468.6.patch, 
> YARN-2468.7.1.patch, YARN-2468.7.patch, YARN-2468.8.patch, YARN-2468.9.patch
>
>
> Currently, when application is finished, NM will start to do the log 
> aggregation. But for Long running service applications, this is not ideal. 
> The problems we have are:
> 1) LRS applications are expected to run for a long time (weeks, months).
> 2) Currently, all the container logs (from one NM) will be written into a 
> single file. The files could become larger and larger.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to