Varun Saxena commented on YARN-2748:

[~zjshen], kindly review this initial patch.

Few points :
* Sub-folders may contain log files which have the same name as parent folder. 
*# We write file name while aggregating logs. To differentiate log files which 
may have same file names(due to subfolders), I think we can write file path 
relative to container log directory  instead. Your views on this. Do you 
envisage any problem elsewhere because of this ?
public void write(DataOutputStream out, Set<File> pendingUploadFiles)
        throws IOException {
      for (File logFile : fileList) {
        // Write the logFile Type
*# In case of file meta data used for differentiating between log files which 
have already been aggregated and which are yet to be, I think file modification 
time may avoid issues which may arise due to clash of names. We can probably 
include subfolder name here as well. Your views on this ?
private String getLogFileMetaData(File file) {
      return containerId.toString() + "_" + file.getName() + "_"
          + file.lastModified();

* I have checked for subfolders in both getPendingLogFilesToUpload and write 
methods. Although NM isnt calling the write method directly, this method still 
has public visibility. Hence, a client application may call it.

> Upload logs in the sub-folders under the local log dir when aggregating logs
> ----------------------------------------------------------------------------
>                 Key: YARN-2748
>                 URL: https://issues.apache.org/jira/browse/YARN-2748
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation
>    Affects Versions: 2.6.0
>            Reporter: Zhijie Shen
>            Assignee: Varun Saxena
>         Attachments: YARN-2748.001.patch
> YARN-2734 has a temporal fix to skip sub folders to avoid exception. Ideally, 
> if the app is creating a sub folder and putting its rolling logs there, we 
> need to upload these logs as well.

This message was sent by Atlassian JIRA

Reply via email to