[ 
https://issues.apache.org/jira/browse/YARN-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225194#comment-14225194
 ] 

Varun Saxena commented on YARN-2748:
------------------------------------

[~zjshen], kindly review this initial patch.

Few points :
* Sub-folders may contain log files which have the same name as parent folder. 
*# We write file name while aggregating logs. To differentiate log files which 
may have same file names(due to subfolders), I think we can write file path 
relative to container log directory  instead. Your views on this. Do you 
envisage any problem elsewhere because of this ?
{code:title=AggregatedLogFormat.java|borderStyle=solid}
public void write(DataOutputStream out, Set<File> pendingUploadFiles)
        throws IOException {
      ...
      for (File logFile : fileList) {
        ...
        // Write the logFile Type
        out.writeUTF(logFile.getName());
        ....
      }
    }
{code}
*# In case of file meta data used for differentiating between log files which 
have already been aggregated and which are yet to be, I think file modification 
time may avoid issues which may arise due to clash of names. We can probably 
include subfolder name here as well. Your views on this ?
{code:title=AggregatedLogFormat.java|borderStyle=solid}
private String getLogFileMetaData(File file) {
      return containerId.toString() + "_" + file.getName() + "_"
          + file.lastModified();
    }
{code}

* I have checked for subfolders in both getPendingLogFilesToUpload and write 
methods. Although NM isnt calling the write method directly, this method still 
has public visibility. Hence, a client application may call it.

> Upload logs in the sub-folders under the local log dir when aggregating logs
> ----------------------------------------------------------------------------
>
>                 Key: YARN-2748
>                 URL: https://issues.apache.org/jira/browse/YARN-2748
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation
>    Affects Versions: 2.6.0
>            Reporter: Zhijie Shen
>            Assignee: Varun Saxena
>         Attachments: YARN-2748.001.patch
>
>
> YARN-2734 has a temporal fix to skip sub folders to avoid exception. Ideally, 
> if the app is creating a sub folder and putting its rolling logs there, we 
> need to upload these logs as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to