Adam Antal created YARN-9519: -------------------------------- Summary: TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config Key: YARN-9519 URL: https://issues.apache.org/jira/browse/YARN-9519 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 3.2.0 Reporter: Adam Antal
The TFile log aggregation file format is not sensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config. In {{LogAggregationTFileController$initInternal}}: {code:java} this.remoteRootLogDir = new Path( conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR)); {code} So the remoteRootLogDir is only aware of the yarn.nodemanager.remote-app-log-dir config, while other file format, like IFile defaults to the file format config, so its priority is bigger. >From {{LogAggregationIndexedFileController$initInternal}}: {code:java} String remoteDirStr = String.format( YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, this.fileControllerName); String remoteDir = conf.get(remoteDirStr); if (remoteDir == null || remoteDir.isEmpty()) { remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); } {code} (Where these configs are: ) {code:java} public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; public static final String NM_REMOTE_APP_LOG_DIR = NM_PREFIX + "remote-app-log-dir"; {code} I suggest TFile should try to obtain the remote dir config from yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not specified falls back to the yarn.nodemanager.remote-app-log-dir config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org