> On Oct. 6, 2017, 12:26 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 173-174 (patched)
> > <https://reviews.apache.org/r/62459/diff/10/?file=1845962#file1845962line173>
> >
> > ````LogAggregationUtils```` is marked ````@Private```` so we shouldn't
> > be using it. Hadoop can change things incompatibly here.
What do you think about borrowing/inlining/copying those functions from Hadoop
2.6 too? I was thinking about it and it is probably better than bringing back
the initial ``ExecutorService`` based approach. I am thinking of a class like:
```java
// TODO: once OOZIE-2983 ("Stream the Launcher AM Logs") is done, remove it.
public class OozieLauncherLogFetcher {
private static final String TMP_FILE_SUFFIX = ".tmp";
final private Configuration hadoopConfig;
public OozieLauncherLogFetcher(final Configuration hadoopConfig) {
this.hadoopConfig = hadoopConfig;
}
// Borrowed code from org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
private static void logDirNotExist(String remoteAppLogDir) {
System.out.println(remoteAppLogDir + "does not exist.");
System.out.println("Log aggregation has not completed or is not
enabled.");
}
// Borrowed code from org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
private static void emptyLogDir(String remoteAppLogDir) {
System.out.println(remoteAppLogDir + "does not have any log files.");
}
// Borrowed code from
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static String getRemoteNodeLogDirSuffix(Configuration conf) {
return conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX,
YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR_SUFFIX);
}
// Borrowed code from
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static Path getRemoteLogSuffixedDir(Path remoteRootLogDir, String
user, String suffix) {
return suffix != null && !suffix.isEmpty() ? new
Path(getRemoteLogUserDir(remoteRootLogDir, user), suffix) :
getRemoteLogUserDir(remoteRootLogDir, user);
}
// Borrowed code from
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static Path getRemoteLogUserDir(Path remoteRootLogDir, String user) {
return new Path(remoteRootLogDir, user);
}
// Borrowed code from
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static Path getRemoteAppLogDir(Path remoteRootLogDir, ApplicationId
appId, String user, String suffix) {
return new Path(getRemoteLogSuffixedDir(remoteRootLogDir, user,
suffix), appId.toString());
}
// Borrowed code from org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
public int dumpAllContainersLogs(ApplicationId appId, String appOwner,
PrintStream out) throws IOException {
Path remoteRootLogDir = new
Path(hadoopConfig.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
String logDirSuffix = getRemoteNodeLogDirSuffix(hadoopConfig);
Path remoteAppLogDir = getRemoteAppLogDir(remoteRootLogDir, appId,
appOwner, logDirSuffix);
RemoteIterator nodeFiles;
try {
Path qualifiedLogDir =
FileContext.getFileContext(hadoopConfig).makeQualified(remoteAppLogDir);
nodeFiles = FileContext.getFileContext(qualifiedLogDir.toUri(),
hadoopConfig).listStatus(remoteAppLogDir);
} catch (FileNotFoundException fileNotFoundException) {
logDirNotExist(remoteAppLogDir.toString());
return -1;
}
boolean foundAnyLogs = false;
while(true) {
FileStatus thisNodeFile;
do {
if (!nodeFiles.hasNext()) {
if (!foundAnyLogs) {
emptyLogDir(remoteAppLogDir.toString());
return -1;
}
return 0;
}
thisNodeFile = (FileStatus)nodeFiles.next();
} while(thisNodeFile.getPath().getName().endsWith(TMP_FILE_SUFFIX));
AggregatedLogFormat.LogReader reader = new
AggregatedLogFormat.LogReader(hadoopConfig, thisNodeFile.getPath());
try {
AggregatedLogFormat.LogKey key = new
AggregatedLogFormat.LogKey();
DataInputStream valueStream = reader.next(key);
while(valueStream != null) {
String containerString = "\n\nContainer: " + key + " on " +
thisNodeFile.getPath().getName();
out.println(containerString);
out.println(StringUtils.repeat("=",
containerString.length()));
while(true) {
try {
AggregatedLogFormat.LogReader.readAContainerLogsForALogType(valueStream, out,
thisNodeFile.getModificationTime());
foundAnyLogs = true;
} catch (EOFException eofException) {
key = new AggregatedLogFormat.LogKey();
valueStream = reader.next(key);
break;
}
}
}
} finally {
reader.close();
}
}
}
}
```
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review187223
-----------------------------------------------------------
On Oct. 4, 2017, 2:18 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> -----------------------------------------------------------
>
> (Updated Oct. 4, 2017, 2:18 p.m.)
>
>
> Review request for oozie.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> A diagnostic tool that collects a bunch of job and other information from
> Oozie in a zip file.
>
>
> Diffs
> -----
>
> docs/src/site/twiki/DG_CommandLineTool.twiki
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa
> pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c
> tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508
> tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION
>
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
> PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java
> PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java
> PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java
> PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
> PRE-CREATION
> tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java
> PRE-CREATION
> tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java
> PRE-CREATION
> tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java
> PRE-CREATION
> tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
> PRE-CREATION
>
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java
> PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/62459/diff/10/
>
>
> Testing
> -------
>
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows,
> and executed the following commands:
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh -numworkflows 2000 -oozie
> http://localhost:11000/oozie -output /tmp``,
> -- ``bin/oozie-diag-bundle-collector.sh -jobs
> 0000001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie
> -output .`` (verified zip the tool generated).
>
>
> Thanks,
>
> Attila Sasvari
>
>
