> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote: > > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java > > Lines 97-98 (patched) > > <https://reviews.apache.org/r/62459/diff/7/?file=1837523#file1837523line97> > > > > Something to consider for all output of the tool, not just here: we're > > outputting most of the info in a human-readable format. Should we think > > about using a machine-readable format? Or maybe having the option for one? > > Or doing both? The idea being that someone would then be able to write > > their own tool that could analyze stuff. We already have some code > > somewhere that converts a WorkflowJob into JSON, so it shouldn't be a lot > > of work to add this either. That might also be a good idea from a > > compatibility perspective - i.e. what's the compatibility story on this > > out? If there's a new field, what do we do?
It is a good idea, and I would create a separate JIRA to discuss/design and implement it. OOZIE-3074 > On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote: > > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java > > Lines 163 (patched) > > <https://reviews.apache.org/r/62459/diff/7/?file=1837523#file1837523line163> > > > > I think the JHS may also be required, in the cases where the RM has > > forgotten about the job. > > > > And what about HDFS? That's required too. > > > > I'm thinking we might be best off not doing these checks. It's too > > complicated (CM spent a lot of effort on this) and we can't check for > > everything (e.g. what if log aggregation is turned off?). Besides, we're > > already handling exceptions below when trying to get the logs - if the RM, > > JHS, HDFS, etc isn't working, the call will fail anyway. I agree with it, but failing fast would be a better experience than a 30 seconds timeout. The default retry policy of YarnClient resulted in a lot of retries and I could not find the proper parameter to control it (I want to restrict retries to a few seconds instead of minutes). > On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote: > > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java > > Lines 185 (patched) > > <https://reviews.apache.org/r/62459/diff/7/?file=1837523#file1837523line185> > > > > Please create a followup JIRA to change this in the future to use > > OOZIE-2983 ("Stream the Launcher AM Logs") once it's done. This will also > > be nice in that we can get rid of the RM up check. I will do so. > On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote: > > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java > > Lines 191 (patched) > > <https://reviews.apache.org/r/62459/diff/7/?file=1837523#file1837523line191> > > > > Is there not a cleaner way to do this than using a CLI like this? I could not find one, please let me know if you have something in mind. > On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote: > > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java > > Lines 221 (patched) > > <https://reviews.apache.org/r/62459/diff/7/?file=1837523#file1837523line221> > > > > This won't work right if using RM HA... > > > > I'd recommend using a ````YarnClient```` and passing it the > > ````hadoopConfig```` so it can figure out the RM address for you. There > > must be a benign simple ````YarnClient```` command you can run to verify > > connectivity. I could not find such command, but please let me know which one do you think of. Methods I tried retried to connect to RM multiple times for more minutes in case of a connection error. > On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote: > > tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java > > Lines 37 (patched) > > <https://reviews.apache.org/r/62459/diff/7/?file=1837525#file1837525line37> > > > > I'm not sure I like the name "BundleXYZ" for these classes. It's > > ambiguous with a Bundle Job. Perhaps > > "DiagBundleXYZ" instead? These classes are in the ``org.apache.oozie.tools.diag`` package that why I thought names like Client, BundleXYZ are not ambigous. Will fix it. - Attila ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62459/#review186754 ----------------------------------------------------------- On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62459/ > ----------------------------------------------------------- > > (Updated Oct. 2, 2017, 9:39 a.m.) > > > Review request for oozie. > > > Repository: oozie-git > > > Description > ------- > > A diagnostic tool that collects a bunch of job and other information from > Oozie in a zip file. > > > Diffs > ----- > > docs/src/site/twiki/DG_CommandLineTool.twiki > d4047671876dcc3279a2ec379bc1d003f5e6f1aa > pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c > tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 > tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java > PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION > > tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java > PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java > PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java > PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java > PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java > PRE-CREATION > tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java > PRE-CREATION > tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java > PRE-CREATION > tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java > PRE-CREATION > tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java > PRE-CREATION > > tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/62459/diff/8/ > > > Testing > ------- > > - new unit tests: TestOozieDiagBundleCollector > - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, > and executed the following commands: > -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed), > -- ``bin/oozie-diag-bundle-collector.sh -numworkflows 2000 -oozie > http://localhost:11000/oozie -output /tmp``, > -- ``bin/oozie-diag-bundle-collector.sh -jobs > 0000001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie > -output .`` (verified zip the tool generated). > > > Thanks, > > Attila Sasvari > >
