subject:"Re\: Review Request 62459\: OOZIE\-2296\: Add an Oozie diagnostic bundle tool"

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-11-15 Thread András Piros via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review191045
---


Ship it!




Ship It!

- András Piros


On Oct. 16, 2017, 12:32 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 16, 2017, 12:32 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml be153c16c45f91a8d57c78d6b7ebe073e7ca1f79 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/OozieLauncherLogFetcher.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/12/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-16 Thread Attila Sasvari via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Oct. 16, 2017, 12:32 p.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml be153c16c45f91a8d57c78d6b7ebe073e7ca1f79 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  
tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/OozieLauncherLogFetcher.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/12/

Changes: https://reviews.apache.org/r/62459/diff/11-12/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-16 Thread Attila Sasvari via Review Board



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 151 (patched)
> > 
> >
> > Should we skip control types?  That's probably fine for the start 
> > action, but what about a decision action?  That has some useful info.  
> > Maybe we should still print these out, but with fewer fields or something 
> > (e.g. no need for a console url).
> 
> Robert Kanter wrote:
> Thinking about this more, I think we should include control types.  It 
> will be helpful for following the flow of a workflow when there's decision 
> and fork nodes.  Please update the patch to handle this.  We don't need to 
> print out all of the fields, so you'll have to figure out which ones make 
> sense.
> 
> Attila Sasvari wrote:
> Sure, I will do so.
> 
> Attila Sasvari wrote:
> I believe only the decision node might interesting to see execution path, 
> but I am not sure. ``info.txt`` already contains execution / transition info:
> ```
> Action Id  : "004-171015215418832-oozie-asas-W@ctrl"
> Name   : "ctrl"
> Type   : "switch"
> Status : "OK"
> Transition : "forking"
> Start Time : "2017-10-15 20:20:36 GMT"
> End Time   : "2017-10-15 20:20:36 GMT"
> Error Code : null
> Error Message  : null
> Console URL: "-"
> Tracker URI: "-"
> External Child Ids : null
> External Id: "-"
> External Status: "forking"
> Data   : null
> Stats  : null
> Credentials: null
> Retries: "0"
> User Retry Int : "10"
> User Retry Count   : "0"
> User Retry Max : "0"
> ```
> 
> If I store resolved action defintion, it would not add too much 
> information:
> ```
> # cat 004-171015215418832-oozie-asas-W/resolved-actions/ctrl.xml  
> 
>   true
>   
> % 
> ```
> 
> I created a workflow with a fork node and also with a decision node. I 
> saw that resolved action config xml were empty for control nodes other than 
> the decision node.
> 
> ```
> 0  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/:start:.xml
>   108  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/ctrl.xml
> 0  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/end.xml
> 0  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/forking.xml
> 0  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/join.xml
> 83834  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/launcher_mr-node.log
> 83843  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/launcher_mr-node2.log
> 47965  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/launcher_shell1.log
> 47965  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/launcher_shell2.log
>  2045  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/mr-node.xml
>  2045  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/mr-node2.xml
>   544  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/shell1.xml
>   544  10-15-17 22:44   
> /004-171015215418832-oozie-asas-W/resolved-actions/shell2.xml
> ```
> Zero byte xml-s are not interesting. I will remove non-decision action 
> types.
> 
> Other thing: it would also good to add later an option to download the 
> final job graph (thanks to OOZIE-2406 now we can download it as a Dot file 
> too). If they have many action, it might help a lot when taking a quick look. 
> 
> Misc: Would it better to have a different pattern for the filename? For 
> example: it might be good to see where the bundle was actually taken (i.e. 
> include fqdn-hostname, there are existing diagnostic tools that use similar 
> pattern).

update: I talked with Andras Piros and gezapeti and decided to include control 
nodes. It makes it easy to see whether the workflow failed or not (you don't 
need to open the info.txt).

 
```
Archive:  
/Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/./oozie-diag-bundle-1508156773346.zip
  Length Date   TimeName
    
 7843  10-16-17 14:26   /006-171015215418832-oozie-asas-W/info.txt
  322  10-16-17 14:26   /006-171015215418832-oozie-asas-W/job.properties
20878  10-16-17 14:26   /006-171015215418832-oozie-asas-W/log.txt
0  10-16-17 14:26

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-16 Thread Attila Sasvari via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Oct. 16, 2017, 7:11 a.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml be153c16c45f91a8d57c78d6b7ebe073e7ca1f79 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  
tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/OozieLauncherLogFetcher.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/11/

Changes: https://reviews.apache.org/r/62459/diff/10-11/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-15 Thread Attila Sasvari via Review Board



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 151 (patched)
> > 
> >
> > Should we skip control types?  That's probably fine for the start 
> > action, but what about a decision action?  That has some useful info.  
> > Maybe we should still print these out, but with fewer fields or something 
> > (e.g. no need for a console url).
> 
> Robert Kanter wrote:
> Thinking about this more, I think we should include control types.  It 
> will be helpful for following the flow of a workflow when there's decision 
> and fork nodes.  Please update the patch to handle this.  We don't need to 
> print out all of the fields, so you'll have to figure out which ones make 
> sense.
> 
> Attila Sasvari wrote:
> Sure, I will do so.

I believe only the decision node might interesting to see execution path, but I 
am not sure. ``info.txt`` already contains execution / transition info:
```
Action Id  : "004-171015215418832-oozie-asas-W@ctrl"
Name   : "ctrl"
Type   : "switch"
Status : "OK"
Transition : "forking"
Start Time : "2017-10-15 20:20:36 GMT"
End Time   : "2017-10-15 20:20:36 GMT"
Error Code : null
Error Message  : null
Console URL: "-"
Tracker URI: "-"
External Child Ids : null
External Id: "-"
External Status: "forking"
Data   : null
Stats  : null
Credentials: null
Retries: "0"
User Retry Int : "10"
User Retry Count   : "0"
User Retry Max : "0"
```

If I store resolved action defintion, it would not add too much information:
```
# cat 004-171015215418832-oozie-asas-W/resolved-actions/ctrl.xml  

  true
  
% 
```

I created a workflow with a fork node and also with a decision node. I saw that 
resolved action config xml were empty for control nodes other than the decision 
node.

```
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/:start:.xml
  108  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/ctrl.xml
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/end.xml
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/forking.xml
0  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/join.xml
83834  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_mr-node.log
83843  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_mr-node2.log
47965  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_shell1.log
47965  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/launcher_shell2.log
 2045  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/mr-node.xml
 2045  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/mr-node2.xml
  544  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/shell1.xml
  544  10-15-17 22:44   
/004-171015215418832-oozie-asas-W/resolved-actions/shell2.xml
```
Zero byte xml-s are not interesting. I will remove non-decision action types.

Other thing: it would also good to add later an option to download the final 
job graph (thanks to OOZIE-2406 now we can download it as a Dot file too). If 
they have many action, it might help a lot when taking a quick look. 

Misc: Would it better to have a different pattern for the filename? For 
example: it might be good to see where the bundle was actually taken (i.e. 
include fqdn-hostname, there are existing diagnostic tools that use similar 
pattern).


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 4, 2017, 2:18 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 4, 2017, 2:18 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-13 Thread Peter Cseh via Review Board



> On Oct. 6, 2017, 12:26 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 173-174 (patched)
> > 
> >
> > LogAggregationUtils is marked @Private so we shouldn't 
> > be using it.  Hadoop can change things incompatibly here.
> 
> Attila Sasvari wrote:
> What do you think about borrowing/inlining/copying those functions from 
> Hadoop 2.6 too? I was thinking about it and it is probably better than 
> bringing back the initial ``ExecutorService`` based approach. I am thinking 
> of a class like:
> 
> ```java
> // TODO: once OOZIE-2983 ("Stream the Launcher AM Logs") is done, remove 
> it.
> public class OozieLauncherLogFetcher {
> private static final String TMP_FILE_SUFFIX = ".tmp";
> final private Configuration hadoopConfig;
> 
> public OozieLauncherLogFetcher(final Configuration hadoopConfig) {
> this.hadoopConfig = hadoopConfig;
> }
> 
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
> private static void logDirNotExist(String remoteAppLogDir) {
> System.out.println(remoteAppLogDir + "does not exist.");
> System.out.println("Log aggregation has not completed or is not 
> enabled.");
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
> private static void emptyLogDir(String remoteAppLogDir) {
> System.out.println(remoteAppLogDir + "does not have any log 
> files.");
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static String getRemoteNodeLogDirSuffix(Configuration conf) {
> return conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX, 
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR_SUFFIX);
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static Path getRemoteLogSuffixedDir(Path remoteRootLogDir, 
> String user, String suffix) {
> return suffix != null && !suffix.isEmpty() ? new 
> Path(getRemoteLogUserDir(remoteRootLogDir, user), suffix) :
> getRemoteLogUserDir(remoteRootLogDir, user);
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static Path getRemoteLogUserDir(Path remoteRootLogDir, String 
> user) {
> return new Path(remoteRootLogDir, user);
> }
> 
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static Path getRemoteAppLogDir(Path remoteRootLogDir, 
> ApplicationId appId, String user, String suffix) {
> return new Path(getRemoteLogSuffixedDir(remoteRootLogDir, user, 
> suffix), appId.toString());
> }
> 
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
> public int dumpAllContainersLogs(ApplicationId appId, String 
> appOwner, PrintStream out) throws IOException {
> Path remoteRootLogDir = new 
> Path(hadoopConfig.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
> String logDirSuffix = getRemoteNodeLogDirSuffix(hadoopConfig);
> Path remoteAppLogDir = getRemoteAppLogDir(remoteRootLogDir, 
> appId, appOwner, logDirSuffix);
> 
> RemoteIterator nodeFiles;
> try {
> Path qualifiedLogDir = 
> FileContext.getFileContext(hadoopConfig).makeQualified(remoteAppLogDir);
> nodeFiles = 
> FileContext.getFileContext(qualifiedLogDir.toUri(), 
> hadoopConfig).listStatus(remoteAppLogDir);
> } catch (FileNotFoundException fileNotFoundException) {
> logDirNotExist(remoteAppLogDir.toString());
> return -1;
> }
> 
> boolean foundAnyLogs = false;
> 
> while(true) {
> FileStatus thisNodeFile;
> do {
> if (!nodeFiles.hasNext()) {
> if (!foundAnyLogs) {
> emptyLogDir(remoteAppLogDir.toString());
> return -1;
> }
> 
> return 0;
> }
> 
> thisNodeFile = (FileStatus)nodeFiles.next();
> } 
> while(thisNodeFile.getPath().getName().endsWith(TMP_FILE_SUFFIX));
> 
> AggregatedLogFormat.LogReader reader = new 
> AggregatedLogFormat.LogReader(hadoopConfig, thisNodeFile.getPath());
> 
> try {
> AggregatedLogFormat.LogKey key = new 
> AggregatedLogFormat.LogKey();
> DataInputStream

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-13 Thread Attila Sasvari via Review Board

> On Oct. 6, 2017, 12:26 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 173-174 (patched)
> > 
> >
> > LogAggregationUtils is marked @Private so we shouldn't 
> > be using it.  Hadoop can change things incompatibly here.

What do you think about borrowing/inlining/copying those functions from Hadoop 
2.6 too? I was thinking about it and it is probably better than bringing back 
the initial ``ExecutorService`` based approach. I am thinking of a class like:

```java
// TODO: once OOZIE-2983 ("Stream the Launcher AM Logs") is done, remove it.
public class OozieLauncherLogFetcher {
private static final String TMP_FILE_SUFFIX = ".tmp";
final private Configuration hadoopConfig;

public OozieLauncherLogFetcher(final Configuration hadoopConfig) {
this.hadoopConfig = hadoopConfig;
}

// Borrowed code from org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
private static void logDirNotExist(String remoteAppLogDir) {
System.out.println(remoteAppLogDir + "does not exist.");
System.out.println("Log aggregation has not completed or is not 
enabled.");
}
// Borrowed code from org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
private static void emptyLogDir(String remoteAppLogDir) {
System.out.println(remoteAppLogDir + "does not have any log files.");
}
// Borrowed code from 
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static String getRemoteNodeLogDirSuffix(Configuration conf) {
return conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX, 
YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR_SUFFIX);
}
// Borrowed code from 
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static Path getRemoteLogSuffixedDir(Path remoteRootLogDir, String 
user, String suffix) {
return suffix != null && !suffix.isEmpty() ? new 
Path(getRemoteLogUserDir(remoteRootLogDir, user), suffix) :
getRemoteLogUserDir(remoteRootLogDir, user);
}
// Borrowed code from 
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static Path getRemoteLogUserDir(Path remoteRootLogDir, String user) {
return new Path(remoteRootLogDir, user);
}

// Borrowed code from 
org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
public static Path getRemoteAppLogDir(Path remoteRootLogDir, ApplicationId 
appId, String user, String suffix) {
return new Path(getRemoteLogSuffixedDir(remoteRootLogDir, user, 
suffix), appId.toString());
}

// Borrowed code from org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
public int dumpAllContainersLogs(ApplicationId appId, String appOwner, 
PrintStream out) throws IOException {
Path remoteRootLogDir = new 
Path(hadoopConfig.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
String logDirSuffix = getRemoteNodeLogDirSuffix(hadoopConfig);
Path remoteAppLogDir = getRemoteAppLogDir(remoteRootLogDir, appId, 
appOwner, logDirSuffix);

RemoteIterator nodeFiles;
try {
Path qualifiedLogDir = 
FileContext.getFileContext(hadoopConfig).makeQualified(remoteAppLogDir);
nodeFiles = FileContext.getFileContext(qualifiedLogDir.toUri(), 
hadoopConfig).listStatus(remoteAppLogDir);
} catch (FileNotFoundException fileNotFoundException) {
logDirNotExist(remoteAppLogDir.toString());
return -1;
}

boolean foundAnyLogs = false;

while(true) {
FileStatus thisNodeFile;
do {
if (!nodeFiles.hasNext()) {
if (!foundAnyLogs) {
emptyLogDir(remoteAppLogDir.toString());
return -1;
}

return 0;
}

thisNodeFile = (FileStatus)nodeFiles.next();
} while(thisNodeFile.getPath().getName().endsWith(TMP_FILE_SUFFIX));

AggregatedLogFormat.LogReader reader = new 
AggregatedLogFormat.LogReader(hadoopConfig, thisNodeFile.getPath());

try {
AggregatedLogFormat.LogKey key = new 
AggregatedLogFormat.LogKey();
DataInputStream valueStream = reader.next(key);

while(valueStream != null) {
String containerString = "\n\nContainer: " + key + " on " + 
thisNodeFile.getPath().getName();
out.println(containerString);
out.println(StringUtils.repeat("=", 
containerString.length()));

while(true) {
try {

AggregatedLogFormat.LogReader.readAContainerLogsForALogType(valueStream, out,

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-05 Thread Robert Kanter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review187223
---




docs/src/site/twiki/DG_CommandLineTool.twiki
Lines 1989 (patched)


Shouldn't this "Done" show up on the previous line like the rest of them?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 173-174 (patched)


LogAggregationUtils is marked @Private so we shouldn't be 
using it.  Hadoop can change things incompatibly here.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 180 (patched)


var20 is a funny name for a variable :)



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 241 (patched)


storeOozieLauncherLog should be called per Action, not per Job.  
Launcher logs are an Action-level concept.

Here, you're just picking the first action's external id to use for the 
job, which may not even exist (e.g. the first action is a decision node or an 
hdfs node.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 242 (patched)


We should get this from the WorkflowJob.  The log reading code 
computes a path based on the user who ran the job, which is likely to be 
different from the user running this tool.


- Robert Kanter


On Oct. 4, 2017, 2:18 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 4, 2017, 2:18 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/10/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-05 Thread Robert Kanter



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 151 (patched)
> > 
> >
> > Should we skip control types?  That's probably fine for the start 
> > action, but what about a decision action?  That has some useful info.  
> > Maybe we should still print these out, but with fewer fields or something 
> > (e.g. no need for a console url).

Thinking about this more, I think we should include control types.  It will be 
helpful for following the flow of a workflow when there's decision and fork 
nodes.  Please update the patch to handle this.  We don't need to print out all 
of the fields, so you'll have to figure out which ones make sense.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 4, 2017, 2:18 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 4, 2017, 2:18 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/10/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-04 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Oct. 4, 2017, 2:18 p.m.)


Review request for oozie.


Changes
---

New approach to fetch oozie launcher logs.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  
tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/10/

Changes: https://reviews.apache.org/r/62459/diff/9-10/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-03 Thread Robert Kanter



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 221 (patched)
> > 
> >
> > This won't work right if using RM HA...
> > 
> > I'd recommend using a YarnClient and passing it the 
> > hadoopConfig so it can figure out the RM address for you.  There 
> > must be a benign simple YarnClient command you can run to verify 
> > connectivity.
> 
> Attila Sasvari wrote:
> I could not find such command, but please let me know which one do you 
> think of. Methods I tried retried to connect to RM multiple times for more 
> minutes in case of a connection error.

Ya, apparently there isn't really one.  Hadoop 3 has a CLI command that would 
tell you the state of each RM 
(https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java#L481)
  Maybe you can steal something from it?  I'm not really sure of a good 
solution here.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 2, 2017, 9:39 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/9/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-03 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186940
---




tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 77 (patched)


I think this should go to stdout instead



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 113 (patched)


I see writeString("\n") at least a dozen of times (actually 18).

I strongly suggest adding a method to the writer and call it writeNewLine().



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
Lines 41 (patched)


We might as we call this DATE_FORMAT. Also, if it's referenced from other 
classes (and I see it is), just make it public.



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
Lines 47 (patched)


I know that this is returned by the parser, but can we switch to 
List here if it's not a big deal? It just feels safer.



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java
Lines 31 (patched)


Nit: I know that zip() is expressive in itself, but still, the name 
compress() sounds better to me.



tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java
Lines 48 (patched)


This is usually thrown by the JVM when a static initializer block fails. 
Could we change this to something else? An ordinary RuntimeException looks good 
enough to me.



tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
Lines 46 (patched)


Nit: space :D



tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
Lines 52 (patched)


Do we need to call flush() here?



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 56 (patched)


Could this whole example metrics be stored in a file and not inline in the 
code?



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 122 (patched)


Simpler:

import static java.nio.file.Files.readAllBytes;
import static java.nio.file.Paths.get;

String s = new String(readAllBytes(get("test.txt";

This part should be extracted somewhere to avoid repeated coding to load a 
file



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 127 (patched)


The string "UTF-8" is repeated a couple of times. Make it const or use 
StandardCharsets.UTF_8.



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 134 (patched)


Same here, I would store this in an external file



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 215 (patched)


Simplify file loading



tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java
Lines 231 (patched)


Handle this catch properly



tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java
Lines 120 (patched)


Simplify file loading


- Peter Bacsko


On okt. 2, 2017, 9:39 de, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated okt. 2, 2017, 9:39 de)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-03 Thread Attila Sasvari



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 191 (patched)
> > 
> >
> > Is there not a cleaner way to do this than using a CLI like this?
> 
> Attila Sasvari wrote:
> I could not find one, please let me know if you have something in mind.

OK, I am going to inline/borrow code from dumpAllContainersLogs in LogsCLI. 
What we do is basically just fetching logs from HDFS. If there is an exception, 
we log it and move on.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 2, 2017, 9:39 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/9/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-02 Thread Attila Sasvari



> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 97-98 (patched)
> > 
> >
> > Something to consider for all output of the tool, not just here: we're 
> > outputting most of the info in a human-readable format.  Should we think 
> > about using a machine-readable format?  Or maybe having the option for one? 
> >  Or doing both?  The idea being that someone would then be able to write 
> > their own tool that could analyze stuff.  We already have some code 
> > somewhere that converts a WorkflowJob into JSON, so it shouldn't be a lot 
> > of work to add this either.  That might also be a good idea from a 
> > compatibility perspective - i.e. what's the compatibility story on this 
> > out?  If there's a new field, what do we do?

It is a good idea, and I would create a separate JIRA to discuss/design and 
implement it. OOZIE-3074


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 163 (patched)
> > 
> >
> > I think the JHS may also be required, in the cases where the RM has 
> > forgotten about the job.
> > 
> > And what about HDFS?  That's required too.
> > 
> > I'm thinking we might be best off not doing these checks.  It's too 
> > complicated (CM spent a lot of effort on this) and we can't check for 
> > everything (e.g. what if log aggregation is turned off?).  Besides, we're 
> > already handling exceptions below when trying to get the logs - if the RM, 
> > JHS, HDFS, etc isn't working, the call will fail anyway.

I agree with it, but failing fast would be a better experience than a 30 
seconds timeout. The default retry policy of YarnClient resulted in a lot of 
retries and I could not find the proper parameter to control it (I want to 
restrict retries to a few seconds instead of minutes).


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 185 (patched)
> > 
> >
> > Please create a followup JIRA to change this in the future to use 
> > OOZIE-2983 ("Stream the Launcher AM Logs") once it's done.  This will also 
> > be nice in that we can get rid of the RM up check.

I will do so.


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 191 (patched)
> > 
> >
> > Is there not a cleaner way to do this than using a CLI like this?

I could not find one, please let me know if you have something in mind.


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 221 (patched)
> > 
> >
> > This won't work right if using RM HA...
> > 
> > I'd recommend using a YarnClient and passing it the 
> > hadoopConfig so it can figure out the RM address for you.  There 
> > must be a benign simple YarnClient command you can run to verify 
> > connectivity.

I could not find such command, but please let me know which one do you think 
of. Methods I tried retried to connect to RM multiple times for more minutes in 
case of a connection error.


> On Sept. 30, 2017, 4:41 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java
> > Lines 37 (patched)
> > 
> >
> > I'm not sure I like the name "BundleXYZ" for these classes.  It's 
> > ambiguous with a Bundle Job.  Perhaps 
> > "DiagBundleXYZ" instead?

These classes are in the ``org.apache.oozie.tools.diag`` package that why I 
thought names like Client, BundleXYZ are not ambigous. Will fix it.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---


On Oct. 2, 2017, 9:39 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 2, 2017, 9:39 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-02 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Oct. 2, 2017, 9:39 a.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  
tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/8/

Changes: https://reviews.apache.org/r/62459/diff/7-8/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-10-02 Thread Attila Sasvari



> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 97-98 (patched)
> > 
> >
> > Put `resolvedActionsDir` and `configEntryWriter` to fields, and you got 
> > two parameters less.

Resolved action dir varies from workflow action to workflow action, and right 
now DiagBundleWriter constructor uses it.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 99-119 (patched)
> > 
> >
> > Using a `StringBuilder`, maybe?

It is a writer not where we write to.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 276-302 (patched)
> > 
> >
> > `StringBuffer`?

We are writing to a writer (StringBuffer's thread-safety is not needed).


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java
> > Lines 43-45 (patched)
> > 
> >
> > `int`

Findbugs would complain about boxing / unboxing.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java
> > Lines 85 (patched)
> > 
> >
> > What about try-with-resources?

Please expand. The writer object is open for a period time and write is called 
multiple times.


> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
> > Lines 58-64 (patched)
> > 
> >
> > `writeSection()`

You cannot easily reuse such function in this class, look at function calls and 
types.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186568
---


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-29 Thread Robert Kanter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186754
---




docs/src/site/twiki/DG_CommandLineTool.twiki
Lines 1948 (patched)


It would be good to add something about the children aspect of the tool 
(e.g. that if you get a coordinator, it will also get some number of child 
workflows)



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 97-98 (patched)


Something to consider for all output of the tool, not just here: we're 
outputting most of the info in a human-readable format.  Should we think about 
using a machine-readable format?  Or maybe having the option for one?  Or doing 
both?  The idea being that someone would then be able to write their own tool 
that could analyze stuff.  We already have some code somewhere that converts a 
WorkflowJob into JSON, so it shouldn't be a lot of work to add this either.  
That might also be a good idea from a compatibility perspective - i.e. what's 
the compatibility story on this out?  If there's a new field, what do we do?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 99-119 (patched)


I'm not sure that's necessary here - we're not concatinating strings, we're 
writing to a Writer.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 151 (patched)


Should we skip control types?  That's probably fine for the start action, 
but what about a decision action?  That has some useful info.  Maybe we should 
still print these out, but with fewer fields or something (e.g. no need for a 
console url).



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 163 (patched)


I think the JHS may also be required, in the cases where the RM has 
forgotten about the job.

And what about HDFS?  That's required too.

I'm thinking we might be best off not doing these checks.  It's too 
complicated (CM spent a lot of effort on this) and we can't check for 
everything (e.g. what if log aggregation is turned off?).  Besides, we're 
already handling exceptions below when trying to get the logs - if the RM, JHS, 
HDFS, etc isn't working, the call will fail anyway.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 185 (patched)


Please create a followup JIRA to change this in the future to use 
OOZIE-2983 ("Stream the Launcher AM Logs") once it's done.  This will also be 
nice in that we can get rid of the RM up check.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 191 (patched)


Is there not a cleaner way to do this than using a CLI like this?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 221 (patched)


This won't work right if using RM HA...

I'd recommend using a YarnClient and passing it the 
hadoopConfig so it can figure out the RM address for you.  There must 
be a benign simple YarnClient command you can run to verify 
connectivity.



tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java
Lines 37 (patched)


I'm not sure I like the name "BundleXYZ" for these classes.  It's ambiguous 
with a Bundle Job.  Perhaps 
"DiagBundleXYZ" instead?



tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java
Lines 32 (patched)


Can use the Java 7 try-with-resources.



tools/src/main/java/org/apache/oozie/tools/diag/Client.java
Lines 32 (patched)


I think you have a typo here: "... so we get that for free"



tools/src/main/java/org/apache/oozie/tools/diag/Client.java
Lines 38 (patched)


I would add a comment here explaining why we're doing this.  (i.e. that 
Oozie has a jsp page with a thread dump, and the Oozie client normally doesn't 
have a way of getting it, so we're doing that here; reusing all the fancy HTTP 
handling code in AuthOozieClient)



tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java
Lines 47-49 (patched)


Strange lining here



tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java
Lines 45-47 (patched)


Strange

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-29 Thread Peter Cseh



> On Sept. 29, 2017, 8:17 a.m., András Piros wrote:
> > tools/src/main/bin/oozie-diag-bundle-collector.sh
> > Lines 55-58 (patched)
> > 
> >
> > Extract function.

I don't like the idea of extracting all these functions in a shell script. None 
of the extracted functions would deduplicate code and none of the will be used 
in other shell scripts either. I find it way easier just to read the shell 
script as it is rather that following function calls in it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186568
---


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-29 Thread András Piros


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186568
---




tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 23-31 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 33-34 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 36 (patched)


`OOZIECP` or `OOZIECLASSPATH` would be a better name.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 37-45 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 48-53 (patched)


Extract function.



tools/src/main/bin/oozie-diag-bundle-collector.sh
Lines 55-58 (patched)


Extract function.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 51 (patched)


`MAX_TIMEOUT_SECONDS` would be a better name.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 60 (patched)


`storeWorkflowJobDetails()` might be a better name.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 67-73 (patched)


This is happening three times in code. Extract to one and reuse.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 69 (patched)


Returning here and having no `else` path seems cleaner to me.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 71-77 (patched)


Extract method.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 76 (patched)


`"Could not create resolved actions directory:"`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 81-83 (patched)


Nice :)



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 88 (patched)


`"Got details for ... successfully"`, maybe?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 93 (patched)


Maybe printing only the `Exception` message is a better idea.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 97-98 (patched)


Put `resolvedActionsDir` and `configEntryWriter` to fields, and you got two 
parameters less.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 99-119 (patched)


Using a `StringBuilder`, maybe?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 118 (patched)


`ACTIONS`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 120 (patched)


Put this just before `while`.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 124-127 (patched)


Substitute w/ a normal `while` loop might be more readable.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 128-149 (patched)


Using a `StringBuilder`, maybe?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 158-160 (patched)


`persistWorkflowAction()`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 170 (patched)


It could implement `Runnable`, in which case you don't need to `return 
null`.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 182 (patched)


`StandardCharsets.UTF_8.toString()` seems cleaner to me.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 189 (patched)


`IllegalStateException`



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 194 (patched)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-28 Thread Peter Cseh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186564
---




tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java
Lines 36 (patched)


Shouldn't this make the findbugs warnings go away?


- Peter Cseh


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-28 Thread Peter Cseh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186563
---




tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java
Lines 1 (patched)


License



tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java
Lines 1 (patched)


License


- Peter Cseh


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-28 Thread Peter Cseh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186561
---




tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 164 (patched)


Please log here that Launcher logs will be missing for this reason.


- Peter Cseh


On Sept. 27, 2017, 2:12 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 27, 2017, 2:12 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/7/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-27 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 27, 2017, 2:12 p.m.)


Review request for oozie.


Changes
---

Added more tests for AppInfoCollector and others.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/7/

Changes: https://reviews.apache.org/r/62459/diff/6-7/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-26 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 26, 2017, 9:03 p.m.)


Review request for oozie.


Changes
---

more tests


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleCompressor.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/AppInfoCollectorTest.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/6/

Changes: https://reviews.apache.org/r/62459/diff/5-6/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-26 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 26, 2017, 10:51 a.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/BundleCollectorDriver.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/Client.java PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ConfigEntryWriter.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/5/

Changes: https://reviews.apache.org/r/62459/diff/4-5/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-25 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 25, 2017, 8:27 a.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/4/

Changes: https://reviews.apache.org/r/62459/diff/3-4/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-25 Thread Attila Sasvari



> On Sept. 23, 2017, 1:18 p.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
> > Lines 250 (patched)
> > 
> >
> > Do we need Integer instead of int?

Findbugs found an issue about boxing/unboxing that is why it returns an Integer.


> On Sept. 23, 2017, 1:18 p.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
> > Lines 258 (patched)
> > 
> >
> > Do we need Integer instead of int?

See my reply above


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186052
---


On Sept. 22, 2017, 12:50 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 22, 2017, 12:50 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/3/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-23 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186059
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 237 (patched)


We have to re-think what and how we test. What's the rationale behind 
selecting these four methods? With proper unit testing validating the negative 
behaviour should be straightforward and these methods could be private.



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 911 (patched)


This annotation looks unnecessary


- Peter Bacsko


On szept. 22, 2017, 12:50 du, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated szept. 22, 2017, 12:50 du)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/3/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-23 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186053
---



My overall impression:

1. The class is too big, almost 1000 lines. Would it be possible to break it up 
somehow? I think the most obvious way to do is put different collector logics 
into different classes.
2. Code is strongly coupled with I/O operations, making it hard to test. 
Certain stuff, like filesystem operations (exists/mkdir) and stream creation 
should be wrapped and injected. See LocalFsOperations in oozie-sharelib. That 
way you can avoid direct IO. A mock factory for creating streams makes testing 
extremely easy because it can just return a StringWriter which can be examined 
in the tests.

- Peter Bacsko


On szept. 22, 2017, 12:50 du, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated szept. 22, 2017, 12:50 du)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/3/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-23 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review186052
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 155 (patched)


Synchronized?



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 168 (patched)


Misplaced brace



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 238 (patched)


Do we need Integer instead of int?



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 250 (patched)


Do we need Integer instead of int?



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 258 (patched)


Do we need Integer instead of int?



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 297 (patched)


We shouldn't just print the exception IMO. At least we log something, 
re-throw it, something that is more appropriate.



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 318 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 337 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 356 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 407 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 463 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 483 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 508 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 523 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 538 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 548 (patched)


There are different places in the code where the job type is checked. I 
believe these should be some mini helper methods and invoke them if necessary 
to determine the type of a job.

Something like

private boolean isWorkflow(String id) {
  return id.endsWith("-W");
}



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 578 (patched)


Besides printing the stack trace, add some error message too



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 677 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 685 (patched)


Extract default encoding to a final static variable



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 788 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 837 (patched)


See my comment above about exception handling



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 872 (patched)


Extract default encoding to a final static variable



tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 878 (patched)

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-22 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 22, 2017, 12:50 p.m.)


Review request for oozie.


Changes
---

Collect Oozie launcher logs


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/3/

Changes: https://reviews.apache.org/r/62459/diff/2-3/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-22 Thread Attila Sasvari



> On Sept. 22, 2017, 8:38 a.m., Peter Cseh wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
> > Lines 602-606 (patched)
> > 
> >
> > Please get the logs for the LauncherMapper based on externalID as well.

There is no LauncherMapper anymore, but I will get logs for the Oozie launcher 
job if possible.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review185956
---


On Sept. 21, 2017, 5:10 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 21, 2017, 5:10 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/2/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-22 Thread Peter Cseh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review185956
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 602-606 (patched)


Please get the logs for the LauncherMapper based on externalID as well.


- Peter Cseh


On Sept. 21, 2017, 5:10 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 21, 2017, 5:10 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/2/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/
---

(Updated Sept. 21, 2017, 5:10 p.m.)


Review request for oozie.


Repository: oozie-git


Description
---

A diagnostic tool that collects a bunch of job and other information from Oozie 
in a zip file.


Diffs (updated)
-

  docs/src/site/twiki/DG_CommandLineTool.twiki 
d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
  pom.xml efccc346932514ada578a3462eb3c3cfe519a323 
  tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
  tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
  tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62459/diff/2/

Changes: https://reviews.apache.org/r/62459/diff/1-2/


Testing
---

- new unit tests: TestOozieDiagBundleCollector
- started Oozie with a pseudo hadoop cluster, submitted a couple workflows, and 
executed the following commands: 
-- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
-- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
http://localhost:11000/oozie -output /tmp``, 
-- ``bin/oozie-diag-bundle-collector.sh  -jobs 
001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
-output .`` (verified zip the tool generated).


Thanks,

Attila Sasvari

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Peter Cseh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review185887
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 869 (patched)


I think we should distinguish "null" values from empty strings as this can 
be an important thing in configurations


- Peter Cseh


On Sept. 21, 2017, 10:31 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 21, 2017, 10:31 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/1/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

2017-09-21 Thread Peter Cseh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review185884
---




tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java
Lines 141-148 (patched)


Can these be fields so the argument lists are not that long?


- Peter Cseh


On Sept. 21, 2017, 10:31 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Sept. 21, 2017, 10:31 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/OozieDiagBundleCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/TestOozieDiagBundleCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/1/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

36 matches

Site Navigation

Mail list logo

Footer information