Thanks Brock, I do get a MR folder structure, like
test/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001 qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_m_000000_0 qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_r_000000_0 qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_r_000000_0/output but no files in them. I'll try to dig and see whether the files get deleted or not created to start with... thanks, ~Remus ________________________________________ From: Brock Noland <br...@cloudera.com> Sent: Thursday, February 6, 2014 11:58 PM To: dev@hive.apache.org Cc: hive-...@hadoop.apache.org Subject: Re: Any way to preserve the .q unit tests MR execution output? Not sure about saving the output of the jobs, but I believe the mr logs should be in target/ somewhere or in /tmp/$USER On Thu, Feb 6, 2014 at 3:34 PM, Remus Rusanu <rem...@microsoft.com> wrote: > When running a test that fails, I can get the output diff and the hive log, > but when the failure occurs in the query execution, I don't see how I can > retrieve whatever was logged by the execution. > > For example, I can see in the target\qfile-results\...q.out: > > > PREHOOK: query: SELECT cint, > ... > AND (ctinyint > cbigint))) > PREHOOK: type: QUERY > PREHOOK: Input: default@alltypesorc > PREHOOK: Output: > file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-mr-10000 > Execution failed with exit status: 2 > Obtaining error information > > Task failed! > Task ID: > Stage-1 > > Logs: > > /Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > > The hive.log is a bit more detailed: > 2014-02-06 12:17:05,741 INFO ql.Driver (SessionState.java:printInfo(504)) - > Launching Job 1 out of 1 > 2014-02-06 12:17:05,742 INFO exec.Task (SessionState.java:printInfo(504)) - > Number of reduce tasks is set to 0 since there's no reduce operator > 2014-02-06 12:17:05,822 INFO mr.ExecDriver (MapRedTask.java:execute(175)) - > Generating plan file > file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml > 2014-02-06 12:17:05,822 INFO log.PerfLogger > (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=serializePlan > from=org.apache.hadoop.hive.ql.exec.Utilities> > 2014-02-06 12:17:05,823 INFO exec.Utilities > (Utilities.java:serializePlan(783)) - Serializing MapredWork via kryo > 2014-02-06 12:17:05,835 INFO log.PerfLogger > (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=serializePlan > start=1391717825822 end=1391717825835 duration=13 > from=org.apache.hadoop.hive.ql.exec.Utilities> > 2014-02-06 12:17:05,836 INFO mr.ExecDriver (MapRedTask.java:execute(207)) - > Executing: /Users/rremus/Development/hive/itests/qtest/../../testutils/hadoop > jar > /Users/rremus/.m2/repository/org/apache/hive/hive-exec/0.13.0-SNAPSHOT/hive-exec-0.13.0-SNAPSHOT.jar > org.apache.hadoop.hive.ql.exec.mr.ExecDriver -plan > file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml > -jobconffile > file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10002/jobconf.xml > 2014-02-06 12:17:15,431 ERROR exec.Task (SessionState.java:printError(513)) - > Execution failed with exit status: 2 > 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) - > Obtaining error information > 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) - > Task failed! > Task ID: > Stage-1 > > Logs: > > 2014-02-06 12:17:15,433 ERROR exec.Task (SessionState.java:printError(513)) - > /Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log > 2014-02-06 12:17:15,433 ERROR mr.ExecDriver (MapRedTask.java:execute(280)) - > Execution failed with exit status: 2 > > Still, at the end of running the test there is nothing left in > target/tmp/localscratchdir, scratchdir or in target/tmp/hadoop-tmp. In this > case, I'm missing the output from the MR job that failed. > Is there a way for me to get this missing output? > > Thanks, > ~Remus > > > > -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
