I just ran a test on my local machine and the MR logs went to /tmp/brock. Brock
On Sat, Feb 8, 2014 at 6:12 AM, Remus Rusanu <rem...@microsoft.com> wrote: > Thanks Brock, > > I do get a MR folder structure, like > > test/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache > qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001 > qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_m_000000_0 > qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_r_000000_0 > qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_r_000000_0/output > > but no files in them. I'll try to dig and see whether the files get deleted > or not created to start with... > > thanks, > ~Remus > ________________________________________ > From: Brock Noland <br...@cloudera.com> > Sent: Thursday, February 6, 2014 11:58 PM > To: dev@hive.apache.org > Cc: hive-...@hadoop.apache.org > Subject: Re: Any way to preserve the .q unit tests MR execution output? > > Not sure about saving the output of the jobs, but I believe the mr > logs should be in target/ somewhere or in /tmp/$USER > > On Thu, Feb 6, 2014 at 3:34 PM, Remus Rusanu <rem...@microsoft.com> wrote: >> When running a test that fails, I can get the output diff and the hive log, >> but when the failure occurs in the query execution, I don't see how I can >> retrieve whatever was logged by the execution. >> >> For example, I can see in the target\qfile-results\...q.out: >> >> >> PREHOOK: query: SELECT cint, >> ... >> AND (ctinyint > cbigint))) >> PREHOOK: type: QUERY >> PREHOOK: Input: default@alltypesorc >> PREHOOK: Output: >> file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-mr-10000 >> Execution failed with exit status: 2 >> Obtaining error information >> >> Task failed! >> Task ID: >> Stage-1 >> >> Logs: >> >> /Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log >> FAILED: Execution Error, return code 2 from >> org.apache.hadoop.hive.ql.exec.mr.MapRedTask >> >> The hive.log is a bit more detailed: >> 2014-02-06 12:17:05,741 INFO ql.Driver (SessionState.java:printInfo(504)) - >> Launching Job 1 out of 1 >> 2014-02-06 12:17:05,742 INFO exec.Task (SessionState.java:printInfo(504)) - >> Number of reduce tasks is set to 0 since there's no reduce operator >> 2014-02-06 12:17:05,822 INFO mr.ExecDriver (MapRedTask.java:execute(175)) - >> Generating plan file >> file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml >> 2014-02-06 12:17:05,822 INFO log.PerfLogger >> (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=serializePlan >> from=org.apache.hadoop.hive.ql.exec.Utilities> >> 2014-02-06 12:17:05,823 INFO exec.Utilities >> (Utilities.java:serializePlan(783)) - Serializing MapredWork via kryo >> 2014-02-06 12:17:05,835 INFO log.PerfLogger >> (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=serializePlan >> start=1391717825822 end=1391717825835 duration=13 >> from=org.apache.hadoop.hive.ql.exec.Utilities> >> 2014-02-06 12:17:05,836 INFO mr.ExecDriver (MapRedTask.java:execute(207)) - >> Executing: >> /Users/rremus/Development/hive/itests/qtest/../../testutils/hadoop jar >> /Users/rremus/.m2/repository/org/apache/hive/hive-exec/0.13.0-SNAPSHOT/hive-exec-0.13.0-SNAPSHOT.jar >> org.apache.hadoop.hive.ql.exec.mr.ExecDriver -plan >> file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml >> -jobconffile >> file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10002/jobconf.xml >> 2014-02-06 12:17:15,431 ERROR exec.Task (SessionState.java:printError(513)) >> - Execution failed with exit status: 2 >> 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) >> - Obtaining error information >> 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) - >> Task failed! >> Task ID: >> Stage-1 >> >> Logs: >> >> 2014-02-06 12:17:15,433 ERROR exec.Task (SessionState.java:printError(513)) >> - /Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log >> 2014-02-06 12:17:15,433 ERROR mr.ExecDriver (MapRedTask.java:execute(280)) - >> Execution failed with exit status: 2 >> >> Still, at the end of running the test there is nothing left in >> target/tmp/localscratchdir, scratchdir or in target/tmp/hadoop-tmp. In this >> case, I'm missing the output from the MR job that failed. >> Is there a way for me to get this missing output? >> >> Thanks, >> ~Remus >> >> >> >> > > > > -- > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
