Dear, How to set different names for the local output directory for each job ? Does SDK support this ?
tks Best Regards Weibo: http://weibo.com/guowee Web: http://www.wbkit.com ----------------------------------------------------- WesternBridge Tech: Professional software service provider. Professional is MANNER as well CAPABILITY. On 2013-3-27, at 上午10:58, ramkrishna vasudevan <[email protected]> wrote: > Interesting. Need to check this. > May be we should configure different names for the local output directory > for each job. By any chance both jobs are writing to the same path? > > Regards > Ram > > On Wed, Mar 27, 2013 at 6:44 AM, GuoWei <[email protected]> wrote: > >> Dear JM, >> >> It's correct. >> >> The Hbase version is 0.94.2 and the hadoop version is 0.20.2 / 1.04。 >> >> We test this on both hadoop version 0.20.2 and 1.04. >> >> The error still there. >> >> Thanks a lot >> >> >> >> Best Regards / 商祺 >> 郭伟 Guo Wei >> ----------------------------------------------------- >> 南京西桥科技有限公司 >> Western Bridge Tech Ltd., Nanjing >> >> 南京市玄武区花园路8号一号楼511 >> No. 511, Building 1, No. 8, Hua Yuan Road >> >> Xuanwu District, Nanjing, PR China >> >> Email: [email protected] >> >> Tel: +86 25 8528 4900 (Operator) >> Mobile: +86 138 1589 8257 >> Fax: +86 25 8528 4980 >> >> Weibo: http://weibo.com/guowee >> Web: http://www.wbkit.com >> ----------------------------------------------------- >> WesternBridge Tech: Professional software service provider. Professional >> is MANNER as well CAPABILITY. >> >> On 2013-3-26, at 下午9:18, Jean-Marc Spaggiari <[email protected]> >> wrote: >> >>> Hi, >>> >>> So basically, you have one job which is reading from A and writing to >>> B, and one wich is reading from A and writing to C, and the 2 jobs >>> are running at the same time. Is that correct? Are you able to >>> reproduce that each time you are running the job? Which HBased and >>> Hadoop versions are you running? >>> >>> JM >>> >>> 2013/3/26 GuoWei <[email protected]>: >>>> Dear, >>>> >>>> When I run two MR Jobs which will read same Hbase table and write to >> another same Hbase table at the same time. The result is one job successful >> finished. And another job crashed. And The following shows the error log. >>>> >>>> Please help me to find out why ? >>>> >>>> >>>> <2013-03-25 15:50:34,026> <INFO > <org.apache.hadoop.mapred.JobClient> >> - map 0% reduce 0%(JobClient.java:monitorAndPrintJob:1301) >>>> <2013-03-25 15:50:36,096> <WARN > <org.apache.hadoop.mapred.Task> - >> Could not find output size (Task.java:calculateOutputSize:948) >>>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find >> output/file.out in any of the configured local directories >>>> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) >>>> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) >>>> at >> org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) >>>> at >> org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) >>>> at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) >>>> at org.apache.hadoop.mapred.Task.done(Task.java:875) >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) >>>> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) >>>> <2013-03-25 15:50:36,100> <INFO > >> <org.apache.hadoop.mapred.LocalJobRunner> - >> (LocalJobRunner.java:statusUpdate:321) >>>> <2013-03-25 15:50:36,102> <INFO > <org.apache.hadoop.mapred.Task> - >> Task 'attempt_local_0001_m_000000_0' done.(Task.java:sendDone:959) >>>> <2013-03-25 15:50:36,111> <WARN > >> <org.apache.hadoop.mapred.FileOutputCommitter> - Output path is null in >> cleanup(FileOutputCommitter.java:cleanupJob:100) >>>> <2013-03-25 15:50:36,111> <WARN > >> <org.apache.hadoop.mapred.LocalJobRunner> - >> job_local_0001(LocalJobRunner.java:run:298) >>>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find >> output/file.out in any of the configured local directories >>>> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) >>>> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) >>>> at >> org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) >>>> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:236) >>>> <2013-03-25 15:50:37,029> <INFO > <org.apache.hadoop.mapred.JobClient> >> - map 100% reduce 0%(JobClient.java:monitorAndPrintJob:1301) >>>> <2013-03-25 15:50:37,030> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Job complete: job_local_0001(JobClient.java:monitorAndPrintJob:1356) >>>> <2013-03-25 15:50:37,031> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Counters: 15(Counters.java:log:585) >>>> <2013-03-25 15:50:37,031> <INFO > <org.apache.hadoop.mapred.JobClient> >> - File Input Format Counters (Counters.java:log:587) >>>> <2013-03-25 15:50:37,031> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Bytes Read=0(Counters.java:log:589) >>>> <2013-03-25 15:50:37,032> <INFO > <org.apache.hadoop.mapred.JobClient> >> - FileSystemCounters(Counters.java:log:587) >>>> <2013-03-25 15:50:37,032> <INFO > <org.apache.hadoop.mapred.JobClient> >> - FILE_BYTES_READ=10294950(Counters.java:log:589) >>>> <2013-03-25 15:50:37,033> <INFO > <org.apache.hadoop.mapred.JobClient> >> - FILE_BYTES_WRITTEN=10432139(Counters.java:log:589) >>>> <2013-03-25 15:50:37,033> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Map-Reduce Framework(Counters.java:log:587) >>>> <2013-03-25 15:50:37,033> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Map output materialized bytes=4006(Counters.java:log:589) >>>> <2013-03-25 15:50:37,034> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Combine output records=0(Counters.java:log:589) >>>> <2013-03-25 15:50:37,034> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Map input records=500(Counters.java:log:589) >>>> <2013-03-25 15:50:37,035> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Physical memory (bytes) snapshot=0(Counters.java:log:589) >>>> <2013-03-25 15:50:37,035> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Spilled Records=500(Counters.java:log:589) >>>> <2013-03-25 15:50:37,035> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Map output bytes=3000(Counters.java:log:589) >>>> <2013-03-25 15:50:37,036> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Total committed heap usage (bytes)=202702848(Counters.java:log:589) >>>> <2013-03-25 15:50:37,036> <INFO > <org.apache.hadoop.mapred.JobClient> >> - CPU time spent (ms)=0(Counters.java:log:589) >>>> <2013-03-25 15:50:37,037> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Virtual memory (bytes) snapshot=0(Counters.java:log:589) >>>> <2013-03-25 15:50:37,037> <INFO > <org.apache.hadoop.mapred.JobClient> >> - SPLIT_RAW_BYTES=105(Counters.java:log:589) >>>> <2013-03-25 15:50:37,038> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Map output records=500(Counters.java:log:589) >>>> <2013-03-25 15:50:37,038> <INFO > <org.apache.hadoop.mapred.JobClient> >> - Combine input >>>> >>>> Thanks a lot. >>>> >>>> >>>> >>>> Best Regards >>>> >>>> Weibo: http://weibo.com/guowee >>>> Web: http://www.wbkit.com >>>> ----------------------------------------------------- >>>> WesternBridge Tech: Professional software service provider. >> Professional is MANNER as well CAPABILITY. >>>> >>> >> >>
