Umsubscribe 2012. 8. 29. 오전 12:44에 "Tao" <[email protected]>님이 작성:
> Hi, all**** > > I use distcp copying data from hadoop1.0.3 to hadoop 2.0.1.**** > > When the file path(or file name) contain Chinese character, an > exception will throw. Like below. I need some help about this.**** > > Thanks.**** > > **** > > ** ** > > ** ** > > ** ** > > [hdfs@host ~]$ hadoop distcp -i -prbugp -m 14 -overwrite -log > /tmp/distcp.log hftp://10.xx.xx.aa:50070/tmp/中文路径测试 hdfs:// > 10.xx.xx.bb:54310/tmp/distcp_test14**** > > 12/08/28 23:32:31 INFO tools.DistCp: Input Options: > DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, > ignoreFailures=true, maxMaps=14, sslConfigurationFile='null', > copyStrategy='uniformsize', sourceFileListing=null, > sourcePaths=[hftp://10.xx.xx.aa:50070/tmp/中文路径测试], targetPath=hdfs:// > 10.xx.xx.bb:54310/tmp/distcp_test14}**** > > 12/08/28 23:32:33 INFO tools.DistCp: DistCp job log path: /tmp/distcp.log* > *** > > 12/08/28 23:32:34 WARN conf.Configuration: io.sort.mb is deprecated. > Instead, use mapreduce.task.io.sort.mb**** > > 12/08/28 23:32:34 WARN conf.Configuration: io.sort.factor is deprecated. > Instead, use mapreduce.task.io.sort.factor**** > > 12/08/28 23:32:34 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable** > ** > > 12/08/28 23:32:36 INFO mapreduce.JobSubmitter: number of splits:1**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.jar is deprecated. > Instead, use mapreduce.job.jar**** > > 12/08/28 23:32:36 WARN conf.Configuration: > mapred.map.tasks.speculative.execution is deprecated. Instead, use > mapreduce.map.speculative**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.reduce.tasks is > deprecated. Instead, use mapreduce.job.reduces**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.mapoutput.value.class is > deprecated. Instead, use mapreduce.map.output.value.class**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.map.class is > deprecated. Instead, use mapreduce.job.map.class**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.job.name is deprecated. > Instead, use mapreduce.job.name**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.inputformat.class is > deprecated. Instead, use mapreduce.job.inputformat.class**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.output.dir is > deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.outputformat.class is > deprecated. Instead, use mapreduce.job.outputformat.class**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.mapoutput.key.class is > deprecated. Instead, use mapreduce.map.output.key.class**** > > 12/08/28 23:32:36 WARN conf.Configuration: mapred.working.dir is > deprecated. Instead, use mapreduce.job.working.dir**** > > 12/08/28 23:32:37 INFO mapred.ResourceMgrDelegate: Submitted application > application_1345831938927_0039 to ResourceManager at baby20/10.1.1.40:8040 > **** > > 12/08/28 23:32:37 INFO mapreduce.Job: The url to track the job: > http://baby20:8088/proxy/application_1345831938927_0039/**** > > 12/08/28 23:32:37 INFO tools.DistCp: DistCp job-id: job_1345831938927_0039 > **** > > 12/08/28 23:32:37 INFO mapreduce.Job: Running job: job_1345831938927_0039* > *** > > 12/08/28 23:32:50 INFO mapreduce.Job: Job job_1345831938927_0039 running > in uber mode : false**** > > 12/08/28 23:32:50 INFO mapreduce.Job: map 0% reduce 0%**** > > 12/08/28 23:33:00 INFO mapreduce.Job: map 100% reduce 0%**** > > 12/08/28 23:33:00 INFO mapreduce.Job: Task Id : > attempt_1345831938927_0039_m_000000_0, Status : FAILED**** > > Error: java.io.IOException: File copy failed: hftp://10.1.1.26:50070/tmp/ > 中文路径测试/part-r-00017 --> hdfs:// > 10.1.1.40:54310/tmp/distcp_test14/part-r-00017**** > > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) > **** > > at > org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)**** > > at > org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)**** > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)**** > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) > **** > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)**** > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)*** > * > > at java.security.AccessController.doPrivileged(Native Method)**** > > at javax.security.auth.Subject.doAs(Subject.java:396)**** > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > **** > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)**** > > Caused by: java.io.IOException: Couldn't run retriable-command: Copying > hftp://10.1.1.26:50070/tmp/中文路径测试/part-r-00017 to hdfs:// > 10.1.1.40:54310/tmp/distcp_test14/part-r-00017**** > > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) > **** > > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) > **** > > ... 10 more**** > > Caused by: > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: > java.io.IOException: HTTP_OK expected, received 500**** > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:201) > **** > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:167) > **** > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:112) > **** > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:90) > **** > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:71) > **** > > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) > **** > > ... 11 more**** > > Caused by: java.io.IOException: HTTP_OK expected, received 500**** > > at > org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderInputStream.checkResponseCode(HftpFileSystem.java:381) > **** > > at > org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:121) > **** > > at > org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) > **** > > at > org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158) > **** > > at java.io.DataInputStream.read(DataInputStream.java:132)**** > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > **** > > at java.io.BufferedInputStream.read(BufferedInputStream.java:317)* > *** > > at java.io.FilterInputStream.read(FilterInputStream.java:90)**** > > at > org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:70) > **** > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:198) > **** > > ... 16 more**** > > ** ** > > ** ** > > ** ** >
