Maybe, API is backward compatible. Try to run the same command on different node (it you ran in on mc00001, try mc00000)
On Tue, Feb 8, 2011 at 8:50 PM, Korb, Michael [USA] <[email protected]> wrote: > I was unable to get the stacktrace. Is there a workaround for the > incompatible APIs? I'm using hftp instead of hdfs because the DistCp guide > (http://hadoop.apache.org/common/docs/r0.20.2/distcp.html) says, "For copying > between two different versions of Hadoop, one will usually use > HftpFileSystem. This is a read-only FileSystem, so DistCp must be run on the > destination cluster (more specifically, on TaskTrackers that can write to the > destination cluster). Each source is specified as > hftp://<dfs.http.address>/<path> (the default dfs.http.address is > <namenode>:50070)." > > Mike > ________________________________________ > From: Vladimir Klimontovich [[email protected]] > Sent: Tuesday, February 08, 2011 12:48 PM > To: [email protected] > Subject: Re: copying between hadoop instances > > Yes, new APIs between old and new version are incompatible. > > Did you managed to get stacktrace from > http://mc00000.mcloud.bah.com:50075/streamFile?filename=/user/cluster/annotated/2009/07/05/_logs/history/mc00002_1291306280950_job_201012021111_0518_cluster_com.bah.mapred.CombineFilesDriver%253A+netflow-smallfi&ugi=hdfs > > And, by the way, why are you using htfp for source instead of hdfs://? > > > > On Tue, Feb 8, 2011 at 8:45 PM, Korb, Michael [USA] > <[email protected]> wrote: >> That address is to a file on the destination fs, but it didn't get copied >> from the source. That is where fs -cp fails every time. Here's what happens >> when I try distcp: >> >> sudo -u hdfs ./hadoop distcp -update hftp://mc00001:50070/ >> hdfs://mc00000:55310/ >> >> 11/02/08 12:38:50 INFO tools.DistCp: srcPaths=[hftp://mc00001:50070/] >> 11/02/08 12:38:50 INFO tools.DistCp: destPath=hdfs://mc00000:55310/ >> Exception in thread "main" java.lang.NoSuchMethodError: >> org.apache.hadoop.mapred.JobConf.getCredentials()Lorg/apache/hadoop/security/Credentials; >> at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:632) >> at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656) >> at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.hadoop.tools.DistCp.main(DistCp.java:908) >> >> I asked about this before and got a response from Ted Dunning. He said, >> "This is due to the security API not being available. You are crossing from >> a cluster with security to one without and that is causing confusion. >> Presumably your client assumes that it is available and your hadoop library >> doesn't provide it. Check your class path very carefully looking for version >> assumptions and confusions." >> >> I don't know where to begin checking my class path for these things... but >> perhaps if I could get distcp working it wouldn't run into the same problems >> as fs -cp. >> >> Thanks, >> Mike >> ________________________________________ >> From: Vladimir Klimontovich [[email protected]] >> Sent: Tuesday, February 08, 2011 12:24 PM >> To: [email protected] >> Subject: Re: copying between hadoop instances >> >> Try to go to >> http://mc00000.mcloud.bah.com:50075/streamFile?filename=/user/cluster/annotated/2009/07/05/_logs/history/mc00002_1291306280950_job_201012021111_0518_cluster_com.bah.mapred.CombineFilesDriver%253A+netflow-smallfi&ugi=hdfs >> and check if browser show you the stacktrace. If could give a lot of >> information. >> >> And what's wrong with distcp (any stacktraces, error messages?) >> >> On Tue, Feb 8, 2011 at 8:06 PM, Korb, Michael [USA] >> <[email protected]> wrote: >>> I have two Hadoop instances running on one cluster of machines for the >>> purpose of upgrading. I'm trying to copy all the files from the old >>> instance to the new one but have been having trouble with both distcp and >>> fs -cp. >>> >>> Most recently, I've been trying, "sudo -u hdfs ./hadoop fs -cp >>> hftp://mc00001:50070/* hdfs://mc00000:55310/" where mc00001 is the namenode >>> of old hadoop and mc00000 is the namenode of new hadoop. >>> >>> I've had some success with this command (some files have actually been >>> copied), but part of the way through the copy, I get this error: >>> >>> cp: Server returned HTTP response code: 500 for URL: >>> http://mc00000.mcloud.bah.com:50075/streamFile?filename=/user/cluster/annotated/2009/07/05/_logs/history/mc00002_1291306280950_job_201012021111_0518_cluster_com.bah.mapred.CombineFilesDriver%253A+netflow-smallfi&ugi=hdfs >>> >>> Is it possible that there could be permissions issues? It also doesn't seem >>> quite right to be copying * since there are directories, but I don't think >>> there's a way to call fs -cp recursively. Could this be causing problems? >>> >>> Thanks, >>> Mike >> >> >> >> -- >> Vladimir Klimontovich >> Cell: +7-926-890-2349, skype: klimontovich >> > > > > -- > Vladimir Klimontovich > Cell: +7-926-890-2349, skype: klimontovich > -- Vladimir Klimontovich Cell: +7-926-890-2349, skype: klimontovich
