I was unable to get the stacktrace. Is there a workaround for the incompatible APIs? I'm using hftp instead of hdfs because the DistCp guide (http://hadoop.apache.org/common/docs/r0.20.2/distcp.html) says, "For copying between two different versions of Hadoop, one will usually use HftpFileSystem. This is a read-only FileSystem, so DistCp must be run on the destination cluster (more specifically, on TaskTrackers that can write to the destination cluster). Each source is specified as hftp://<dfs.http.address>/<path> (the default dfs.http.address is <namenode>:50070)."
Mike ________________________________________ From: Vladimir Klimontovich [[email protected]] Sent: Tuesday, February 08, 2011 12:48 PM To: [email protected] Subject: Re: copying between hadoop instances Yes, new APIs between old and new version are incompatible. Did you managed to get stacktrace from http://mc00000.mcloud.bah.com:50075/streamFile?filename=/user/cluster/annotated/2009/07/05/_logs/history/mc00002_1291306280950_job_201012021111_0518_cluster_com.bah.mapred.CombineFilesDriver%253A+netflow-smallfi&ugi=hdfs And, by the way, why are you using htfp for source instead of hdfs://? On Tue, Feb 8, 2011 at 8:45 PM, Korb, Michael [USA] <[email protected]> wrote: > That address is to a file on the destination fs, but it didn't get copied > from the source. That is where fs -cp fails every time. Here's what happens > when I try distcp: > > sudo -u hdfs ./hadoop distcp -update hftp://mc00001:50070/ > hdfs://mc00000:55310/ > > 11/02/08 12:38:50 INFO tools.DistCp: srcPaths=[hftp://mc00001:50070/] > 11/02/08 12:38:50 INFO tools.DistCp: destPath=hdfs://mc00000:55310/ > Exception in thread "main" java.lang.NoSuchMethodError: > org.apache.hadoop.mapred.JobConf.getCredentials()Lorg/apache/hadoop/security/Credentials; > at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:632) > at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:908) > > I asked about this before and got a response from Ted Dunning. He said, "This > is due to the security API not being available. You are crossing from a > cluster with security to one without and that is causing confusion. > Presumably your client assumes that it is available and your hadoop library > doesn't provide it. Check your class path very carefully looking for version > assumptions and confusions." > > I don't know where to begin checking my class path for these things... but > perhaps if I could get distcp working it wouldn't run into the same problems > as fs -cp. > > Thanks, > Mike > ________________________________________ > From: Vladimir Klimontovich [[email protected]] > Sent: Tuesday, February 08, 2011 12:24 PM > To: [email protected] > Subject: Re: copying between hadoop instances > > Try to go to > http://mc00000.mcloud.bah.com:50075/streamFile?filename=/user/cluster/annotated/2009/07/05/_logs/history/mc00002_1291306280950_job_201012021111_0518_cluster_com.bah.mapred.CombineFilesDriver%253A+netflow-smallfi&ugi=hdfs > and check if browser show you the stacktrace. If could give a lot of > information. > > And what's wrong with distcp (any stacktraces, error messages?) > > On Tue, Feb 8, 2011 at 8:06 PM, Korb, Michael [USA] > <[email protected]> wrote: >> I have two Hadoop instances running on one cluster of machines for the >> purpose of upgrading. I'm trying to copy all the files from the old instance >> to the new one but have been having trouble with both distcp and fs -cp. >> >> Most recently, I've been trying, "sudo -u hdfs ./hadoop fs -cp >> hftp://mc00001:50070/* hdfs://mc00000:55310/" where mc00001 is the namenode >> of old hadoop and mc00000 is the namenode of new hadoop. >> >> I've had some success with this command (some files have actually been >> copied), but part of the way through the copy, I get this error: >> >> cp: Server returned HTTP response code: 500 for URL: >> http://mc00000.mcloud.bah.com:50075/streamFile?filename=/user/cluster/annotated/2009/07/05/_logs/history/mc00002_1291306280950_job_201012021111_0518_cluster_com.bah.mapred.CombineFilesDriver%253A+netflow-smallfi&ugi=hdfs >> >> Is it possible that there could be permissions issues? It also doesn't seem >> quite right to be copying * since there are directories, but I don't think >> there's a way to call fs -cp recursively. Could this be causing problems? >> >> Thanks, >> Mike > > > > -- > Vladimir Klimontovich > Cell: +7-926-890-2349, skype: klimontovich > -- Vladimir Klimontovich Cell: +7-926-890-2349, skype: klimontovich
