Re: distcp port for 0.17.2
Hi, The dfs.http.address is for human use, not program interoperability. You can visit http://whatever.address.your.namenode.has:50070 in a web browser and see statistics about your filesystem. The address of cluster 2 is in its fs.default.name. This should be set to something like hdfs://cluster2.master.name:9000/ The file:// protocol only refers to paths on the current machine in its real (non-DFS) filesystem. - Aaron On Wed, Oct 22, 2008 at 3:47 PM, bzheng [EMAIL PROTECTED] wrote: Thanks. The fs.default.name is file:/// and dfs.http.address is 0.0.0.0:50070. I tried: hadoop dfs -ls /path/file to make sure file exists on cluster1 hadoop distcp file:///cluster1_master_node_ip:50070/path/file file:///cluster2_master_node_ip:50070/path/file It gives this error message: 08/10/22 15:43:47 INFO util.CopyFiles: srcPaths=[file:/cluster1_master_node_ip:50070/path/file] 08/10/22 15:43:47 INFO util.CopyFiles: destPath=file:/cluster2_master_node_ip:50070/path/file With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source file:/cluster1_master_node_ip:50070/path/file does not exist. at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) If I use hdfs:// instead of file:///, I get: Copy failed: java.net.SocketTimeoutException: timed out waiting for rpc response at org.apache.hadoop.ipc.Client.call(Client.java:559) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313) at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102) at org.apache.hadoop.dfs.DFSClient.init(DFSClient.java:178) at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175) at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:572) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) s29752-hadoopuser wrote: Hi, There is no such thing called distcp port. distcp uses (generic) file system API and so it does not care about the file system implementation details like port number. It is common to use distcp with HDFS or HFTP. The urls will look like hdfs://namenode:port/path and hftp://namenode:port/path for HDFS and HFTP, respectively. The HDFS and HFTP ports are specified by fs.default.name and dfs.http.address, respectively. Nicholas Sze - Original Message From: bzheng [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Wednesday, October 22, 2008 11:57:43 AM Subject: distcp port for 0.17.2 What's the port number for distcp in 0.17.2? I can't find any documentation on distcp for version 0.17.2. For version 0.18, the documentation says it's 8020. I'm using a standard install and the only open ports associated with hadoop are 50030, 50070, and 50090. None of them work with distcp. So, how do you use distcp in 0.17.2? are there any extra setup/configuration needed? Thanks in advance for your help. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html Sent from the Hadoop core-user mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20121246.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: distcp port for 0.17.2
It's working for me now. Turns out the cluster have multiple network interfaces and I was using the wrong one. Thanks. Aaron Kimball-3 wrote: Hi, The dfs.http.address is for human use, not program interoperability. You can visit http://whatever.address.your.namenode.has:50070 in a web browser and see statistics about your filesystem. The address of cluster 2 is in its fs.default.name. This should be set to something like hdfs://cluster2.master.name:9000/ The file:// protocol only refers to paths on the current machine in its real (non-DFS) filesystem. - Aaron On Wed, Oct 22, 2008 at 3:47 PM, bzheng [EMAIL PROTECTED] wrote: Thanks. The fs.default.name is file:/// and dfs.http.address is 0.0.0.0:50070. I tried: hadoop dfs -ls /path/file to make sure file exists on cluster1 hadoop distcp file:///cluster1_master_node_ip:50070/path/file file:///cluster2_master_node_ip:50070/path/file It gives this error message: 08/10/22 15:43:47 INFO util.CopyFiles: srcPaths=[file:/cluster1_master_node_ip:50070/path/file] 08/10/22 15:43:47 INFO util.CopyFiles: destPath=file:/cluster2_master_node_ip:50070/path/file With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source file:/cluster1_master_node_ip:50070/path/file does not exist. at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) If I use hdfs:// instead of file:///, I get: Copy failed: java.net.SocketTimeoutException: timed out waiting for rpc response at org.apache.hadoop.ipc.Client.call(Client.java:559) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313) at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102) at org.apache.hadoop.dfs.DFSClient.init(DFSClient.java:178) at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175) at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:572) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) s29752-hadoopuser wrote: Hi, There is no such thing called distcp port. distcp uses (generic) file system API and so it does not care about the file system implementation details like port number. It is common to use distcp with HDFS or HFTP. The urls will look like hdfs://namenode:port/path and hftp://namenode:port/path for HDFS and HFTP, respectively. The HDFS and HFTP ports are specified by fs.default.name and dfs.http.address, respectively. Nicholas Sze - Original Message From: bzheng [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Wednesday, October 22, 2008 11:57:43 AM Subject: distcp port for 0.17.2 What's the port number for distcp in 0.17.2? I can't find any documentation on distcp for version 0.17.2. For version 0.18, the documentation says it's 8020. I'm using a standard install and the only open ports associated with hadoop are 50030, 50070, and 50090. None of them work with distcp. So, how do you use distcp in 0.17.2? are there any extra setup/configuration needed? Thanks in advance for your help. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html Sent from the Hadoop core-user mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20121246.html Sent from the Hadoop core-user mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20137577.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
distcp port for 0.17.2
What's the port number for distcp in 0.17.2? I can't find any documentation on distcp for version 0.17.2. For version 0.18, the documentation says it's 8020. I'm using a standard install and the only open ports associated with hadoop are 50030, 50070, and 50090. None of them work with distcp. So, how do you use distcp in 0.17.2? are there any extra setup/configuration needed? Thanks in advance for your help. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: distcp port for 0.17.2
Thanks. The fs.default.name is file:/// and dfs.http.address is 0.0.0.0:50070. I tried: hadoop dfs -ls /path/file to make sure file exists on cluster1 hadoop distcp file:///cluster1_master_node_ip:50070/path/file file:///cluster2_master_node_ip:50070/path/file It gives this error message: 08/10/22 15:43:47 INFO util.CopyFiles: srcPaths=[file:/cluster1_master_node_ip:50070/path/file] 08/10/22 15:43:47 INFO util.CopyFiles: destPath=file:/cluster2_master_node_ip:50070/path/file With failures, global counters are inaccurate; consider running with -i Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source file:/cluster1_master_node_ip:50070/path/file does not exist. at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) If I use hdfs:// instead of file:///, I get: Copy failed: java.net.SocketTimeoutException: timed out waiting for rpc response at org.apache.hadoop.ipc.Client.call(Client.java:559) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313) at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102) at org.apache.hadoop.dfs.DFSClient.init(DFSClient.java:178) at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175) at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:572) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) s29752-hadoopuser wrote: Hi, There is no such thing called distcp port. distcp uses (generic) file system API and so it does not care about the file system implementation details like port number. It is common to use distcp with HDFS or HFTP. The urls will look like hdfs://namenode:port/path and hftp://namenode:port/path for HDFS and HFTP, respectively. The HDFS and HFTP ports are specified by fs.default.name and dfs.http.address, respectively. Nicholas Sze - Original Message From: bzheng [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Wednesday, October 22, 2008 11:57:43 AM Subject: distcp port for 0.17.2 What's the port number for distcp in 0.17.2? I can't find any documentation on distcp for version 0.17.2. For version 0.18, the documentation says it's 8020. I'm using a standard install and the only open ports associated with hadoop are 50030, 50070, and 50090. None of them work with distcp. So, how do you use distcp in 0.17.2? are there any extra setup/configuration needed? Thanks in advance for your help. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html Sent from the Hadoop core-user mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20121246.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
