I am copying from Site1 to Site2 in Amazon EC2. Each site has mounted 1
hadoop instance with 1 NN and 2 DNs. I think my problem is something
related distcp command being unable to query one of the HDFS' clusters'
DNs like Harsh told. Amazon EC2 use private IPs to communicate between
hosts in the same site and public IPs to connect between sites. I am
using distcp to copy big data (256MB each file) between different sites
using the public IP:
hadoop distcp hdfs://publicIP1:9000/wiki hdfs://publicIP2:9000/wiki
Checking the logs (hadoop-ubuntu-namenode-ip-XX-XXX-XXX-94.log), I also
got this problem but I don't know if it is related:
2013-08-28 07:40:17,537 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 9000, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.abandonBlock from
XX.XXX.XXX.150:51844: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
on /wiki/.distcp.tmp.attempt_1377674569447_0001_m_000000_1: File does
not exist. H
older DFSClient_attempt_1377674569447_0001_m_000000_1_-1864394963_1 does
not have any open files.
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
on /wiki/.distcp.tmp.attempt_1377674569447_0001_m_000000_1: File does
not exist. Holder DFSClient_
attempt_1377674569447_0001_m_000000_1_-1864394963_1 does not have any
open files.
Here is the report:
Site1
$ hdfs dfsadmin -report
13/08/28 11:17:52 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Configured Capacity: 25365413888 (23.62 GB)
Present Capacity: 20488749056 (19.08 GB)
DFS Remaining: 20487512064 (19.08 GB)
DFS Used: 1236992 (1.18 MB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)
Live datanodes:
Name: XX.XXX.XXX.243:50010 (ip-XX-XXX-XXX-243.eu-west-1.compute.internal)
Hostname: ip-XX-XXX-XXX-243.eu-west-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 618496 (604 KB)
Non DFS Used: 2489556992 (2.32 GB)
DFS Remaining: 10192531456 (9.49 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.37%
Last contact: Wed Aug 28 11:17:51 UTC 2013
Name: XX.XXX.XXX.58:50010 (ip-XX-XXX-XXX-58.eu-west-1.compute.internal)
Hostname: ip-XX-XXX-XXX-58.eu-west-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 618496 (604 KB)
Non DFS Used: 2387107840 (2.22 GB)
DFS Remaining: 10294980608 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.17%
Last contact: Wed Aug 28 11:17:51 UTC 2013
Site2:
$ hdfs dfsadmin -report
13/08/28 11:17:55 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Configured Capacity: 25365413888 (23.62 GB)
Present Capacity: 20593844224 (19.18 GB)
DFS Remaining: 20593770496 (19.18 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)
Live datanodes:
Name: XX.XXX.XXX.95:50010 (ip-XX-XXX-XXX-95.ap-southeast-1.compute.internal)
Hostname: ip-XX-XXX-XXX-95.ap-southeast-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 2385768448 (2.22 GB)
DFS Remaining: 10296901632 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013
Name: XX.XXX.XXX.96:50010 (ip-XX-XXX-XXX-96.ap-southeast-1.compute.internal)
Hostname: ip-XX-XXX-XXX-96.ap-southeast-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 2385801216 (2.22 GB)
DFS Remaining: 10296868864 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013
Any suggestion to fix this problem?
On 08/28/2013 12:09 PM, Jitendra Yadav wrote:
Hi,
Also can you please share the dfs heath check report of your cluster?
Thanks
On Wed, Aug 28, 2013 at 3:46 PM, xeon <[email protected]
<mailto:[email protected]>> wrote:
Hi,
I don't have the "dfs.hosts.exclude" property defined, but I
still get the error "There are 2 datanode(s) running and 2 node(s)
are excluded in this operation." when I run the distcp command.
Any help?
