Thats cool. On Sep 28, 2015 8:44 PM, "Daniel Watrous" <dwmaill...@gmail.com> wrote:
> After talking with the Vagrant community I decided I was being too clever > trying to run the datanodes on a separate subnet from the master node. I > changed my configuration to have all three hosts on the same subnet and > everything works just as expected. > > Thanks for all your help and input. > > On Mon, Sep 28, 2015 at 9:07 AM, Daniel Watrous <dwmaill...@gmail.com> > wrote: > >> Vinay, >> >> There is no gateway to 51.*. These are IP addresses that I set in my >> Vagrantfile for virtualbox as part of a private network: >> master.vm.network "private_network", ip: 192.168.51.4 >> >> This allows me to spin up all the hosts for my cluster automatically and >> know that they always have the same IP addresses. >> >> From hadoop-data1 (192.168.52.4) I have unrestricted access to >> hadoop-master (192.168.51.4) >> >> hadoop@hadoop-data1:~$ ifconfig >> eth1 Link encap:Ethernet HWaddr 08:00:27:b9:55:25 >> inet addr:192.168.52.4 Bcast:192.168.52.255 Mask:255.255.255.0 >> hadoop@hadoop-data1:~$ ping hadoop-master >> PING hadoop-master (192.168.51.4) 56(84) bytes of data. >> 64 bytes from hadoop-master (192.168.51.4): icmp_seq=1 ttl=63 time=3.13 ms >> 64 bytes from hadoop-master (192.168.51.4): icmp_seq=2 ttl=63 time=2.72 ms >> >> I'm not sure I understand exactly what you're asking for, but from the >> master I can run this >> >> vagrant@hadoop-master:~$ sudo netstat -tnulp | grep 54310 >> tcp 0 0 0.0.0.0:54310 0.0.0.0:* >> LISTEN 22944/java >> >> I understand what you're saying about a gateway often existing at that >> address for a subnet. I'm not familiar enough with Vagrant to answer this >> right now, but I will put in a question there. >> >> I can also change the other two IP addresses to be on the same 51. >> subnet. I may try that next. >> >> >> >> On Mon, Sep 28, 2015 at 8:33 AM, Vinayakumar B <vinayakum...@apache.org> >> wrote: >> >>> 192.168.51.1 might be gateway to 51.* subnet right? >>> >>> Can you verify whether connections from outside 51 subnet, to 51.4 >>> machine using other subnet IP as remote IP. ? >>> >>> You can create any connection, may not be namenode-datanode. >>> >>> for ex: Connection from 192.168.52.4 dn to 192.168.51.4 namenode should >>> result in following, when checked using netstat command in namenode >>> machine. "netstat -tnulp | grep <NN_RPC_PORT>" >>> >>> Output should be something like below >>> >>> tcp 0 0 192.168.51.4:54310 192.168.52.4:32567 >>> LISTEN - >>> >>> >>> If the Foreign Ip is listing as 192.168.51.1 instead of 192.168.52.4, >>> then the gateway, is not passing original client IP forward, its >>> re-creating connections with its own IP. in such case problem will be with >>> the gateway. >>> >>> Its just a guess, reality could be different. >>> >>> please check and let me know. >>> >>> -Vinay >>> >>> On Mon, Sep 28, 2015 at 6:45 PM, Daniel Watrous <dwmaill...@gmail.com> >>> wrote: >>> >>>> Thanks to Namikaze pointing out that I should have sent the namenode >>>> log as a pastbin >>>> >>>> http://pastebin.com/u33bBbgu >>>> >>>> >>>> On Mon, Sep 28, 2015 at 8:02 AM, Daniel Watrous <dwmaill...@gmail.com> >>>> wrote: >>>> >>>>> I have posted the namenode logs here: >>>>> https://gist.github.com/dwatrous/dafaa7695698f36a5d93 >>>>> >>>>> Thanks for all the help. >>>>> >>>>> On Sun, Sep 27, 2015 at 10:28 AM, Brahma Reddy Battula < >>>>> brahmareddy.batt...@hotmail.com> wrote: >>>>> >>>>>> Thanks for sharing the logs. >>>>>> >>>>>> Problem is interesting..can you please post namenode logs and dual IP >>>>>> configurations(thinking problem with gateway while sending requests from >>>>>> 52.1 segment to 51.1 segment..) >>>>>> >>>>>> Thanks And Regards >>>>>> Brahma Reddy Battula >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> Date: Fri, 25 Sep 2015 12:19:00 -0500 >>>>>> >>>>>> Subject: Re: Problem running example (wrong IP address) >>>>>> From: dwmaill...@gmail.com >>>>>> To: user@hadoop.apache.org >>>>>> >>>>>> hadoop-master http://pastebin.com/yVF8vCYS >>>>>> hadoop-data1 http://pastebin.com/xMEdf01e >>>>>> hadoop-data2 http://pastebin.com/prqd02eZ >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 25, 2015 at 11:53 AM, Brahma Reddy Battula < >>>>>> brahmareddy.batt...@hotmail.com> wrote: >>>>>> >>>>>> sorry,I am not able to access the logs, could please post in paste >>>>>> bin or attach the 192.168.51.6( as your query is why different IP) >>>>>> DN logs and namenode logs here..? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thanks And Regards >>>>>> Brahma Reddy Battula >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> Date: Fri, 25 Sep 2015 11:16:55 -0500 >>>>>> Subject: Re: Problem running example (wrong IP address) >>>>>> From: dwmaill...@gmail.com >>>>>> To: user@hadoop.apache.org >>>>>> >>>>>> >>>>>> Brahma, >>>>>> >>>>>> Thanks for the reply. I'll keep this conversation here in the user >>>>>> list. The /etc/hosts file is identical on all three nodes >>>>>> >>>>>> hadoop@hadoop-data1:~$ cat /etc/hosts >>>>>> 127.0.0.1 localhost >>>>>> 192.168.51.4 hadoop-master >>>>>> 192.168.52.4 hadoop-data1 >>>>>> 192.168.52.6 hadoop-data2 >>>>>> >>>>>> hadoop@hadoop-data2:~$ cat /etc/hosts >>>>>> 127.0.0.1 localhost >>>>>> 192.168.51.4 hadoop-master >>>>>> 192.168.52.4 hadoop-data1 >>>>>> 192.168.52.6 hadoop-data2 >>>>>> >>>>>> hadoop@hadoop-master:~$ cat /etc/hosts >>>>>> 127.0.0.1 localhost >>>>>> 192.168.51.4 hadoop-master >>>>>> 192.168.52.4 hadoop-data1 >>>>>> 192.168.52.6 hadoop-data2 >>>>>> >>>>>> Here are the startup logs for all three nodes: >>>>>> https://gist.github.com/dwatrous/7241bb804a9be8f9303f >>>>>> https://gist.github.com/dwatrous/bcd85cda23d6eca3a68b >>>>>> https://gist.github.com/dwatrous/922c4f773aded0137fa3 >>>>>> >>>>>> Thanks for your help. >>>>>> >>>>>> >>>>>> On Fri, Sep 25, 2015 at 10:33 AM, Brahma Reddy Battula < >>>>>> brahmareddy.batt...@huawei.com> wrote: >>>>>> >>>>>> Seems DN started in three machines and failed in >>>>>> hadoop-data1(192.168.52.4).. >>>>>> >>>>>> >>>>>> 192.168.51.6 : giving IP as 192.168.51.1 >>>>>> <http://192.168.51.1:50010>...can >>>>>> you please check /etc/hosts file of 192.168.51.6 (might be >>>>>> 192.168.51.1 <http://192.168.51.1:50010> is configured in /etc/hosts) >>>>>> >>>>>> 192.168.52.4 : datanode startup might be failed ( you can check this >>>>>> node logs) >>>>>> >>>>>> 192.168.51.4 : <http://192.168.51.4:50010> Datanode starup is >>>>>> success..which is in master node.. >>>>>> >>>>>> >>>>>> >>>>>> Thanks & Regards >>>>>> Brahma Reddy Battula >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From:* Daniel Watrous [dwmaill...@gmail.com] >>>>>> *Sent:* Friday, September 25, 2015 8:41 PM >>>>>> *To:* user@hadoop.apache.org >>>>>> *Subject:* Re: Problem running example (wrong IP address) >>>>>> >>>>>> I'm still stuck on this and posted it to stackoverflow: >>>>>> >>>>>> http://stackoverflow.com/questions/32785256/hadoop-datanode-binds-wrong-ip-address >>>>>> >>>>>> Thanks, >>>>>> Daniel >>>>>> >>>>>> On Fri, Sep 25, 2015 at 8:28 AM, Daniel Watrous <dwmaill...@gmail.com >>>>>> > wrote: >>>>>> >>>>>> I could really use some help here. As you can see from the output >>>>>> below, the two attached datanodes are identified with a non-existent IP >>>>>> address. Can someone tell me how that gets selected or how to explicitly >>>>>> set it. Also, why are both datanodes shown under the same name/IP? >>>>>> >>>>>> hadoop@hadoop-master:~$ hdfs dfsadmin -report >>>>>> Configured Capacity: 84482326528 (78.68 GB) >>>>>> Present Capacity: 75745546240 (70.54 GB) >>>>>> DFS Remaining: 75744862208 (70.54 GB) >>>>>> DFS Used: 684032 (668 KB) >>>>>> DFS Used%: 0.00% >>>>>> Under replicated blocks: 0 >>>>>> Blocks with corrupt replicas: 0 >>>>>> Missing blocks: 0 >>>>>> Missing blocks (with replication factor 1): 0 >>>>>> >>>>>> ------------------------------------------------- >>>>>> Live datanodes (2): >>>>>> >>>>>> Name: 192.168.51.1:50010 (192.168.51.1) >>>>>> Hostname: hadoop-data1 >>>>>> Decommission Status : Normal >>>>>> Configured Capacity: 42241163264 (39.34 GB) >>>>>> DFS Used: 303104 (296 KB) >>>>>> Non DFS Used: 4302479360 (4.01 GB) >>>>>> DFS Remaining: 37938380800 (35.33 GB) >>>>>> DFS Used%: 0.00% >>>>>> DFS Remaining%: 89.81% >>>>>> Configured Cache Capacity: 0 (0 B) >>>>>> Cache Used: 0 (0 B) >>>>>> Cache Remaining: 0 (0 B) >>>>>> Cache Used%: 100.00% >>>>>> Cache Remaining%: 0.00% >>>>>> Xceivers: 1 >>>>>> Last contact: Fri Sep 25 13:25:37 UTC 2015 >>>>>> >>>>>> >>>>>> Name: 192.168.51.4:50010 (hadoop-master) >>>>>> Hostname: hadoop-master >>>>>> Decommission Status : Normal >>>>>> Configured Capacity: 42241163264 (39.34 GB) >>>>>> DFS Used: 380928 (372 KB) >>>>>> Non DFS Used: 4434300928 (4.13 GB) >>>>>> DFS Remaining: 37806481408 (35.21 GB) >>>>>> DFS Used%: 0.00% >>>>>> DFS Remaining%: 89.50% >>>>>> Configured Cache Capacity: 0 (0 B) >>>>>> Cache Used: 0 (0 B) >>>>>> Cache Remaining: 0 (0 B) >>>>>> Cache Used%: 100.00% >>>>>> Cache Remaining%: 0.00% >>>>>> Xceivers: 1 >>>>>> Last contact: Fri Sep 25 13:25:38 UTC 2015 >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 24, 2015 at 5:05 PM, Daniel Watrous <dwmaill...@gmail.com >>>>>> > wrote: >>>>>> >>>>>> The IP address is clearly wrong, but I'm not sure how it gets set. >>>>>> Can someone tell me how to configure it to choose a valid IP address? >>>>>> >>>>>> On Thu, Sep 24, 2015 at 3:26 PM, Daniel Watrous <dwmaill...@gmail.com >>>>>> > wrote: >>>>>> >>>>>> I just noticed that both datanodes appear to have chosen that IP >>>>>> address and bound that port for HDFS communication. >>>>>> >>>>>> http://screencast.com/t/OQNbrWFF >>>>>> >>>>>> Any idea why this would be? Is there some way to specify which >>>>>> IP/hostname should be used for that? >>>>>> >>>>>> On Thu, Sep 24, 2015 at 3:11 PM, Daniel Watrous <dwmaill...@gmail.com >>>>>> > wrote: >>>>>> >>>>>> When I try to run a map reduce example, I get the following error: >>>>>> >>>>>> hadoop@hadoop-master:~$ hadoop jar >>>>>> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar >>>>>> pi 10 30 >>>>>> Number of Maps = 10 >>>>>> Samples per Map = 30 >>>>>> 15/09/24 20:04:28 INFO hdfs.DFSClient: Exception in >>>>>> createBlockOutputStream >>>>>> java.io.IOException: Got error, status message , ack with >>>>>> firstBadLink as 192.168.51.1:50010 >>>>>> at >>>>>> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140) >>>>>> at >>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1334) >>>>>> at >>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >>>>>> at >>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449) >>>>>> 15/09/24 20:04:28 INFO hdfs.DFSClient: Abandoning >>>>>> BP-852923283-127.0.1.1-1443119668806:blk_1073741825_1001 >>>>>> 15/09/24 20:04:28 INFO hdfs.DFSClient: Excluding datanode >>>>>> DatanodeInfoWithStorage[192.168.51.1:50010 >>>>>> ,DS-45f6e06d-752e-41e8-ac25-ca88bce80d00,DISK] >>>>>> 15/09/24 20:04:28 WARN hdfs.DFSClient: Slow waitForAckedSeqno took >>>>>> 65357ms (threshold=30000ms) >>>>>> Wrote input for Map #0 >>>>>> >>>>>> I'm not sure why it's trying to access 192.168.51.1:50010, which >>>>>> isn't even a valid IP address in my setup. >>>>>> >>>>>> Daniel >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
