Please take a look at next hop 5 (nh —get 5) to see which interface it points at (Oif field) and the MAC rewrite information. Also, please check “dropstats” for drop counters.
Raja From: abhishek jain <[email protected]<mailto:[email protected]>> Date: Thursday, September 17, 2015 at 12:20 AM To: Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>> Cc: Prabhjot Singh Sethi <[email protected]<mailto:[email protected]>>, sonia verma <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Vigneshwaran R <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi Rajagopalan Below is the output at compute node with Ip adress 192.168.7.4.. rt --dump 0 --family inet | grep 192.168.7.4 192.168.7.4/32<http://192.168.7.4/32> 32 T - 5 - 192.168.7.40/32<http://192.168.7.40/32> 22 T - 8 - 192.168.7.41/32<http://192.168.7.41/32> 22 T - 8 - 192.168.7.42/32<http://192.168.7.42/32> 22 T - 8 - 192.168.7.43/32<http://192.168.7.43/32> 22 T - 8 - 192.168.7.44/32<http://192.168.7.44/32> 22 T - 8 - 192.168.7.45/32<http://192.168.7.45/32> 22 T - 8 - 192.168.7.46/32<http://192.168.7.46/32> 22 T - 8 - 192.168.7.47/32<http://192.168.7.47/32> 22 T - 8 - 192.168.7.48/32<http://192.168.7.48/32> 22 T - 8 - 192.168.7.49/32<http://192.168.7.49/32> 22 T - 8 - On Wed, Sep 16, 2015 at 10:05 PM, Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>> wrote: Please try “rt —dump 0 —family inet | grep <ip addr>”. Also, please run “dropstats” to see if any drop counters are going up in vrouter. Raja From: abhishek jain <[email protected]<mailto:[email protected]>> Date: Wednesday, September 16, 2015 at 8:18 AM To: Prabhjot Singh Sethi <[email protected]<mailto:[email protected]>> Cc: sonia verma <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Vigneshwaran R <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi Prabhjot Thanks for your help. The command curl http://localhost:8085/Snh_Inet4UcRouteReq?x=0 is not working on the compute node. Is there any other way/utility to see receive route? On Wed, Sep 16, 2015 at 8:39 PM, Prabhjot Singh Sethi <[email protected]<mailto:[email protected]>> wrote: While contrail-vrouter-agent is not running running kernel vrouter module might be putting physical interface to cross-connect mode Due to which you see ping being functional, however when it is taken out of cross-connect mode (by contrail-vrouter-agent) it might be becoming non-functional due to missing routes can you due following Check agent using "curl http://localhost:8085/Snh_Inet4UcRouteReq?x=0"; And check vrouter using rt and nh utility to check if receive route is installed. Assuming your config file is fine,agent should have installed receive route for vhost ip in fabric vrf (index 0). — Prabhjot From: abhishek jain <[email protected]<mailto:[email protected]>> Date: Wednesday, 16 September 2015 7:51 pm To: Prabhjot Singh Sethi <[email protected]<mailto:[email protected]>> Cc: Vigneshwaran R <[email protected]<mailto:[email protected]>>, Pedro Marques <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi Prabhjot Thanks for the reply. Once we are running contrail-vrouter-agent on our powerpc board,we are not able to ping to the outside world.Below are the details of the powerpc compute node system.. ifconfig fm1-mac1 Link encap:Ethernet HWaddr 00:11:22:33:44:55 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8332 errors:0 dropped:0 overruns:0 frame:0 TX packets:1645 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:531249 (518.7 KiB) TX bytes:179344 (175.1 KiB) Memory:fe4e0000-fe4e0fff lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:1497 errors:0 dropped:0 overruns:0 frame:0 TX packets:1497 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:114980 (112.2 KiB) TX bytes:114980 (112.2 KiB) pkt0 Link encap:Ethernet HWaddr 52:ae:60:c5:53:16 inet6 addr: fe80::50ae:60ff:fec5:5316/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:19 errors:0 dropped:0 overruns:0 frame:0 TX packets:59 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:1957 (1.9 KiB) TX bytes:7492 (7.3 KiB) vhost0 Link encap:Ethernet HWaddr 00:11:22:33:44:55 inet addr:192.168.7.4 Bcast:192.168.7.255 Mask:255.255.252.0 inet6 addr: fe80::211:22ff:fe33:4455/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7229 errors:0 dropped:0 overruns:0 frame:0 TX packets:1204 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:535036 (522.4 KiB) TX bytes:124274 (121.3 KiB) vif --list Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload vif0/0 OS: fm1-mac1 (Speed 100, Duplex 1) Type:Physical HWaddr:00:11:22:33:44:55 IPaddr:0 Vrf:0 Flags:L3L2Vp MTU:1514 Ref:9 RX packets:6370 bytes:466140 errors:0 TX packets:1083 bytes:128992 errors:0 vif0/1 OS: vhost0 Type:Host HWaddr:00:11:22:33:44:55 IPaddr:c0a80704 Vrf:0 Flags:L3L2 MTU:1514 Ref:3 RX packets:876 bytes:89854 errors:0 TX packets:6169 bytes:427245 errors:0 vif0/2 OS: pkt0 Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0 Vrf:65535 Flags:L3 MTU:1514 Ref:2 RX packets:19 bytes:1957 errors:0 TX packets:51 bytes:6844 errors:0 vif0/4350 OS: pkt3 Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0 Vrf:65535 Flags:L3L2 MTU:9136 Ref:1 RX packets:0 bytes:0 errors:0 TX packets:0 bytes:0 errors:0 vif0/4351 OS: pkt1 Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0 Vrf:65535 Flags:L3L2 MTU:9136 Ref:1 RX packets:0 bytes:0 errors:0 TX packets:0 bytes:0 errors:0 ps -ef | grep contrail root 4953 4929 54 14:12 pts/0 00:04:06 contrail-vrouter-agent root 5092 4446 0 14:20 ttyS0 00:00:00 grep contrail However once the contrail-vrouter-agent is stopped,we are able to ping to the outside world with ip address at vhost0 interface. Thanks --Abhishek Jain On Wed, Sep 16, 2015 at 7:37 PM, Prabhjot Singh Sethi <[email protected]<mailto:[email protected]>> wrote: As per logs your node is non-functional because contrail-vrouter-agent is not able to connect to discovery/controller/collector. —8<— snippet — 8<— Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down —8<— snippet — 8<— Please check if discovery server or controller configured in conf file is reachable. — Prabhjot From: Dev <[email protected]<mailto:[email protected]>> on behalf of Vigneshwaran R <[email protected]<mailto:[email protected]>> Date: Wednesday, 16 September 2015 5:36 pm To: Pedro Marques <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi Pedro, I have verified all these stuffs, 1. Physical interface and IP address is mention correctly 2. Ensured that both vif create and ad command uses the same mac address of the physical address 3. When I tcpdump the physical interface, request is coming to the interface from outside, but request / reply is not going out from the interface. When we compare the contrail-vrouter-agent log of x86 compute node and my target board, Below log is present in x86-compute node but not in my target board compute node, ################################################################################### 2015-09-16 Wed 16:11:39:972.687 IST ubuntu1 [Thread 140653374400320, Pid 14304]: SANDESH: Send FAILED: 1442400099972672 TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 192.168.7.241:52894::192.168.6.171:8086(-1) < Active session connection complete controller/src/io/tcp_session.cc 182 ################################################################################### Similarly, below log is present in target-compute node but not in x86 compute node, #################################################################################### 2015-09-16 Wed 10:59:09:498.421 UTC t4240rdb [Thread 4147115104, Pid 23841]: SANDESH: Send FAILED: 807646037 Agent [SYS_INFO]: AgentRouteLog: Path add 192.168.5.254/32<http://192.168.5.254/32> in VRF default-domain:default-project:ip-fabric:__default__ Local controller/src/vnsw/agent/oper/agent_route.cc 411 2015-09-16 Wed 10:59:16:483.615 UTC t4240rdb [Thread 4154557536, Pid 23841]: SANDESH: Send FAILED: 814631121 NodeStatusUVE: data= [ name = t4240rdb process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collec tor name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0<http://0.0.0.0:0>, ] ] status = Down description = Idle : EvStart -> Disconnect ], [ type = Discovery nam e = Collector server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Initializing description = Subscribe ], [ type = Discovery nam e = dns-server server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Down description = Timeout was reached ], [ type = Discovery name = xmpp-server server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Initializing description = Subscribe ], ] ] description = Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down ], ] ] ] 2015-09-16 Wed 10:59:16:483.876 UTC t4240rdb [Thread 4154557536, Pid 23841]: SANDESH: Send FAILED: 814631422 NodeStatusUVE: data= [ name = t42 40rdb process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collec tor name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0<http://0.0.0.0:0>, ] ] status = Down description = Idle : EvStart -> Disconnect ], [ type = Discovery nam e = Collector server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Initializing description = Subscribe ], [ type = Discovery nam e = dns-server server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Down description = Timeout was reached ], [ type = Discovery name = xmpp-server server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Down description = Timeout was reached ], ] ] description = Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down ], ] ] ] 2015-09-16 Wed 10:59:16:484.117 UTC t4240rdb [Thread 4154557536, Pid 23841]: SANDESH: Send FAILED: 814631641 NodeStatusUVE: data= [ name = t42 40rdb process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collec tor name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0<http://0.0.0.0:0>, ] ] status = Down description = Idle : EvStart -> Disconnect ], [ type = Discovery nam e = Collector server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Down description = Timeout was reached ], [ type = Discovery n ame = dns-server server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Down description = Timeout was reached ], [ type = Discover y name = xmpp-server server_addrs= [ [ (*_iter6) = 192.168.6.171:5998<http://192.168.6.171:5998>, ] ] status = Down description = Timeout was reached ], ] ] descriptio n = Collector, Discovery:Collector, Discovery:dns-server, Discovery:xmpp-server connection down ], ] ] ] 2015-09-16 Wed 10:59:21:244.820 UTC t4240rdb [Thread 4149212256, Pid 23841]: SANDESH: Send FAILED: 819392459 Agent [SYS_INFO]: AgentRouteLog: Added route 192.168.5.255/32<http://192.168.5.255/32> in VRF default-domain:default-project:ip-fabric:__default__ Local controller/src/vnsw/agent/oper/agent_route.cc 38 ################################################################################## Is this SANDESH : Send Failed could be any issue ?? or it is related to some other issue ?? On Tue, Sep 15, 2015 at 9:50 PM, Pedro Marques <[email protected]<mailto:[email protected]>> wrote: Vignesh, 1. Ensure that /etc/contrail/contrail-vrouter-agent.conf has the correct values for physical interface and IP address. 2. Ensure that vif —create/—add commands use the mac address of the physical interface. 3. tcpdump the physical interface. On Sep 15, 2015, at 7:07 AM, Vigneshwaran R <[email protected]<mailto:[email protected]>> wrote: Hi All, Now I'm able to run vif command successfully after inserting the kernel module. I just created virtual interface using vif --create and vif --add command. I'm able to see vhost0 interface and I just assigned IP to vhost0 interface Then I ran the contrail-vrouter-agent binary in background and Im not able to ping from both the side. But If I kill the contrail-vrouter-agent binary, I'm able to ping from both the sides. Below is my contrail-vrouter-agent log ############################################################### [Thread 4014994528, Pid 7254]: VnswInterfaceListenerBase Event ADD_ROUTE Interface vhost0 Addr 19 2.168.0.0 prefixlen 24 Gateway 0.0.0.0 Flags 0 Protocol 2 [Thread 4127192160, Pid 7254]: SANDESH: Send FAILED: 387035093 Agent [SYS_INFO]: AgentRouteLog: A dded route 192.168.0.1/32<http://192.168.0.1/32> in VRF default-domain:default-project:ip-fabric:__default__ Local controller/src/vnsw/agent/oper/agent_route.cc 389 ############################################################### vif --list ############################################################## Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload vif0/0 OS: fm1-mac1 (Speed 100, Duplex 1) Type:Physical HWaddr:00:01:02:03:04:05 IPaddr:0 Vrf:0 Flags:L3L2Vp MTU:1514 Ref:3 RX packets:390 bytes:37905 errors:0 TX packets:95 bytes:19258 errors:0 vif0/1 OS: vhost0 Type:Host HWaddr:00:01:02:03:04:05 IPaddr:c0a80069 Vrf:0 Flags:L3L2 MTU:1514 Ref:3 RX packets:96 bytes:19888 errors:0 TX packets:372 bytes:35979 errors:0 vif0/2 OS: pkt0 Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0 Vrf:65535 Flags:L3 MTU:1514 Ref:2 RX packets:1 bytes:70 errors:0 TX packets:18 bytes:1857 errors:0 vif0/4350 OS: pkt3 Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0 Vrf:65535 Flags:L3L2 MTU:9136 Ref:1 RX packets:0 bytes:0 errors:0 TX packets:0 bytes:0 errors:0 vif0/4351 OS: pkt1 Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0 Vrf:65535 Flags:L3L2 MTU:9136 Ref:1 RX packets:0 bytes:0 errors:0 TX packets:0 bytes:0 errors:0 ############################################################### What Could be the reason and solution ?? Kindly help me in this.. Thanks, --Vignesh On Tue, Sep 15, 2015 at 9:46 AM, Vigneshwaran R <[email protected]<mailto:[email protected]>> wrote: Hi Pedro, Rajagopalan and Sanju, Once I inserted the kernel module, I'm able to run vif commands succesfully. Thanks for all your timely help. :) Thanks, --Vignesh On Tue, Sep 15, 2015 at 12:35 AM, Sanju Abraham <[email protected]<mailto:[email protected]>> wrote: Vignesh, Can you please check if you inserted the vrouter kernel module that you have successfully built. Follow the steps below: 1-> copy the build vrouter.ko from ~top-vrouter-build-dir/vrouter/vrouter.ko to /lib/modules/`uname –r`/updates/dkms (create updates/dkms if missing). 2-> depmod from /lib/modules/`uname –r` 3-> modprobe vrouter Verify is vrouter kernel module is loaded. >> lsmod |grep vrouter Use vif once above step is successful Thanks Sanju From: Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>> Date: Monday, September 14, 2015 at 11:31 AM To: Vigneshwaran R <[email protected]<mailto:[email protected]>>, Pedro Marques <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi Vignesh, Did you step through the “vif” code to see what error you are hitting? Raja From: Vigneshwaran R <[email protected]<mailto:[email protected]>> Date: Monday, September 14, 2015 at 7:33 AM To: Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>>, Pedro Marques <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi Rajagopalan/Pedro, We compiled vif binary for powerpc platform and tried to create vhost0 interface using vif --create <i/f> <mac add> command, then when I execute vif --list command it is giving the below error, vif --list Error registering NetLink client: Success (0) How to resolve this error for creating vhost0 interface on my target board which is powerpc. On Sat, Sep 12, 2015 at 11:33 AM, Vigneshwaran R <[email protected]<mailto:[email protected]>> wrote: Okay Raja, Thanks On Fri, Sep 11, 2015 at 11:07 PM, Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>> wrote: Hi Vignesh, The vhost0 interface is created from startup scripts using the “vif” utility. Please take a look at the "vif —create” and "vif —add” options. Raja From: Vigneshwaran R <[email protected]<mailto:[email protected]>> Date: Thursday, September 10, 2015 at 7:49 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [opencontrail-dev] Error in Contrail-vRouter-Agent Binary Hi All, We have ported contrail-vrouter-agent binary to powerpc. I'm trying to run contrail-vrouter-binary in powerpc board to make board act as compute node for launching VM. I inserted the dataplane component (kernel loadable module), and when running userspace component (contrail-vrouter-binary) it is giving issue.. contrail-vrouter-agent log4cplus:ERROR No appenders could be found for logger (SANDESH). log4cplus:ERROR Please initialize the log4cplus system properly. Error in config file </etc/contrail/contrail-vrouter-agent.conf>. Error parsing vmware_mode from <> Config file </etc/contrail/contrail-vrouter-agent.conf> parsing completed. vim contrail-vrouter-agent.log 2015-09-10 Thu 23:15:47:962.834 UTC t4240rdb [Thread 4154224640, Pid 30515]: Error reading interface <vhost0>. Error number 19 : No such device I Just cross compiled cross compiled both vrouter.ko and contrail-vrouter-agent to powerpc and trying to run.. What should I do to create this vhost0 interface in compute node ?? so that I can make use of this board as an compute node ?? Kindly help me in this ... Thanks, --Vignesh _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
_______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
