Pam, Thanks for your response. Now the issue is identified. Using the ip address of a container to connect to a service is causing the issues in some cases(I don’t know why and it is not an issue until few days ago). I replaced the ip address with container name and I don’t see any issue with the CSIT verify jobs.
Thanks, Sunil From: DRAGOSH, PAM Sent: Thursday, September 27, 2018 7:45 PM To: [email protected]; [email protected]; UNNAVA, SUNIL <[email protected]> Cc: HERNANDEZ-HERRERO, JORGE <[email protected]>; Gildas Lanilis <[email protected]> Subject: Re: [onap-discuss] CSIT error Folks, The scripts are doing 2 docker-compose up and downs. You cannot guarantee IP addresse’s between those calls. So why is this being done? The first calls seem to be getting IP addresses, saving them to files. Then shut everything down and bring it all up again. Maybe you have reasons. I would re-initialize the IP address variable after the 2nd docker-compose up to ensure you have the right one. Insert this line: DMAAP_MR_IP=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dockercompose_dmaap_1) At line 88. Before you start curl dumping dmaap messages and then call the Robot script to test it. Pam From: <[email protected]<mailto:[email protected]>> on behalf of Jessica Wagantall <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, September 27, 2018 at 5:37 PM To: "UNNAVA, SUNIL" <[email protected]<mailto:[email protected]>> Cc: "HERNANDEZ-HERRERO, JORGE" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Gildas Lanilis <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] CSIT error Thanks for your reply Jorge! What could be the cause of docker-compose starting and stopping? Is this being controlled via script? I wonder if we can make sure docker-compose is running before executing these tasks Thanks! Jess On Thu, Sep 27, 2018 at 9:07 AM UNNAVA, SUNIL <[email protected]<mailto:[email protected]>> wrote: Hi Jorge, Thanks for sharing your thoughts. IP address of the Kafka and Zookeeper address are not changed from the first time( Otherwise MR container will not start), I can’t identify the IP address of the MR based on the log. We don’t see any issue in local and it is working with out any issues earlier. Jessica, Please let me know, when we can connect and debug the issue. Thanks, Sunil From: HERNANDEZ-HERRERO, JORGE Sent: Thursday, September 27, 2018 11:05 AM To: [email protected]<mailto:[email protected]>; UNNAVA, SUNIL <[email protected]<mailto:[email protected]>> Cc: Gildas Lanilis <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: RE: CSIT error Hello Sunil, Giving it a shot .. Errno 101 has to do with network connectivity. By looking at from looking at the logs in https://jenkins.onap.org/view/dmaap/job/dmaap-master-csit-mrpubsub/405/console<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmaster-2Dcsit-2Dmrpubsub_405_console&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jwTiArcEj6aUX0HjV0M3dT12gUtk7rC07xpgpVZkS_4&m=n3Q1EZhMXvsFJaF4JmiXAxA9RCIrBMenqFgoxYeQkzc&s=9u6AAYhDdDKq6c9OcVTcNElgPU4dQ1_lT0luWlQsPSo&e=>, I wonder since the logs show going through a couple docker-compose up and docker-compose down, the DMAAP_MR_IP is staled from the first time. See below .. Just an idea, in case it helps: 19:35:08 DMaaP Service Running 19:35:08 ++ break 19:35:08 +++ docker inspect '--format={{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dockercompose_dmaap_1 19:35:09 ++ DMAAP_MR_IP=172.17.0.4 19:35:09 +++ docker inspect '--format={{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dockercompose_kafka_1 19:35:09 ++ KAFKA_IP=172.17.0.3 19:35:09 +++ docker inspect '--format={{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dockercompose_zookeeper_1 19:35:09 ++ ZOOKEEPER_IP=172.17.0.2 19:35:09 ++ echo DMAAP_MR_IP=172.17.0.4 19:35:09 DMAAP_MR_IP=172.17.0.4 19:35:09 ++ echo KAFKA_IP=172.17.0.3 19:35:09 KAFKA_IP=172.17.0.3 19:35:09 ++ echo ZOOKEEPER_IP=172.17.0.2 19:35:09 ZOOKEEPER_IP=172.17.0.2 19:35:09 ++ docker-compose down 19:35:10 [33mWARNING:[0m Dependency conflict: an older version of the 'docker-py' package may be polluting the namespace. If you're experiencing crashes, run the following command to remedy the issue: 19:35:10 pip uninstall docker-py; pip uninstall docker; pip install docker 19:35:10 Stopping dockercompose_dmaap_1 ... 19:35:10 Stopping dockercompose_kafka_1 ... 19:35:10 Stopping dockercompose_zookeeper_1 ... .. The DMAAP_MR_IP may change .. 20 seconds later, I don’t see the variable updated in the logs, so may be stale, and explain the 101 errno .. 19:35:38 Creating network "dockercompose_default" with the default driver 19:35:38 Creating dockercompose_zookeeper_1 ... 19:35:38 Creating dockercompose_zookeeper_1 19:35:39 [1A[2K Creating dockercompose_zookeeper_1 ... [32mdone[0m [1BCreating dockercompose_kafka_1 ... 19:35:39 Creating dockercompose_kafka_1 19:35:39 [1A[2K Creating dockercompose_kafka_1 ... [32mdone[0m [1BCreating dockercompose_dmaap_1 ... 19:35:39 Creating dockercompose_dmaap_1 19:35:40 [1A[2K Creating dockercompose_dmaap_1 ... [32mdone[0m [1B++ for i in '{1..50}' 19:35:40 +++ docker inspect --format '{{ .State.Running }}' dockercompose_dmaap_1 19:35:40 ++ '[' true ']' 19:35:40 +++ docker inspect --format '{{ .State.Running }}' dockercompose_zookeeper_1 19:35:40 ++ '[' true ']' 19:35:40 +++ docker inspect --format '{{ .State.Running }}' dockercompose_dmaap_1 19:35:41 ++ '[' true ']' 19:35:41 ++ echo 'DMaaP Service Running' 19:35:41 DMaaP Service Running 19:35:41 ++ break 19:35:41 ++ for i in '{1..50}' 19:35:41 ++ curl -sS -m 1 172.17.0.4:3904/events/TestTopic<https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.0.4-3A3904_events_TestTopic&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jwTiArcEj6aUX0HjV0M3dT12gUtk7rC07xpgpVZkS_4&m=n3Q1EZhMXvsFJaF4JmiXAxA9RCIrBMenqFgoxYeQkzc&s=z6TDE0wgIeCxVCsNSyzfQXRZ5uv3PaubNxNc2ovoJhs&e=> 19:35:41 curl: (7) Failed to connect to 172.17.0.4 port 3904: Network is unreachable 19:35:41 ++ echo sleep 1 19:35:41 sleep 1 Jorge From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of UNNAVA, SUNIL Sent: Thursday, September 27, 2018 9:43 AM To: [email protected]<mailto:[email protected]> Cc: Gildas Lanilis <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: [onap-discuss] CSIT error ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi, We are facing some issues while running the CSIT job. CSIT job is failing all of sudden with the following error. Is any one faced this issue before or any thoughts on the error. 19:57:39 Evaluating expression 'requests.post('http://172.17.0.4:3904/events/TestTopic1<https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.0.4-3A3904_events_TestTopic1&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=AOclne09odx6cmeimzFUhQ&m=BGNWLRX6ZHV59v7jG5BAHr714_5_83XTpocoK2LFsdQ&s=AO0xEaH8E9cXF8eTto0_nXe-XzMjHUb4L8UmsOfIAn4&e=>',data='{"topicName": "TestTopic1"}', headers={u'Accept': u'application/json', u'Content-Type': u'application/json'},verify=False)' failed: ConnectionError: HTTPConnectionPool(host='172.17.0.4', port=3904): Max retries exceeded with url: /events/TestTopic1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f30891afd90>: Failed to establish a new connection: [Errno 101] Network is unreachable',)) https://jenkins.onap.org/view/dmaap/job/dmaap-master-csit-mrpubsub/<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmaster-2Dcsit-2Dmrpubsub_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=AOclne09odx6cmeimzFUhQ&m=BGNWLRX6ZHV59v7jG5BAHr714_5_83XTpocoK2LFsdQ&s=ym7uoFV_8HmxBAeUa5HsVJiEcuL8rlsB8w2ljnaEWDs&e=> Thanks, Sunil -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12718): https://lists.onap.org/g/onap-discuss/message/12718 Mute This Topic: https://lists.onap.org/mt/26332356/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
