Hi Brian, Thanks for the reply.
When we try to execute "vcpe.py loop" command from K8S nodes (i.e. elaltornd-k8s-04 & elaltornd-k8s-01 where policy-Pap & Policy-api are running respectively) we are getting below error. Please let us know anything we are missing ? root@elaltornd-nfs:~/integration/test/vcpe# kubectl get pods -n onap -o wide| egrep "policy-api|policy-pap" rnd-policy-pap-595954cc4f-zf8wl 2/2 Running 0 7d16h 10.42.7.23 elaltornd-k8s-04 <none> <none> rnd-policy-policy-api-55c66b46b5-8sqwj 1/1 Running 0 7d16h 10.42.10.25 elaltornd-k8s-01 <none> <none> rnd-policy-policy-pap-dfdc5789b-lp4cn 1/1 Running 0 7d16h 10.42.7.22 elaltornd-k8s-04 <none> <none> root@elaltornd-k8s-04:~/integration/test/vcpe# python vcpe.py loop ---------------------------------------------------------------------------------------------------- vcpe.py: Brief info about this program vcpe.py init: Add customer service data to SDNC and SO DBs. vcpe.py infra: Deploy infrastructure, including DHCP, AAA, DNS, Web Server, vBNG, vGMUX, vBRG. vcpe.py brg: Deploy brg only (for testing after infra succeeds). vcpe.py customer: Deploy customer service, including vGW and VxLANs vcpe.py loop: Test closed loop control ---------------------------------------------------------------------------------------------------- 2020-03-12 13:25:17,334 INFO vcpecommon.__init__(): Initializing configuration 2020-03-12 13:25:17,335 DEBUG vcpecommon.get_vm_ip(): nova --os-auth-url http://192.168.9.75:5000 --os-project-domain-name default --os-password root123 --os-region-name RegionOne --os-username admin --os-user-domain-id default --os-tenant-id 70968f3b213c4c8d81d92c06a9dddf7c --os-project-domain-id default list ERROR (ConnectFailure): Unable to establish connection to http://controller:8774/v2.1: HTTPConnectionPool(host='controller', port=8774): Max retries exceeded with url: /v2.1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe4932b5dd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)) 2020-03-12 13:25:18,201 ERROR vcpecommon.get_vm_ip(): Cannot find all desired IP addresses for ['sdc', 'so', 'sdnc', 'robot', 'aai-inst1', 'dcae-bootstrap', 'brg', 'mux']. 2020-03-12 13:25:18,203 ERROR vcpecommon.get_vm_ip(): { "aai-inst1": "20.0.0.11", "dcae-bootstrap": "20.0.0.11", "robot": "20.0.0.11", "sdc": "20.0.0.11", "sdnc": "20.0.0.11", "so": "20.0.0.11" } 2020-03-12 13:25:18,203 ERROR vcpecommon.get_vm_ip(): Temporarily continue.. remember to check back vcpecommon.py line: 396 2020-03-12 13:25:18,203 INFO __name__.closed_loop(): Setting up closed loop policy Traceback (most recent call last): File "vcpe.py", line 249, in <module> closed_loop(22) File "vcpe.py", line 164, in closed_loop vcpecommon.set_closed_loop_policy(policy_template_file) File "/root/integration/test/vcpe/vcpecommon.py", line 345, in set_closed_loop_policy p_api_cluster_ip = self.get_k8s_service_cluster_ip(self.policy_api_service_name) File "/root/integration/test/vcpe/vcpecommon.py", line 557, in get_k8s_service_cluster_ip config.load_kube_config() File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 645, in load_kube_config persist_config=persist_config) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 613, in _get_kube_config_loader_for_yaml_file **kwargs) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 153, in __init__ self.set_active_context(active_context) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 173, in set_active_context context_name = self._config['current-context'] File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 495, in __getitem__ v = self.safe_get(key) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 491, in safe_get key in self.value): TypeError: argument of type 'NoneType' is not iterable root@elaltornd-k8s-01:~/integration/test/vcpe# python vcpe.py loop ---------------------------------------------------------------------------------------------------- vcpe.py: Brief info about this program vcpe.py init: Add customer service data to SDNC and SO DBs. vcpe.py infra: Deploy infrastructure, including DHCP, AAA, DNS, Web Server, vBNG, vGMUX, vBRG. vcpe.py brg: Deploy brg only (for testing after infra succeeds). vcpe.py customer: Deploy customer service, including vGW and VxLANs vcpe.py loop: Test closed loop control ---------------------------------------------------------------------------------------------------- 2020-03-12 13:43:33,604 INFO vcpecommon.__init__(): Initializing configuration 2020-03-12 13:43:33,604 DEBUG vcpecommon.get_vm_ip(): nova --os-auth-url http://192.168.9.75:5000 --os-project-domain-name default --os-password root123 --os-region-name RegionOne --os-username admin --os-user-domain-id default --os-tenant-id 70968f3b213c4c8d81d92c06a9dddf7c --os-project-domain-id default list ERROR (ConnectFailure): Unable to establish connection to http://controller:8774/v2.1: HTTPConnectionPool(host='controller', port=8774): Max retries exceeded with url: /v2.1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f23491acdd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)) 2020-03-12 13:43:34,471 ERROR vcpecommon.get_vm_ip(): Cannot find all desired IP addresses for ['sdc', 'so', 'sdnc', 'robot', 'aai-inst1', 'dcae-bootstrap', 'brg', 'mux']. 2020-03-12 13:43:34,472 ERROR vcpecommon.get_vm_ip(): { "aai-inst1": "20.0.0.11", "dcae-bootstrap": "20.0.0.11", "robot": "20.0.0.11", "sdc": "20.0.0.11", "sdnc": "20.0.0.11", "so": "20.0.0.11" } 2020-03-12 13:43:34,472 ERROR vcpecommon.get_vm_ip(): Temporarily continue.. remember to check back vcpecommon.py line: 396 2020-03-12 13:43:34,472 INFO __name__.closed_loop(): Setting up closed loop policy Traceback (most recent call last): File "vcpe.py", line 249, in <module> closed_loop(22) File "vcpe.py", line 164, in closed_loop vcpecommon.set_closed_loop_policy(policy_template_file) File "/root/integration/test/vcpe/vcpecommon.py", line 345, in set_closed_loop_policy p_api_cluster_ip = self.get_k8s_service_cluster_ip(self.policy_api_service_name) File "/root/integration/test/vcpe/vcpecommon.py", line 557, in get_k8s_service_cluster_ip config.load_kube_config() File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 645, in load_kube_config persist_config=persist_config) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 613, in _get_kube_config_loader_for_yaml_file **kwargs) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 153, in __init__ self.set_active_context(active_context) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 173, in set_active_context context_name = self._config['current-context'] File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 495, in __getitem__ v = self.safe_get(key) File "/usr/local/lib/python2.7/dist-packages/kubernetes/config/kube_config.py", line 491, in safe_get key in self.value): TypeError: argument of type 'NoneType' is not iterable Regards, Manjunath. ________________________________ From: FREEMAN, BRIAN D <[email protected]> Sent: Thursday, March 12, 2020 6:26 PM To: Devangam Manjunatha <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Cc: JOMY JOSE <[email protected]>; Velugubantla Praveen <[email protected]>; Naveen S. Sankad <[email protected]>; Deepak K <[email protected]> Subject: RE: [onap-discuss] [El-Alto][VCPE]- Error while running "vcpe.py loop" as part of "closed loop testing" You understand the problem. Running from the nfs server requires the appropriate nodeport and k8 node ip’s to be used. If you use vcpe.py loop per the current design then you need to be inside a k8 node so that you can use the k8 dns / reachability. Brian From: Devangam Manjunatha <[email protected]> Sent: Wednesday, March 11, 2020 8:24 PM To: FREEMAN, BRIAN D <[email protected]>; [email protected] Cc: JOMY JOSE <[email protected]>; Velugubantla Praveen <[email protected]>; Naveen S. Sankad <[email protected]>; Deepak K <[email protected]> Subject: Re: [onap-discuss] [El-Alto][VCPE]- Error while running "vcpe.py loop" as part of "closed loop testing" Dear Team, Please let us know how to below issue. Regards, Manjunath. ________________________________ From: Devangam Manjunatha <[email protected]<mailto:[email protected]>> Sent: Wednesday, 11 March 2020, 21:17 To: FREEMAN, BRIAN D Cc: JOMY JOSE; Velugubantla Praveen; Naveen S. Sankad; Deepak K Subject: [onap-discuss] [El-Alto][VCPE]- Error while running "vcpe.py loop" as part of "closed loop testing" Hi Brian, On ONAP El-Alto, when we try to execute "vcpe.py loop" command as part of "Closed Loop Test". We are getting below error. root@elaltornd-nfs:~/integration/test/vcpe# python vcpe.py loop ---------------------------------------------------------------------------------------------------- vcpe.py: Brief info about this program vcpe.py init: Add customer service data to SDNC and SO DBs. vcpe.py infra: Deploy infrastructure, including DHCP, AAA, DNS, Web Server, vBNG, vGMUX, vBRG. vcpe.py brg: Deploy brg only (for testing after infra succeeds). vcpe.py customer: Deploy customer service, including vGW and VxLANs vcpe.py loop: Test closed loop control ---------------------------------------------------------------------------------------------------- 2020-03-11 13:03:11,379 INFO vcpecommon.__init__(): Initializing configuration 2020-03-11 13:03:11,379 DEBUG vcpecommon.get_vm_ip(): nova --os-auth-url http://192.168.9.75:5000<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__192.168.9.75-3A5000%26d%3DDwQFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3DFj56rGtGKRzMHaPWBqn-XYVxnQfCJBOaqzxOr-E6yEU%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145993960876&sdata=8Ax9K7xRfVZMJfba5Ac74Bb%2Fr9xcmfhe6FoV3YbKcR0%3D&reserved=0> --os-project-domain-name default --os-password root123 --os-region-name RegionOne --os-username admin --os-user-domain-id default --os-tenant-id 70968f3b213c4c8d81d92c06a9dddf7c --os-project-domain-id default list 2020-03-11 13:03:13,269 INFO __name__.closed_loop(): Setting up closed loop policy ############################### // This print statement we added in vcpecommon.py file 10.43.21.204 // This print statement we added in vcpecommon.py file 10.43.157.233 // This print statement we added in vcpecommon.py file {u'content': u'controlLoop%3A%0D%0A++version%3A+2.0.0%0D%0A++controlLoopName%3A+ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e%0D%0A++trigger_policy%3A+unique-policy-id-1-restart%0D%0A++timeout%3A+3600%0D%0A++abatement%3A+true%0D%0A+%0D%0Apolicies%3A%0D%0A++-+id%3A+unique-policy-id-1-restart%0D%0A++++name%3A+Restart+the+VM%0D%0A++++description%3A%0D%0A++++actor%3A+APPC%0D%0A++++recipe%3A+Restart%0D%0A++++target%3A%0D%0A++++++type%3A+VM%0D%0A++++retry%3A+3%0D%0A++++timeout%3A+1200%0D%0A++++success%3A+final_success%0D%0A++++failure%3A+final_failure%0D%0A++++failure_timeout%3A+final_failure_timeout%0D%0A++++failure_retries%3A+final_failure_retries%0D%0A++++failure_exception%3A+final_failure_exception%0D%0A++++failure_guard%3A+final_failure_guard', u'policy-id': u'operational.vcpe'} // This print statement we added in vcpecommon.py file Traceback (most recent call last): File "vcpe.py", line 249, in <module> closed_loop(22) File "vcpe.py", line 164, in closed_loop vcpecommon.set_closed_loop_policy(policy_template_file) File "/root/integration/test/vcpe/vcpecommon.py", line 361, in set_closed_loop_policy verify=False, headers=self.policy_headers) File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='10.43.157.233', port=6969): Max retries exceeded with url: /policy/pap/v1/pdps (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe1f5c7b190>: Failed to establish a new connection: [Errno 110] Connection timed out',)) root@elaltornd-nfs:~/integration/test/vcpe# >From above error, it is clear that host "10.43.157.233" is not reachable. >"vcpe.py loop" we are executing from nfs vm. We noticed that there is a JIRA(i.e. https://jira.onap.org/browse/INT-1323<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__jira.onap.org_browse_INT-2D1323%26d%3DDwMFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3D2Uywd6GszpYTsRe-Fycl7tH92y1idUIR4GkWRlp32P8%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145993970872&sdata=ShNBaaB8y%2FZFqp77ClHFo4gK4kCu0Tw8zexiaFoNF%2BQ%3D&reserved=0>) for this issue. W.r.t above JIRA, we cross checked El-Alto , Policy-api & Policy-pap cluster details as shown below. root@elaltornd-nfs:~/integration/test/vcpe# helm ls | head -2 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE rnd 1 Tue Mar 3 20:37:49 2020 DEPLOYED onap-5.0.0 El Alto onap root@elaltornd-nfs:~/integration/test/vcpe# root@elaltornd-nfs:~/integration/test/vcpe# root@elaltornd-nfs:~/integration/test/vcpe# kubectl get svc -n onap | egrep "policy-api|policy-pap" policy-api ClusterIP 10.43.21.204 <none> 6969/TCP 7d16h policy-pap ClusterIP 10.43.157.233 <none> 6969/TCP 7d16h root@elaltornd-nfs:~/integration/test/vcpe# Also we checked connectivity as below. root@elaltornd-nfs:~/integration/test/vcpe# curl -kI https://10.43.21.204:6969<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__10.43.21.204-3A6969%26d%3DDwQFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3DaZ59ffMlIZp-37wQSDRfEw3DmvTrnqnprtINX-nlrb4%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145993980870&sdata=MlUea%2BCt63Xi%2F1EKYP3qlVHjUIr0jMKLba7kvgpCElY%3D&reserved=0> curl: (7) Failed to connect to 10.43.21.204 port 6969: Connection timed out root@elaltornd-nfs:~/integration/test/vcpe# After scanning through JIRA, We came to know that, We are executing "vcpe.py loop" in nfs VM, Where as Bartek executing in k8s cluster. At the end of JIRA, I can see below comment from Catherine. Please let what are those code changes. Catherine Lefevre<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__jira.onap.org_secure_ViewProfile.jspa-3Fname-3DKatel34%26d%3DDwMFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3DSctPc9pLoFSxMKaVf5MYJOqNJOvVqeClhbA0MPAiBJU%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145993980870&sdata=12HJi%2BzEpnf9ihY31NcsxaiAz%2BTBFS%2FbuIYKd3k5wuc%3D&reserved=0> added a comment - 21/Jan/20 11:45 AM Brian Freeman<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__jira.onap.org_secure_ViewProfile.jspa-3Fname-3Dbdfreeman1421%26d%3DDwMFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3DVbnrvVHgyduX9trnwMM1R72-qUYXN-YUQ3qJCQAPBTc%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145993990867&sdata=g6ASGXw%2FEchp%2F7O1MMddvQMIgkOOXdqV9TQ%2FpifJXvY%3D&reserved=0> - the code has been merged. Can you please let us know if we can close it? thanks Please let us know how to fix this issue. Regards, Manjunath. L&T Technology Services Ltd www.LTTS.com<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.LTTS.com%26d%3DDwQFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3Dt7wjZyyfD6ZWIavmr8P7H18H6RxA0PNycjvuEOznVkk%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145994000863&sdata=%2FCwZCxmxQ6aIYJ6sYaP5eChxS9pv3LKhnZC20efoBho%3D&reserved=0> L&T Technology Services Limited (LTTS) is committed to safeguard your data privacy. For more information to view our commitment towards data privacy under GDPR, please visit the privacy policy on our website www.Ltts.com<https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.Ltts.com%26d%3DDwQFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DWxkVzOU7-KXW93Yg95R24mwmQ5yMU_57zpyZ6r1IFlo%26s%3DxR0rJDRtqMlJg96fOuBgt-gD59JTJSOXSR0ByTBymQs%26e%3D&data=02%7C01%7CDevangam.Manjunatha%40Ltts.com%7C31f436538a7e4ebe9a3308d7c684cd4f%7C311b33788e8a4b5ea33fe80a3d8ba60a%7C0%7C0%7C637196145994000863&sdata=2EdWovB%2BAmXnygu%2Bx0Z5ZzzzJT2ZVOQalZREtEq6zws%3D&reserved=0>. This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system. L&T Technology Services Ltd www.LTTS.com L&T Technology Services Limited (LTTS) is committed to safeguard your data privacy. For more information to view our commitment towards data privacy under GDPR, please visit the privacy policy on our website www.Ltts.com. This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20264): https://lists.onap.org/g/onap-discuss/message/20264 Mute This Topic: https://lists.onap.org/mt/71892755/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
