In addition to what Adil said, did you select template=Y when building the VNF template with CDT? From the logs, I see that APPC is going through the VNFC path. You should select template=N in the CDT GUI for health check operations, as they don’t require a template.
Marco From: "Mir, Adil Kamal" <[email protected]> Date: Tuesday, September 3, 2019 at 11:02 AM To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "PLATANIA, MARCO (MARCO)" <[email protected]>, Ferrero Marco Angelo <[email protected]>, "HERNANDEZ-HERRERO, JORGE" <[email protected]> Cc: D'Alessandro Alessandro Gerardo <[email protected]>, Signorelli Marco <[email protected]> Subject: RE: [onap-discuss] R: VF Module Scale Out Use Case - questions about closed-loop chain. Hi Aniello, did you run the heatbridge? – that’s what puts the OAM IP for the vlb in AAI. You can run it using robot or just manually try putting the OAM IP in your AAI and see if that solves the issue. From: <[email protected]> on behalf of Aniello Paolo Malinconico <[email protected]> Reply-To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Date: Tuesday, September 3, 2019 at 10:05 AM To: "PLATANIA, MARCO (MARCO)" <[email protected]>, Ferrero Marco Angelo <[email protected]>, "[email protected]" <[email protected]>, "HERNANDEZ-HERRERO, JORGE" <[email protected]> Cc: D'Alessandro Alessandro Gerardo <[email protected]>, Signorelli Marco <[email protected]> Subject: [EXT][onap-discuss] R: VF Module Scale Out Use Case - questions about closed-loop chain. Hi all, to unlock the “clamp lock”, you need to delete the entry into infra_active_request table for the requestdb database. Once deleted, it should continue to work. In fact, now the SO starts the bpmn for the scaleout action, but I have still some problems with the APPC. In few works: When the controlloop is activated, it sends a scaleout request to the SO which starts the bpmn "WorkflowBB". This workflow is composed by 5 or 6 Building Blocks (Healthcheck, Create VFmodule, Assign, Activate, Healthcheck). However, the first BB is an healthcheck operation, managed by the APPC controller. So the APPC makes some GET queries against the AAI to retrieve some information, for example the oam vlb ip for the healthcheck. For example (from appc karaf log): 2019-09-03T13:37:54,109 | INFO | appc-dispatcher-3 | AAIService | 460 - org.onap.ccsdk.sli.adaptors.aai-service-provider - 0.4.1 | Request URL : https://aai.onap:8443/aai/v14/network/vnfcs/vnfc/RegionOne_ONAP-NF_20<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai.onap-3A8443_aai_v14_network_vnfcs_vnfc_RegionOne-5FONAP-2DNF-5F20&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=PzeCKdE3RcPDajABmE2xP_kGD_JPr5Km7GgMlX8Ijac&s=yCRRV4ufyAFCOzPfpXwqiA85hcvQuI48T-odk3fOJ9E&e=> 190821T135252709Z_vlb_001 2019-09-03T13:37:54,137 | INFO | appc-dispatcher-3 | AAIService | 460 - org.onap.ccsdk.sli.adaptors.aai-service-provider - 0.4.1 | HttpURLConnection result: 200 : OK 2019-09-03T13:37:54,137 | INFO | appc-dispatcher-3 | metric | 205 - org.onap.ccsdk.sli.core.sli-common - 0.4.1 | 2019-09-03T13:37:54,143 | INFO | appc-dispatcher-3 | AAIService | 460 - org.onap.ccsdk.sli.adaptors.aai-service-provider - 0.4.1 | Response code : 200, OK 2019-09-03T13:37:54,144 | INFO | appc-dispatcher-3 | AAIService | 460 - org.onap.ccsdk.sli.adaptors.aai-service-provider - 0.4.1 | Response data : {"vnfc-name":"RegionOne_ONAP-NF_20190821T135252709Z_vlb_001","nfc-n aming-code":"vLB","nfc-function":"vLB","in-maint":false,"is-closed-loop-disabled":false,"resource-version":"1566468976695","relationship-list":{"relationship":[{"related-to":"vserver","relationship-label":"tosca.relationships.HostedOn"," related-link":"/aai/v14/cloud-infrastructure/cloud-regions/cloud-region/CloudOwner/RegionOne/tenants/tenant/a9d84b4e2c1e47bfaa4d26d2d894c9c2/vservers/vserver/ccadfb9d-0386-43d5-b007-887ec9ce3fcc","relationship-data":[{"relationship-key": "cloud-region.cloud-owner","relationship-value":"CloudOwner"},{"relationship-key":"cloud-region.cloud-region-id","relationship-value":"RegionOne"},{"relationship-key":"tenant.tenant-id","relationship-value":"a9d84b4e2c1e47bfaa4d26d2d894c 9c2"},{"relationship-key":"vserver.vserver-id","relationship-value":"ccadfb9d-0386-43d5-b007-887ec9ce3fcc"}],"related-to-property":[{"property-key":"vserver.vserver-name","property-value":"RegionOne_ONAP-NF_20190821T135252709Z_vlb_001"}] }]}} 2019-09-03T13:37:54,156 | INFO | appc-dispatcher-3 | AaiService | 493 - org.onap.appc.aai.client - 1.5.2 | AAIResponse: SUCCESS 2019-09-03T13:37:54,157 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | SORTED VSERVERS [{vnfc-count=1, vnfc-name=RegionOne_ONAP-NF_20190821T135252709Z_vlb_001, cloud-region-id=Re gionOne, vserver-selflink=http://163.162.95.137:8774/v2.1/a9d84b4e2c1e47bfaa4d26d2d894c9c2/servers/ccadfb9d-0386-43d5-b007-887ec9ce3fcc<https://urldefense.proofpoint.com/v2/url?u=http-3A__163.162.95.137-3A8774_v2.1_a9d84b4e2c1e47bfaa4d26d2d894c9c2_servers_ccadfb9d-2D0386-2D43d5-2Db007-2D887ec9ce3fcc&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=PzeCKdE3RcPDajABmE2xP_kGD_JPr5Km7GgMlX8Ijac&s=2aAsmGaQ807omBDhRQB-FdDu4-86bVzpxnETLSwKLWo&e=>, tenant-id=a9d84b4e2c1e47bfaa4d26d2d894c9c2, group-notation=null, cloud-owner=CloudOwner, vserver-nam e=RegionOne_ONAP-NF_20190821T135252709Z_vlb_001, vnfc-ipaddress-v4-oam-vip=null, vf-module-id=6e3b7f98-f87f-4aa7-b2eb-f1a8b0ed85d9, vnfc-type=vLB, vnfc-function-code=vLB, vserver-id=ccadfb9d-0386-43d5-b007-887ec9ce3fcc}] 2019-09-03T13:37:54,159 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Final Context 2019-09-03T13:37:54,160 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vnfc-count Value = 1 2019-09-03T13:37:54,162 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vnfc-name Value = RegionOne_ONAP-NF_20190821T135252709Z_vlb_001 2019-09-03T13:37:54,167 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].cloud-region-id Value = RegionOne 2019-09-03T13:37:54,168 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vserver-selflink Value = http://163.162.95.137:8774/v2.1/a9d84b4<https://urldefense.proofpoint.com/v2/url?u=http-3A__163.162.95.137-3A8774_v2.1_a9d84b4&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=PzeCKdE3RcPDajABmE2xP_kGD_JPr5Km7GgMlX8Ijac&s=PeSdyrkPUIlWNXRgKlEntnJcqw4ZYR3v74BVWpa81Fc&e=> e2c1e47bfaa4d26d2d894c9c2/servers/ccadfb9d-0386-43d5-b007-887ec9ce3fcc 2019-09-03T13:37:54,170 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].tenant-id Value = a9d84b4e2c1e47bfaa4d26d2d894c9c2 2019-09-03T13:37:54,172 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].group-notation Value = null 2019-09-03T13:37:54,174 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].cloud-owner Value = CloudOwner 2019-09-03T13:37:54,175 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vserver-name Value = RegionOne_ONAP-NF_20190821T135252709Z_vlb_0 01 2019-09-03T13:37:54,175 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vnfc-ipaddress-v4-oam-vip Value = null 2019-09-03T13:37:54,176 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vf-module-id Value = 6e3b7f98-f87f-4aa7-b2eb-f1a8b0ed85d9 2019-09-03T13:37:54,176 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vnfc-type Value = vLB 2019-09-03T13:37:54,176 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vnfc-function-code Value = vLB 2019-09-03T13:37:54,176 | INFO | appc-dispatcher-3 | AAIResourceNode | 493 - org.onap.appc.aai.client - 1.5.2 | Populating Context Key = tmp.vnfInfo.vm[0].vserver-id Value = ccadfb9d-0386-43d5-b007-887ec9ce3fcc As you can see, some information are missing from the AAI response (i.e. oam-vlb ip is missed). Once retrieve the oam vlb ip, the APPC tries to make an healthcheck operation versus the vLB VM by making: http://null:8183/restconf/operational/health-vnf-onap-plugin:health-vnf-onap-plugin-state/health-check<https://urldefense.proofpoint.com/v2/url?u=http-3A__null-3A8183_restconf_operational_health-2Dvnf-2Donap-2Dplugin-3Ahealth-2Dvnf-2Donap-2Dplugin-2Dstate_health-2Dcheck&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=PzeCKdE3RcPDajABmE2xP_kGD_JPr5Km7GgMlX8Ijac&s=JNjTPY2qnu_aNXmjsKzy5dvQvGfRwoe1LNHRNo2A9sA&e=> Of course, this this request fails because the ip is missing. And so the APPC fails returning to the SO bpmn the following error message: 2019-09-03T13:37:54,471 | ERROR | appc-dispatcher-3 | RestServiceNode | 486 - org.onap.appc.flow.controller - 1.5.2 | Error Message : Error While Sending Rest Request java.lang.Exception: Error While Sending Rest Request at org.onap.appc.flow.controller.executorImpl.RestExecutor.execute(RestExecutor.java:81) ~[?:?] at org.onap.appc.flow.controller.node.RestServiceNode.sendRequest(RestServiceNode.java:86) ~[?:?] Can anyone help me? Any workaround to solve this problem? Thanks, Aniello Paolo Malinconico Aniello Paolo Malinconico Net Reply Via Cardinal Massaia, 83 10147 - Torino - ITALY phone: +39 011 29100 [email protected]<mailto:[email protected]> www.reply.it<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.reply.it&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=PzeCKdE3RcPDajABmE2xP_kGD_JPr5Km7GgMlX8Ijac&s=-QBWtCnV2m739joG2Mhr1lBm0jdPVsz95XaBzJ-k76Y&e=> [Net Reply] Da: PLATANIA, MARCO (MARCO) <[email protected]> Inviato: lunedì 2 settembre 2019 18:07 A: Ferrero Marco Angelo <[email protected]>; [email protected]; HERNANDEZ-HERRERO, JORGE <[email protected]> Cc: Malinconico Aniello Paolo <[email protected]>; D'Alessandro Alessandro Gerardo <[email protected]>; Signorelli Marco <[email protected]> Oggetto: Re: VF Module Scale Out Use Case - questions about closed-loop chain. Policy doesn’t execute closed loop for each received event. For a given event, Policy runs one closed loop every X minutes, this is why the VNF is locked. At the expiration of the timeout (lock), the closed loop is executed again. I can’t remember where locks are kept, Jorge can help with that. Guard policies are pushed by the user from CLAMP typically (unless you call Policy API directly), so they do what the user tells them to do. Note however that for scale out, Policy doesn’t call APPC, it calls SO (which later on calls APPC for VNF health check and reconfiguration). I’m not sure whether your configuration reflects this. You shouldn’t expect Policy to publish in APPC-LCM-READ. Marco From: Ferrero Marco Angelo <[email protected]<mailto:[email protected]>> Date: Monday, September 2, 2019 at 11:13 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "PLATANIA, MARCO (MARCO)" <[email protected]<mailto:[email protected]>>, "PLATANIA, MARCO (MARCO)" <[email protected]<mailto:[email protected]>>, "HERNANDEZ-HERRERO, JORGE" <[email protected]<mailto:[email protected]>> Cc: "Malinconico Aniello Paolo <[email protected]<mailto:[email protected]>> ([email protected]<mailto:[email protected]>)" <[email protected]<mailto:[email protected]>>, D'Alessandro Alessandro Gerardo <[email protected]<mailto:[email protected]>>, Signorelli Marco <[email protected]<mailto:[email protected]>> Subject: VF Module Scale Out Use Case - questions about closed-loop chain. Hi everybody, We are testing the VF Module Scale Out use case. Our vLB correctly sends data to the VES Collector and it correctly publishes data on DMaaP topic unauthenticated.VES_MEASUREMENT_OUTPUT. TCA correctly read data from this topic and, as the threshold is violated, correctly publishes data on DMAAP topic unauthenticated.DCAE_CL_OUTPUT Policy Drools read data from this topic and applies frequency guard but it does not propagate the event even if it seems send evets to DMaap topic POLICY-CL-MGT Who consumes messages from this topic (POLICY-CL-MGT) and who writes to topic APPC-LCM-READ in order to propagate the commands to the APPC ? Accessing to network.log file of policy drools we have found the following messages : [2019-08-31T00:37:54.089+00:00|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.4.2:usecases][OUT| { "AAI": { "vserver.prov-status": "ACTIVE", "vserver.resource-version": "1566468980498", "vserver.is-closed-loop-disabled": "false", "vserver.vserver-name2": "RegionOne_ONAP-NF_20190821T135252709Z_vlb_001", "vserver.vserver-id": "ccadfb9d-0386-43d5-b007-887ec9ce3fcc", "vserver.vserver-selflink": "http://163.162.95.137:8774/v2.1/a9d84b4e2c1e47bfaa4d26d2d894c9c2/servers/ccadfb9d-0386-43d5-b007-887e<https://urldefense.proofpoint.com/v2/url?u=http-3A__163.162.95.137-3A8774_v2.1_a9d84b4e2c1e47bfaa4d26d2d894c9c2_servers_ccadfb9d-2D0386-2D43d5-2Db007-2D887e&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=9F0hi2lvf516P4W9z1pFsP2SQyQELSaXXduf8XCZUR0&s=a66daKjKvJGjrCVBjQAagtnJGP6GkDySc0AOFhJkPRU&e=> "vserver.in-maint": "false", "vserver.vserver-name": "RegionOne_ONAP-NF_20190821T135252709Z_vlb_001" }, "closedLoopAlarmStart": 1567211854626801, "closedLoopControlName": "LOOP_vLoadBalancer_20190821_v1_0_vLoadBalancer_201908210_k8s-tca-clamp-policy", "version": "1.0.2", "requestId": "2d530ae5-cdc5-41a3-83fe-0166431d26e7", "closedLoopEventClient": "DCAE_INSTANCE_ID.dcae-tca", "targetType": "VM", "target": "vserver.vserver-name", "from": "policy:usecases", "policyScope": "onap.policies.controlloop.Operational:1.0.0", "policyName": "OPERATIONAL_vLoadBalancer_20190821_v1_0_vLoadBalancer_201908210_k8s-tca-clamp-policy.EVENT.MANAGER", "policyVersion": "1.0.0", "notification": "REJECTED", "message": "The target RegionOne_ONAP-NF_20190821T135252709Z_vlb_001 is already locked", "notificationTime": "2019-08-31 00:37:54.089000+00:00", "history": [] } [2019-08-31T00:38:08.124+00:00|qtp1534754611-39]10.42.10.195 - - [31/Aug/2019:00:38:08 +0000] "GET /healthcheck HTTP/1.1" 200 102 root@cp-1:~/oom/kubernetes# 00|qtp1534754611-35]10.42.10.195 - - [31/Aug/2019:00:38:23 +0000] "GET /healthcheck HTTP/1.1" 200 102 It seems that we vserver entry in the AAI (vserver relative to the vLB) is locked ? Why it is locked ? Who locks it ? Where are kept the locks ? I have also accessed the policy db root@dev-policy-policydb-868884d57f-g4h9g:/# mysql -uroot -psecret Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 776820 Server version: 10.2.14-MariaDB-10.2.14+maria~jessie mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +---------------------+ | Database | +---------------------+ | information_schema | | log | | migration | | mysql | | onap_sdk | | operationshistory | | operationshistory10 | | performance_schema | | policyadmin | | pooling | | support | +---------------------+ 11 rows in set (0.02 sec) And I have checked the operationhistory schema : I made the resume query MariaDB [operationshistory]> select count(*),message from operationshistory group by message; +----------+---------------------------+ | count(*) | message | +----------+---------------------------+ | 1474 | 999 Failed | | 11411 | Operation denied by Guard | +----------+---------------------------+ 2 rows in set (0.03 sec) And I always got ‘999 Failed’ every time the guard let it me execute the policy. Who send this code ? Where I can get the log of the policy executor that generate this message ? Thank you in advance for your precious help, Marco Ferrero ------------------------------------------------------------------ TIM Marco Angelo Ferrero Mobile & Fixed Innovation IP, Transport & Core Innovation TIM S.p.A. Via G. Reiss Romoli, 274 – 10148 TORINO +39 011 2286739 +39 331 6001354 [https://img.tim.it:443/sdr/banner-mail-dipe.jpg]<https://urldefense.proofpoint.com/v2/url?u=https-3A__on.tim.it_banner-2Dmail-2Ddip&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=9F0hi2lvf516P4W9z1pFsP2SQyQELSaXXduf8XCZUR0&s=4U5QxZWZQCB1NrAZBIB5Jfnzl7eOlV1Oc2DkNq1QtUs&e=> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. Rispetta l'ambiente. Non stampare questa mail se non è necessario. ________________________________ External Email: Please use caution when opening links and attachments / Courriel externe: Soyez prudent avec les liens et documents joints -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18801): https://lists.onap.org/g/onap-discuss/message/18801 Mute This Topic: https://lists.onap.org/mt/33126910/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
