Itohan, I’m concerned that your AAI is way out of date, since it should map the AAI user to [email protected]<mailto:[email protected]>, whereas this version is trying to map the user to [email protected]<mailto:[email protected]> which is very old behavior in AAI – what version are you running?
Thanks, jimmy From: <[email protected]> on behalf of "Ukponmwan, Itohan" <[email protected]> Reply-To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Date: Tuesday, October 30, 2018 at 1:37 PM To: "[email protected]" <[email protected]>, "Ukponmwan, Itohan" <[email protected]>, "FREEMAN, BRIAN D" <[email protected]>, "[email protected]" <[email protected]>, "GATHMAN, JONATHAN C" <[email protected]>, "FORSYTH, JAMES" <[email protected]>, "Ying, Ruoyu" <[email protected]> Cc: "Ranganathan, Dileep" <[email protected]>, "Huang, Haibin" <[email protected]> Subject: Re: [onap-discuss] [OOM][AAI]Request to AAI got access denied I see the following authentication issues in the logs of the aai-resources pod. 2018-10-30T17:03:43.190+0000 INFO [cadi] Authenticating via redirection: user/pass combo invalid via AAF from 10.42.96.241 for [email protected] from 10.42.96.241:44780 2018-10-30T17:03:43.190+0000 WARN [cadi] Trans: user=n/a[],ip=10.42.96.241,ms=46.157780,validate=46.149456,code=0.000000 2018-10-30T17:03:50.321+0000 INFO [cadi] [email protected] remains in Missed Credential Map (0 invalid tries) for 52 more seconds 2018-10-30T17:04:04.684+0000 INFO [cadi] Authenticating via redirection: user/pass combo invalid via AAF from 10.42.96.241 for [email protected] from 10.42.96.241:45304 2018-10-30T17:04:04.684+0000 WARN [cadi] Trans: user=n/a[],ip=10.42.96.241,ms=6.283113,validate=6.275950,code=0.000000 2018-10-30T17:04:09.743+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:04:50.321+0000 INFO [cadi] [email protected] has been removed from Missed Credential Map (2 invalid tries) 2018-10-30T17:04:50.321+0000 INFO [cadi] AAFLurPerm removed 0 and renewed 0 expired Permissions out of 0 and removed 1 password misses out of 1 2018-10-30T17:06:09.743+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:07:02.003+0000 INFO [cadi] Authenticating via redirection: user/pass combo invalid via AAF from 10.42.96.241 for [email protected] from 10.42.96.241:49544 2018-10-30T17:07:02.003+0000 WARN [cadi] Trans: user=n/a[],ip=10.42.96.241,ms=17.682188,validate=17.670084,code=0.000000 2018-10-30T17:07:50.324+0000 INFO [cadi] [email protected] remains in Missed Credential Map (0 invalid tries) for 11 more seconds 2018-10-30T17:08:09.744+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:08:50.324+0000 INFO [cadi] [email protected] has been removed from Missed Credential Map (0 invalid tries) 2018-10-30T17:08:50.324+0000 INFO [cadi] AAFLurPerm removed 0 and renewed 0 expired Permissions out of 0 and removed 1 password misses out of 1 2018-10-30T17:10:09.745+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:12:09.746+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:14:09.745+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:16:09.745+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk 2018-10-30T17:18:09.745+0000 INFO [cadi] Persist Cache: removed 0 of 0 items from memory and 0 of 0 from disk From: [email protected] [mailto:[email protected]] On Behalf Of Ukponmwan, Itohan Sent: Tuesday, October 30, 2018 10:25 AM To: FREEMAN, BRIAN D <[email protected]>; [email protected]; [email protected]; GATHMAN, JONATHAN C <[email protected]>; FORSYTH, JAMES <[email protected]>; Ying, Ruoyu <[email protected]> Cc: Ranganathan, Dileep <[email protected]>; Huang, Haibin <[email protected]> Subject: Re: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi Brian, I am running it in integration-SB-06. -Itohan From: FREEMAN, BRIAN D [mailto:[email protected]] Sent: Tuesday, October 30, 2018 10:21 AM To: Ukponmwan, Itohan <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; GATHMAN, JONATHAN C <[email protected]<mailto:[email protected]>>; FORSYTH, JAMES <[email protected]<mailto:[email protected]>>; Ying, Ruoyu <[email protected]<mailto:[email protected]>> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [OOM][AAI]Request to AAI got access denied Itohan – which environment so we can track whether its a new issue or just that the fix from yesterday hasnt been propagated. Brian From: Ukponmwan, Itohan <[email protected]<mailto:[email protected]>> Sent: Tuesday, October 30, 2018 1:11 PM To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>>; GATHMAN, JONATHAN C <[email protected]<mailto:[email protected]>>; FORSYTH, JAMES <[email protected]<mailto:[email protected]>>; Ying, Ruoyu <[email protected]<mailto:[email protected]>> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [OOM][AAI]Request to AAI got access denied I am also facing this issue. When I query AAI. I get the following error; {"requestError":{"policyException":{"messageId":"POL3300","text":"Unauthorized (msg=%1) (ec=%2)","variables":["Unauthorized","ERR.5.1.3300"]}}}. Is there a workaround for this? -Itohan From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Krzysztof Kuzmicki Sent: Tuesday, October 30, 2018 8:19 AM To: FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; GATHMAN, JONATHAN C <[email protected]<mailto:[email protected]>>; FORSYTH, JAMES <[email protected]<mailto:[email protected]>>; Ying, Ruoyu <[email protected]<mailto:[email protected]>> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: Odp.: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi Unfortunately 2.1.5 and 2.1.5-SNAPSHOT :(. br, Krzysztof ________________________________ Od: FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>> Wysłane: wtorek, 30 października 2018 15:20:59 Do: [email protected]<mailto:[email protected]>; Kuzmicki, Krzysztof (Nokia - PL/Wroclaw); GATHMAN, JONATHAN C; FORSYTH, JAMES; [email protected]<mailto:[email protected]> DW: Ranganathan, Dileep; Huang, Haibin; GATHMAN, JONATHAN C Temat: RE: [onap-discuss] [OOM][AAI]Request to AAI got access denied Can you check if you are using the 2.1.6-SNAPSHOT image of AAF ? and if so we might need to get Jonathan support to add a credential until a fix is merged. Brian From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Krzysztof Kuzmicki Sent: Tuesday, October 30, 2018 10:15 AM To: GATHMAN, JONATHAN C <[email protected]<mailto:[email protected]>>; FORSYTH, JAMES <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: Odp.: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi We noticed similar issue last night during PnP PNF tests in ONAP installed in ONAP_5G tenant in TLAB. Here is exception: 2018-10-29T15:48:01.786Z|abefeefc-1b83-4614-a829-d98a6c5335be| org.onap.so.client.RestClient - RestClientSSL using default SSL context! 2018-10-29T15:48:01.793Z|abefeefc-1b83-4614-a829-d98a6c5335be| o.o.so.logging.jaxrs.filter.PayloadLoggingFilter - Making GET request to: https://aai.onap:8443/aai/v14/business/customers/customer/%7Bsome%20subscriber%20id%7D<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai.onap-3A8443_aai_v14_business_customers_customer_-257Bsome-2520subscriber-2520id-257D&d=DwQF-A&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=CAXb1pTUl0YtlTW2Baz_x86bQWJsjWpH3uWp_zWQRIA&s=veZQWw_Ja0zN34WLN3aUap1BtcYWHXQgslYnubEcquc&e=> Request Headers: {Authorization=[Basic TVNPOk1TTw==], X-FromAppId=[MSO], X-TransactionId=[], Accept=[application/json], X-ONAP-RequestID=[abefeefc-1b83-4614-a829-d98a6c5335be], X-ONAP-InvocationID=[a4c2c792-0a05-4d3d-be0e-0e161bb30549], X-ONAP-PartnerName=[SO]} 2018-10-29T15:48:01.869Z|abefeefc-1b83-4614-a829-d98a6c5335be| o.o.so.logging.jaxrs.filter.PayloadLoggingFilter - Response from GET: https://aai.onap:8443/aai/v14/business/customers/customer/%7Bsome%20subscriber%20id%7D<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai.onap-3A8443_aai_v14_business_customers_customer_-257Bsome-2520subscriber-2520id-257D&d=DwQF-A&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=CAXb1pTUl0YtlTW2Baz_x86bQWJsjWpH3uWp_zWQRIA&s=veZQWw_Ja0zN34WLN3aUap1BtcYWHXQgslYnubEcquc&e=> Response Headers: {connection=[close], Date=[Mon, 29 Oct 2018 15:48:01 GMT], Strict-Transport-Security=[max-age=16000000; includeSubDomains; preload;], WWW-Authenticate=[Basic realm="people.osaaf.org"]}{"requestError":{"policyException":{"messageId":"POL3300","text":"Unauthorized (msg=%1) (ec=%2)","variables":["Unauthorized","ERR.5.1.3300"]}}} 2018-10-29T15:48:01.876Z|abefeefc-1b83-4614-a829-d98a6c5335be| o.o.s.bpmn.servicedecomposition.tasks.BBInputSetup - Error calling A&AI. Request-Id=abefeefc-1b83-4614-a829-d98a6c5335be {"requestError":{"policyException":{"messageId":"POL3300","text":"Unauthorized (msg=%1) (ec=%2)","variables":["Unauthorized","ERR.5.1.3300"]}}} javax.ws.rs.ForbiddenException: Error calling A&AI. Request-Id=abefeefc-1b83-4614-a829-d98a6c5335be {"requestError":{"policyException":{"messageId":"POL3300","text":"Unauthorized (msg=%1) (ec=%2)","variables":["Unauthorized","ERR.5.1.3300"]}}} (...) at java.lang.Thread.run(Thread.java:748) 2018-10-29T15:48:01.878Z|abefeefc-1b83-4614-a829-d98a6c5335be| org.onap.so.client.exception.ExceptionBuilder - Building a WorkflowException for Subflow 2018-10-29T15:48:01.878Z|abefeefc-1b83-4614-a829-d98a6c5335be| org.onap.so.client.exception.ExceptionBuilder - Outgoing WorkflowException is WorkflowException[processKey=ExecuteBuildingBlock,errorCode=7000,errorMessage=Error calling A&AI. Request-Id=abefeefc-1b83-4614-a829-d98a6c5335be {"requestError":{"policyException":{"messageId":"POL3300","text":"Unauthorized (msg=%1) (ec=%2)","variables":["Unauthorized","ERR.5.1.3300"]}}},workStep=*] 2018-10-29T15:48:01.879Z|abefeefc-1b83-4614-a829-d98a6c5335be| org.onap.so.client.exception.ExceptionBuilder - Throwing MSOWorkflowException 2018-10-29T15:48:01.905Z|abefeefc-1b83-4614-a829-d98a6c5335be| o.onap.so.logging.jaxrs.filter.SpringClientFilter - Could not Target Entity: http://so-catalog-db-adapter.onap:8082/rainy_day_handler_macro/search/findOneByFlowNameAndServiceTypeAndVnfTypeAndErrorCodeAndWorkStep?flowName=UnassignServiceInstanceBB&serviceType=%2A&vnfType=%2A&errorCode=7000&workStep=%2A<https://urldefense.proofpoint.com/v2/url?u=http-3A__so-2Dcatalog-2Ddb-2Dadapter.onap-3A8082_rainy-5Fday-5Fhandler-5Fmacro_search_findOneByFlowNameAndServiceTypeAndVnfTypeAndErrorCodeAndWorkStep-3FflowName-3DUnassignServiceInstanceBB-26serviceType-3D-252A-26vnfType-3D-252A-26errorCode-3D7000-26workStep-3D-252A&d=DwQF-A&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=CAXb1pTUl0YtlTW2Baz_x86bQWJsjWpH3uWp_zWQRIA&s=aOE2n0FYIehKlou3PgvVj4pVz0Vv56ABQWJwLY4c20s&e=> Is that the same issue? br, Krzysztof ________________________________ Od: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> w imieniu użytkownika Ying, Ruoyu <[email protected]<mailto:[email protected]>> Wysłane: wtorek, 30 października 2018 14:55:21 Do: GATHMAN, JONATHAN C; FORSYTH, JAMES; [email protected]<mailto:[email protected]> DW: Ranganathan, Dileep; Huang, Haibin Temat: Re: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi all, I am now able to access AAI after restarting the aaf-locate pod. Thanks for your help. But it still maybe a bug here. Best Regards, Ruoyu From: Ying, Ruoyu Sent: Tuesday, October 30, 2018 9:24 PM To: 'GATHMAN, JONATHAN C' <[email protected]<mailto:[email protected]>>; FORSYTH, JAMES <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi Jonathan, We’re having this issue in both OOF and VFC tenant in the integration lab. And I got this issue after redeploying other components(I think AAI may also got upgraded at that time). And I check the aaf-locate service port, it seems fine here. Thanks. Best Regards, Ruoyu From: GATHMAN, JONATHAN C [mailto:[email protected]] Sent: Tuesday, October 30, 2018 9:18 PM To: FORSYTH, JAMES <[email protected]<mailto:[email protected]>>; Ying, Ruoyu <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [OOM][AAI]Request to AAI got access denied Is this on SB04?, and what time was the connection problem? I have been monitoring SB04 all morning, and it continues to be fine. The “seems to stop sometimes” is because it appears that SB04 is bounced, and there is something different about the Persistence mechanism on SB04, so that it doesn’t come up right all the time. Otherwise, if no-one restarts, it is very stable. I also checked early on “The Dailies” and they are up and functional as well. -- Jonathan Gathman Principled-System Architect ATO Tech Dev/SEAT/Platform Architecture and Technology Management AT&T Services, Inc. 2349 Oaker, Arnold, MO 63010 m 314-550-3312 | [email protected]<mailto:[email protected]> From: "FORSYTH, JAMES" <[email protected]<mailto:[email protected]>> Date: Tuesday, October 30, 2018 at 8:13 AM To: "Ying, Ruoyu" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Ranganathan, Dileep" <[email protected]<mailto:[email protected]>>, "Huang, Haibin" <[email protected]<mailto:[email protected]>>, "GATHMAN, JONATHAN C" <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi, Ruoyu, Yes, we’ve observed that the aaf-locate service stops sometimes and users get access denied in AAI. AAF team, do you more info or a workaround? Thanks, jimmy From: "Ying, Ruoyu" <[email protected]<mailto:[email protected]>> Date: Tuesday, October 30, 2018 at 9:10 AM To: "FORSYTH, JAMES" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Ranganathan, Dileep" <[email protected]<mailto:[email protected]>>, "Huang, Haibin" <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi jimmy, I check the aaf pods, they are running normally. However, I did see an error log inside aai-resources like this: Authenticating Service unavailable from 10.42.192.221:38084 ( No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0<https://urldefense.proofpoint.com/v2/url?u=https-3A__aaf-2Dlocate.onap-3A8095_locate_AAF-5FNS.service-3A2.0&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=ppCUj4qlFS8I48q0jwv4CT0RPvB8S_NXSh9SkEVDwWc&s=lN1MYjQy3wcoiCP2w259iCYN-iTtA0WRKRk26AwYxyQ&e=>' ) Any suggestion here? Thanks. Best Regards, Ruoyu From: FORSYTH, JAMES [mailto:[email protected]] Sent: Tuesday, October 30, 2018 9:02 PM To: [email protected]<mailto:[email protected]>; Ying, Ruoyu <[email protected]<mailto:[email protected]>> Cc: Ranganathan, Dileep <[email protected]<mailto:[email protected]>>; Huang, Haibin <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [OOM][AAI]Request to AAI got access denied First step – you should verify if AAF is running. If you look at the logs for the aai-resources pod it will usually tell you if it can’t get an answer from AAF. Thanks, -jimmy From: <[email protected]<mailto:[email protected]>> on behalf of "Ying, Ruoyu" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, October 30, 2018 at 9:00 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Ranganathan, Dileep" <[email protected]<mailto:[email protected]>>, "Huang, Haibin" <[email protected]<mailto:[email protected]>> Subject: [onap-discuss] [OOM][AAI]Request to AAI got access denied Hi AAI team, Got an issue while curling AAI in OOF tenant in the integration lab. With Basic auth enabled and ssl cert verification off, the response of all the requests from AAI looks like this: { "timestamp": 1540900280234, "status": 403, "error": "Forbidden", "message": "Access Denied", "path": "/aai/v14/cloud-infrastructure/complexes/complex/DLLSTX233" } Any solution or workaround to solve it? Thanks. Best Regards, Ruoyu -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13412): https://lists.onap.org/g/onap-discuss/message/13412 Mute This Topic: https://lists.onap.org/mt/27796588/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
