Hello Jonathan and thanks for the reply. Both deployment were « tests » deployment and we didn’t put any specific stuff on the certificate side except that both open stacks have valid certificates (but different obviously). I imagine that’s not the problem as AAF is not related to Openstack...
Le 18 juin 2018 à 21:11, GATHMAN, JONATHAN C <[email protected]<mailto:[email protected]>> a écrit : Good morning, The first thing to do is to work out Certificate Issues. Unfortunately, because CAs are an organizational decision, we do not have documents that cover these situations yet. However, we will make sure to document these solutions as we go. What we need to do is find out about Orange’s certificate strategy (high level, no details necessary), and how to utilize that within AAF. Here are some questions: 1. Will Orange utilize an Internal CA strategy, or get all certificates from an Externally purchased CA (i.e. Digicert/other)? 2. Have you obtained Certificates from Orange’s CA policy already? * Did you cover all possible internal and external FQDNs vis SANs? (meaning, did you cover elements like “aaf_service”, which K8 Miscroservices need versus external machine access, which anything outside of K8 needs? * If you are unsure, and you are comfortable with it, perhaps you can send the certificate, or a printout of its contents. -- Jonathan Gathman Principled-System Architect ATO Tech Dev/SEAT/Platform Architecture and Technology Management AT&T Services, Inc. 2349 Oaker, Arnold, MO 63010 m 314-550-3312 | [email protected]<mailto:[email protected]> From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, June 18, 2018 at 7:59 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "GATHMAN, JONATHAN C" <[email protected]<mailto:[email protected]>>, "GANDHAM, SAI" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [ONAP] [Beijing] [AAF] troubleshooting Hi ONAP/AAF commiters I have a question regarding AAF troubleshooting. We deployed ONAP Beijing on 2 infrastructures. On the first one, everything is fine, it works as expected. On the second one, all the healthcheck tests are OK except one: Basic AAF Health Check the only major difference I see between the 2 infrastructures is the Ceph cluster used for storage. I had a look at the VM on the infrastructure where there is a problem The dockers re all up& running I checked the version of the images, everything seems OK ubuntu@onap-aaf<mailto:ubuntu@onap-aaf>:~$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7b63f1ffafd2 nexus3.onap.org:10001/onap/aaf/aaf_cm:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_cm:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:8150->8150/tcp aaf_cm 35a45fb03da0 nexus3.onap.org:10001/onap/aaf/aaf_fs:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_fs:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:80->8096/tcp aaf_fs 0bfb0ea77ff7 nexus3.onap.org:10001/onap/aaf/aaf_gui:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_gui:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:8200->8200/tcp aaf_gui cb21ae125285 nexus3.onap.org:10001/onap/aaf/aaf_hello:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_hello:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:8130->8130/tcp aaf_hello 6ce585d1d9c1 nexus3.onap.org:10001/onap/aaf/aaf_locate:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_locate:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:8095->8095/tcp aaf_locate 5e44bb32b6d7 nexus3.onap.org:10001/onap/aaf/aaf_oauth:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_oauth:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:8140->8140/tcp aaf_oauth 9221206740f7 nexus3.onap.org:10001/onap/aaf/aaf_service:2.1.1-SNAPSHOT<http://nexus3.onap.org:10001/onap/aaf/aaf_service:2.1.1-SNAPSHOT> "/bin/bash -c /opt..." 2 days ago Up 2 days 0.0.0.0:8100->8100/tcp aaf_service d35d94abc002 cassandra:3.11 "docker-entrypoint..." 3 days ago Up 3 days 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp aaf_cass Cassandra docker seems OK No Java exception only some Warning WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-06-18 07:15:01,066 NoSpamLogger.java:94 - Out of 14 commit log syncs over the past 142.52s with average duration of 313.09ms, 4 have exceeded the configured commit interval by an average of 978.27ms but there are Java exception in the aff dockers aaf_service/ aaf_oauth Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING at com.datastax.driver.core.Responses$Error.asException(Responses.java:147) at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:220) at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:196) at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442) at com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433) at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408) at com.google.common.util.concurrent.Futures$2$1.run(Futures.java:1177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) aaf_hello/aaf_gui/aaf_fs org.onap.aaf.misc.env.APIException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found at org.onap.aaf.cadi.http.HClient.send(HClient.java:148) at org.onap.aaf.cadi.client.Rcli.update(Rcli.java:498) at org.onap.aaf.cadi.register.RemoteRegistrant.update(RemoteRegistrant.java:133) at org.onap.aaf.cadi.register.Registrar$RegistrationTimerTask.run(Registrar.java:66) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker.process_record(Handshaker.java:987) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at sun.net<http://sun.net>.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.protocol.https.HttpsClient.afterConnect-28HttpsClient.java-3A559&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jxRielOoGyNWi5tOCRKusgUTQisLiqE_bMTrKwvJIvI&m=UK3Gxc8XCME9b3geqhw4d87Nv5KiVY9jj-gN2tlVCDI&s=XkrmhF2eYGsxU-IEqW0ppwQSZhZXZWuOz1H8omsPHKY&e=>) at sun.net<http://sun.net>.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.protocol.https.AbstractDelegateHttpsURLConnection.connect-28AbstractDelegateHttpsURLConnection.java-3A185&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jxRielOoGyNWi5tOCRKusgUTQisLiqE_bMTrKwvJIvI&m=UK3Gxc8XCME9b3geqhw4d87Nv5KiVY9jj-gN2tlVCDI&s=J1iXEMX60VipaFODQ2uLBdKQXZ-uY9U-KluZNL9Ve8A&e=>) at sun.net<http://sun.net>.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.protocol.https.HttpsURLConnectionImpl.connect-28HttpsURLConnectionImpl.java-3A162&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jxRielOoGyNWi5tOCRKusgUTQisLiqE_bMTrKwvJIvI&m=UK3Gxc8XCME9b3geqhw4d87Nv5KiVY9jj-gN2tlVCDI&s=R6x0cHeppGu9yWFfSbUiEQzFeB_qRc2GNZXjfpWVXEI&e=>) at org.onap.aaf.cadi.http.HClient.send(HClient.java:142) ... 5 more Caused by: sun.security.validator.ValidatorException: No trusted certificate found at sun.security.validator.SimpleValidator.buildTrustedChain(SimpleValidator.java:397) at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:134) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596) ... 16 more I stopped/cleaned/restarted them. Then I cleaned everything and relaunch the aaf_install script but without success, I still got the java exceptions. I tried to investigate on the certificate issues but I am not familiar with aaf architecture. Do you have any recommendation for further troubleshooting? Thanks /Morgan errors when running ete.sh health on robot VM (logs attached) ... Basic A&AI Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1b4e0c6450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]> [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1b4e0c6890>: Failed to establish a new connection: [Errno 111] Connection refused',)': /authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]> [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1b4e0c6c50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]> | FAIL | ConnectionError: HTTPSConnectionPool(host='aaf.api.simpledemo.onap.org<http://aaf.api.simpledemo.onap.org>', port=8100): Max retries exceeded with url: /authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]> (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1b5097db10>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic AAF SMS Health Check | PASS | ------------------------------------------------------------------------------ ... _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ onap-discuss mailing list [email protected]<mailto:[email protected]> https://lists.onap.org/mailman/listinfo/onap-discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
