Good morning,
The first thing to do is to work out Certificate Issues. Unfortunately,
because CAs are an organizational decision, we do not have documents that cover
these situations yet. However, we will make sure to document these solutions
as we go.
What we need to do is find out about Orange’s certificate strategy (high
level, no details necessary), and how to utilize that within AAF.
Here are some questions:
1. Will Orange utilize an Internal CA strategy, or get all certificates from
an Externally purchased CA (i.e. Digicert/other)?
2. Have you obtained Certificates from Orange’s CA policy already?
* Did you cover all possible internal and external FQDNs vis SANs?
(meaning, did you cover elements like “aaf_service”, which K8 Miscroservices
need versus external machine access, which anything outside of K8 needs?
* If you are unsure, and you are comfortable with it, perhaps you can
send the certificate, or a printout of its contents.
--
Jonathan Gathman
Principled-System Architect
ATO Tech Dev/SEAT/Platform Architecture and Technology Management
AT&T Services, Inc.
2349 Oaker, Arnold, MO 63010
m 314-550-3312 |
[email protected]<mailto:[email protected]>
From: "[email protected]" <[email protected]>
Date: Monday, June 18, 2018 at 7:59 AM
To: "[email protected]" <[email protected]>,
"GATHMAN, JONATHAN C" <[email protected]>, "GANDHAM, SAI" <[email protected]>,
"[email protected]" <[email protected]>
Cc: "[email protected]" <[email protected]>,
"[email protected]" <[email protected]>
Subject: [ONAP] [Beijing] [AAF] troubleshooting
Hi ONAP/AAF commiters
I have a question regarding AAF troubleshooting.
We deployed ONAP Beijing on 2 infrastructures.
On the first one, everything is fine, it works as expected.
On the second one, all the healthcheck tests are OK except one: Basic AAF
Health Check
the only major difference I see between the 2 infrastructures is the Ceph
cluster used for storage.
I had a look at the VM on the infrastructure where there is a problem
The dockers re all up& running
I checked the version of the images, everything seems OK
ubuntu@onap-aaf<mailto:ubuntu@onap-aaf>:~$ sudo docker ps
CONTAINER ID IMAGE
COMMAND CREATED STATUS PORTS
NAMES
7b63f1ffafd2 nexus3.onap.org:10001/onap/aaf/aaf_cm:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:8150->8150/tcp aaf_cm
35a45fb03da0 nexus3.onap.org:10001/onap/aaf/aaf_fs:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:80->8096/tcp aaf_fs
0bfb0ea77ff7 nexus3.onap.org:10001/onap/aaf/aaf_gui:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:8200->8200/tcp aaf_gui
cb21ae125285 nexus3.onap.org:10001/onap/aaf/aaf_hello:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:8130->8130/tcp aaf_hello
6ce585d1d9c1 nexus3.onap.org:10001/onap/aaf/aaf_locate:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:8095->8095/tcp aaf_locate
5e44bb32b6d7 nexus3.onap.org:10001/onap/aaf/aaf_oauth:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:8140->8140/tcp aaf_oauth
9221206740f7 nexus3.onap.org:10001/onap/aaf/aaf_service:2.1.1-SNAPSHOT
"/bin/bash -c /opt..." 2 days ago Up 2 days
0.0.0.0:8100->8100/tcp aaf_service
d35d94abc002 cassandra:3.11
"docker-entrypoint..." 3 days ago Up 3 days 7000-7001/tcp,
7199/tcp, 9042/tcp, 9160/tcp aaf_cass
Cassandra docker seems OK
No Java exception
only some Warning
WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-06-18 07:15:01,066 NoSpamLogger.java:94
- Out of 14 commit log syncs over the past 142.52s with average duration of
313.09ms, 4 have exceeded the configured commit interval by an average of
978.27ms
but there are Java exception in the aff dockers
aaf_service/ aaf_oauth
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot
execute this query as it might involve data filtering and thus may have
unpredictable performance. If you want to execute this query despite the
performance unpredictability, use ALLOW FILTERING
at com.datastax.driver.core.Responses$Error.asException(Responses.java:147)
at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:220)
at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:196)
at
com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1442)
at
com.google.common.util.concurrent.Futures$AsyncChainingFuture.doTransform(Futures.java:1433)
at
com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1408)
at com.google.common.util.concurrent.Futures$2$1.run(Futures.java:1177)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
aaf_hello/aaf_gui/aaf_fs
org.onap.aaf.misc.env.APIException: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: No trusted certificate found
at org.onap.aaf.cadi.http.HClient.send(HClient.java:148)
at org.onap.aaf.cadi.client.Rcli.update(Rcli.java:498)
at org.onap.aaf.cadi.register.RemoteRegistrant.update(RemoteRegistrant.java:133)
at
org.onap.aaf.cadi.register.Registrar$RegistrationTimerTask.run(Registrar.java:66)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: No trusted certificate found
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.protocol.https.HttpsClient.afterConnect-28HttpsClient.java-3A559&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jxRielOoGyNWi5tOCRKusgUTQisLiqE_bMTrKwvJIvI&m=UK3Gxc8XCME9b3geqhw4d87Nv5KiVY9jj-gN2tlVCDI&s=XkrmhF2eYGsxU-IEqW0ppwQSZhZXZWuOz1H8omsPHKY&e=>)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.protocol.https.AbstractDelegateHttpsURLConnection.connect-28AbstractDelegateHttpsURLConnection.java-3A185&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jxRielOoGyNWi5tOCRKusgUTQisLiqE_bMTrKwvJIvI&m=UK3Gxc8XCME9b3geqhw4d87Nv5KiVY9jj-gN2tlVCDI&s=J1iXEMX60VipaFODQ2uLBdKQXZ-uY9U-KluZNL9Ve8A&e=>)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.protocol.https.HttpsURLConnectionImpl.connect-28HttpsURLConnectionImpl.java-3A162&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=jxRielOoGyNWi5tOCRKusgUTQisLiqE_bMTrKwvJIvI&m=UK3Gxc8XCME9b3geqhw4d87Nv5KiVY9jj-gN2tlVCDI&s=R6x0cHeppGu9yWFfSbUiEQzFeB_qRc2GNZXjfpWVXEI&e=>)
at org.onap.aaf.cadi.http.HClient.send(HClient.java:142)
... 5 more
Caused by: sun.security.validator.ValidatorException: No trusted certificate
found
at
sun.security.validator.SimpleValidator.buildTrustedChain(SimpleValidator.java:397)
at
sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:134)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
... 16 more
I stopped/cleaned/restarted them.
Then I cleaned everything and relaunch the aaf_install script but without
success, I still got the java exceptions.
I tried to investigate on the certificate issues but I am not familiar with aaf
architecture.
Do you have any recommendation for further troubleshooting?
Thanks
/Morgan
errors when running ete.sh health on robot VM (logs attached)
...
Basic A&AI Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF Health Check [ WARN ]
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None))
after connection broken by
'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
0x7f1b4e0c6450>: Failed to establish a new connection: [Errno 111] Connection
refused',)':
/authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]>
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
0x7f1b4e0c6890>: Failed to establish a new connection: [Errno 111] Connection
refused',)':
/authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]>
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
0x7f1b4e0c6c50>: Failed to establish a new connection: [Errno 111] Connection
refused',)':
/authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]>
| FAIL |
ConnectionError: HTTPSConnectionPool(host='aaf.api.simpledemo.onap.org',
port=8100): Max retries exceeded with url:
/authz/perms/user/[email protected]<mailto:/authz/perms/user/[email protected]>
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection
object at 0x7f1b5097db10>: Failed to establish a new connection: [Errno 111]
Connection refused',))
------------------------------------------------------------------------------
Basic AAF SMS Health Check | PASS |
------------------------------------------------------------------------------
...
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
_______________________________________________
onap-discuss mailing list
[email protected]
https://lists.onap.org/mailman/listinfo/onap-discuss
