Hi AAF team,
We are blocked due to AAF-SMS self signed certificate expiry in Dublin &
ElAlto . We did try to regenerate the SSL certificates but the quorum does
not get formed due to the following exceptions.
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate
signed by unknown authority
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate signed
by unknown authority
I guess we are missing some steps. We would appreciate any quick help to
resolve this one to continue with our Dublin & ElAlto deployments.
Here is the step by step workaround that we tried.
Regards
Vivek
1.
Download the following files from AAF-SMS Container from /sms/certs
folder
aaf_root_ca.cer, aaf-sms.pr, aaf-sms.pub
Password : secretmanagementservicesecretpassword
# It seems like the aaf-sms service uses the first self signed certificate
from aaf-sms.pub
# file. This one expired on June/03/2020
# Not sure about the reason why there is certificate in aaf-sms.pub which
will expire on
# Aug 17 18:51:37 2023 GMT. Maybe the developer forgot to overwrite the
previous one.
1.
Created CSR from existing certificate
openssl x509 -x509toreq -in aaf-sms.pub -out aaf-sms.csr -signkey aaf-sms.pr
1.
Created 509v3 extensions file since the above command does not copy the
v3 extensions
cat v3.ext
subjectAltName = DNS:aaf-sms, DNS:aaf-sms-db.onap, DNS:
aaf-sms.api.simpledemo.onap.org, DNS:aaf-sms.onap, DNS:
aaf-sms.simpledemo.onap.org
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
keyUsage = critical,digitalSignature, keyEncipherment,
nonRepudiation
extendedKeyUsage = clientAuth, serverAuth
authorityKeyIdentifier = keyid, issuer
Not sure, is this the right approach?
1.
Generated new self signed certificate with
openssl x509 -signkey aaf-sms.pr -in aaf-sms.csr -req -days 1825 -out
aaf-sms-new.cert -extfile v3.ext
1.
Copied the above file aaf-sms-new.cert to
~/oom/kubernetes/aaf/charts/aaf-sms/resources/config/aaf-sms.pub
1.
Updated the AAF-SMS
~/oom/kubernetes/aaf/charts/aaf-sms/templates/deployment.yaml . Please
refer to the attached git diff file *aaf-sms-deployment.yaml.diff*.
1.
AAF-SMS SSL certificate is updated, please refer the attached openssl
command output log file *aaf-sms-logs-after-ssl-patch.txt*
# It is having the valid dates
Not Before: Jun 9 17:32:50 2020 GMT
Not After : Jun 8 17:32:50 2025 GMT
1.
Redeployed AAF again but still AAF-SMS jobs are failing with some other
SSL exception and the quorum is not forming for SMS vault. Looks like we
are missing some steps.
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl get
pods -n onap | grep aaf-
dev-aaf-aaf-cass-76c8c584f-xp6kr 1/1
Running 0 69s
dev-aaf-aaf-cm-6d7959c65-9rsnp 1/1
Running 0 69s
dev-aaf-aaf-fs-9898d569-kfsp9 1/1
Running 0 69s
dev-aaf-aaf-gui-586484467-kfx84 1/1
Running 0 69s
dev-aaf-aaf-locate-6867c7fb9d-pzq4r 1/1
Running 0 68s
dev-aaf-aaf-oauth-59cb84d56d-2t5z4 1/1
Running 0 68s
dev-aaf-aaf-service-5c598d6fb-4np26 1/1
Running 0 68s
dev-aaf-aaf-sms-6f5f8c57b-cgbvw 1/1
Running 0 68s
dev-aaf-aaf-sms-preload-dxnwv 1/1
Running 0 68s
dev-aaf-aaf-sms-quorumclient-0 1/1
Running 0 68s
dev-aaf-aaf-sms-quorumclient-1 1/1
Running 0 58s
dev-aaf-aaf-sms-quorumclient-2 1/1
Running 0 47s
dev-aaf-aaf-sms-vault-0 2/2
Running 1 68s
dev-aaf-aaf-sshsm-distcenter-xvxkv 0/1
Completed 0 68s
dev-aaf-aaf-sshsm-testca-qkwcp 0/1
Completed 0 68s
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl
logs -n onap dev-aaf-aaf-sms-preload-dxnwv
Processing /preload/config/has.json
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate
signed by unknown authority
Waiting for SMS to accept requests...
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp
10.43.161.118:10443: connect: connection refused
Waiting for SMS to accept requests...
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp
10.43.161.118:10443: connect: connection refused
Waiting for SMS to accept requests...
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate
signed by unknown authority
Waiting for SMS to accept requests...
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl
logs -n onap dev-aaf-aaf-sms-quorumclient-0
INFO: 2020/06/09 17:55:01 quorumclient.go:77: Starting Log for Quorum Client
ERROR: 2020/06/09 17:55:01 auth.go:226: Read from file: open
auth/dev-aaf-aaf-sms-quorumclient-0/id: no such file or directory
ERROR: 2020/06/09 17:55:01 quorumclient.go:86: Read ID: open
auth/dev-aaf-aaf-sms-quorumclient-0/id: no such file or directory
WARNING: 2020/06/09 17:55:01 quorumclient.go:87: Unable to find an ID for
this client. Generating...
ERROR: 2020/06/09 17:55:01 auth.go:226: Read from file: open
auth/dev-aaf-aaf-sms-quorumclient-0/shard: no such file or directory
ERROR: 2020/06/09 17:55:01 quorumclient.go:99: Read Shard: open
auth/dev-aaf-aaf-sms-quorumclient-0/shard: no such file or directory
WARNING: 2020/06/09 17:55:01 quorumclient.go:100: Unable to find a shard
file. Registering with SMS...
ERROR: 2020/06/09 17:55:01 auth.go:226: Read from file: open
auth/dev-aaf-aaf-sms-quorumclient-0/prkey: no such file or directory
ERROR: 2020/06/09 17:55:01 quorumclient.go:40: LoadPGP Private Key: open
auth/dev-aaf-aaf-sms-quorumclient-0/prkey: no such file or directory
INFO: 2020/06/09 17:55:01 quorumclient.go:41: No Private Key found.
Generating...
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp
10.43.161.118:10443: connect: connection timed out
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate signed
by unknown authority
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp
10.43.161.118:10443: connect: connection refused
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21298): https://lists.onap.org/g/onap-discuss/message/21298
Mute This Topic: https://lists.onap.org/mt/74780453/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl get
pods -n onap | grep aaf-
dev-aaf-aaf-cass-76c8c584f-xp6kr 1/1 Running
0 69s
dev-aaf-aaf-cm-6d7959c65-9rsnp 1/1 Running
0 69s
dev-aaf-aaf-fs-9898d569-kfsp9 1/1 Running
0 69s
dev-aaf-aaf-gui-586484467-kfx84 1/1 Running
0 69s
dev-aaf-aaf-locate-6867c7fb9d-pzq4r 1/1 Running
0 68s
dev-aaf-aaf-oauth-59cb84d56d-2t5z4 1/1 Running
0 68s
dev-aaf-aaf-service-5c598d6fb-4np26 1/1 Running
0 68s
dev-aaf-aaf-sms-6f5f8c57b-cgbvw 1/1 Running
0 68s
dev-aaf-aaf-sms-preload-dxnwv 1/1 Running
0 68s
dev-aaf-aaf-sms-quorumclient-0 1/1 Running
0 68s
dev-aaf-aaf-sms-quorumclient-1 1/1 Running
0 58s
dev-aaf-aaf-sms-quorumclient-2 1/1 Running
0 47s
dev-aaf-aaf-sms-vault-0 2/2 Running
1 68s
dev-aaf-aaf-sshsm-distcenter-xvxkv 0/1 Completed
0 68s
dev-aaf-aaf-sshsm-testca-qkwcp 0/1 Completed
0 68s
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl logs -n
onap dev-aaf-aaf-sms-preload-dxnwv
Processing /preload/config/has.json
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate signed
by unknown authority
Waiting for SMS to accept requests...
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp
10.43.161.118:10443: connect: connection refused
Waiting for SMS to accept requests...
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp
10.43.161.118:10443: connect: connection refused
Waiting for SMS to accept requests...
Get https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate signed
by unknown authority
Waiting for SMS to accept requests...
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl logs -n
onap dev-aaf-aaf-sms-quorumclient-0
INFO: 2020/06/09 17:55:01 quorumclient.go:77: Starting Log for Quorum Client
ERROR: 2020/06/09 17:55:01 auth.go:226: Read from file: open
auth/dev-aaf-aaf-sms-quorumclient-0/id: no such file or directory
ERROR: 2020/06/09 17:55:01 quorumclient.go:86: Read ID: open
auth/dev-aaf-aaf-sms-quorumclient-0/id: no such file or directory
WARNING: 2020/06/09 17:55:01 quorumclient.go:87: Unable to find an ID for this
client. Generating...
ERROR: 2020/06/09 17:55:01 auth.go:226: Read from file: open
auth/dev-aaf-aaf-sms-quorumclient-0/shard: no such file or directory
ERROR: 2020/06/09 17:55:01 quorumclient.go:99: Read Shard: open
auth/dev-aaf-aaf-sms-quorumclient-0/shard: no such file or directory
WARNING: 2020/06/09 17:55:01 quorumclient.go:100: Unable to find a shard file.
Registering with SMS...
ERROR: 2020/06/09 17:55:01 auth.go:226: Read from file: open
auth/dev-aaf-aaf-sms-quorumclient-0/prkey: no such file or directory
ERROR: 2020/06/09 17:55:01 quorumclient.go:40: LoadPGP Private Key: open
auth/dev-aaf-aaf-sms-quorumclient-0/prkey: no such file or directory
INFO: 2020/06/09 17:55:01 quorumclient.go:41: No Private Key found.
Generating...
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp 10.43.161.118:10443:
connect: connection timed out
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: x509: certificate signed by
unknown authority
ERROR: 2020/06/09 17:57:21 quorumclient.go:176: Connect to SMS: Get
https://aaf-sms.onap:10443/v1/sms/quorum/status: dial tcp 10.43.161.118:10443:
connect: connection refused
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ kubectl exec -n
onap dev-robot-robot-745bf4cb95-fsqzp -it -- /bin/bash
root@dev-robot-robot-745bf4cb95-fsqzp:/# export SERVER_IP=aaf-sms.onap
root@dev-robot-robot-745bf4cb95-fsqzp:/# export SERVER_PORT=10443
root@dev-robot-robot-745bf4cb95-fsqzp:/# echo | openssl s_client -showcerts
-servername gnupg.org -connect ${SERVER_IP}:${SERVER_PORT} 2>/dev/null |
openssl x509 -inform pem -noout -text
unable to load certificate
140283162730944:error:0909006C:PEM routines:get_name:no start
line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
root@dev-robot-robot-745bf4cb95-fsqzp:/# echo | openssl s_client -showcerts
-servername gnupg.org -connect ${SERVER_IP}:${SERVER_PORT} 2>/dev/null |
openssl x509 -inform pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
cf:1f:0d:fe:db:43:6e:1f
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = aaf-sms, emailAddress = , OU = [email protected],
OU = OSAAF, O = ONAP, C = US
Validity
Not Before: Jun 9 17:32:50 2020 GMT
Not After : Jun 8 17:32:50 2025 GMT
Subject: CN = aaf-sms, emailAddress = , OU = [email protected],
OU = OSAAF, O = ONAP, C = US
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:a7:3e:6b:7d:55:1e:f4:2f:71:31:14:96:88:81:
c4:d5:e4:f4:da:9e:30:48:b7:f6:de:d4:c4:e7:4e:
97:60:26:77:bd:cb:b1:c1:29:68:d4:85:09:d0:c1:
e3:6b:76:f1:0a:b1:46:33:ad:ac:3f:42:2d:24:9f:
20:25:12:4a:0c:bd:6c:da:33:4c:60:e4:0e:76:fe:
78:dc:27:51:e8:12:9b:f0:6d:4a:f8:43:42:2b:13:
23:4c:dc:9f:b6:90:77:5c:ba:cc:a2:6d:fe:4f:b8:
3d:a4:b7:88:fb:62:11:a0:9a:d4:0b:0b:4b:5e:15:
17:c7:53:b6:f6:a0:e6:67:e8:1d:a2:f0:0c:8e:88:
a7:fa:03:0a:a3:80:cd:03:ae:8d:e0:3e:fb:17:be:
55:09:30:d1:2c:12:9d:d4:13:cc:aa:ba:17:19:9e:
9b:cb:fe:f2:78:9c:c7:5e:2c:96:dc:04:80:f5:8e:
da:7a:f9:03:e5:11:58:90:f2:7a:73:6a:bc:5a:06:
60:9b:8c:7d:7a:f1:cb:d1:26:77:0a:46:ab:b2:c3:
ca:ac:e2:8d:6d:c0:8f:12:fc:c7:4c:e5:46:0e:74:
ff:e3:ea:c1:a8:27:af:61:21:22:08:2f:86:14:bc:
fe:9b:84:da:d3:cf:df:39:87:98:9d:94:ba:ad:94:
8c:19
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:aaf-sms, DNS:aaf-sms-db.onap,
DNS:aaf-sms.api.simpledemo.onap.org, DNS:aaf-sms.onap,
DNS:aaf-sms.simpledemo.onap.org
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
E2:94:C6:7F:11:FB:71:E3:FC:C1:E1:0A:94:EC:8F:C9:5B:D1:56:C9
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Authority Key Identifier:
keyid:E2:94:C6:7F:11:FB:71:E3:FC:C1:E1:0A:94:EC:8F:C9:5B:D1:56:C9
Signature Algorithm: sha256WithRSAEncryption
78:eb:92:6c:ad:16:a2:30:97:b0:49:2e:f6:eb:dc:09:50:a6:
92:b0:73:a2:fb:88:e7:7f:fc:11:3a:d1:73:b3:fc:9f:bc:7d:
9c:cf:d2:21:7e:bb:0e:71:a7:5d:12:7e:1a:e7:3d:2c:2a:03:
2c:8d:3d:73:e6:50:9a:83:ed:82:e2:d6:ad:26:36:7e:12:ec:
41:48:9a:e5:51:c9:01:b1:a3:5c:cc:e1:3a:96:58:66:5d:9e:
b6:2a:0a:4a:8d:c1:6e:d4:72:0a:29:4f:7b:81:7b:16:3a:fd:
f6:c6:24:3f:ba:b4:ed:af:6e:73:56:8b:f8:32:07:8b:a3:25:
5f:0c:28:91:70:5c:ff:1e:77:5d:fe:f8:ee:b4:ee:74:73:fe:
e8:b2:4a:f2:23:cc:74:03:f5:0c:c1:f2:1f:f0:60:c1:18:5f:
80:c1:9d:bb:3e:f7:33:c3:46:b0:59:bd:05:01:be:af:22:6f:
87:27:22:1c:ea:63:b6:01:7a:38:6a:82:28:f1:e4:e2:b6:60:
0f:9e:1a:a0:49:e7:7d:19:3f:11:1e:ec:f4:6b:4d:f3:22:a7:
dc:56:6d:33:c0:81:af:fd:cb:5e:57:8d:88:18:ce:4a:5f:b9:
2e:f0:4c:fd:bd:cd:98:21:56:88:93:b4:5c:50:a7:79:d0:f6:
72:b5:22:ed
root@dev-robot-robot-745bf4cb95-fsqzp:/#
onap@workstation:~/oom/kubernetes/aaf/charts/aaf-sms/templates$ git diff deployment.yaml
diff --git a/kubernetes/aaf/charts/aaf-sms/templates/deployment.yaml b/kubernetes/aaf/charts/aaf-sms/templates/deployment.yaml
index ca35fdc5..0a494b93 100644
--- a/kubernetes/aaf/charts/aaf-sms/templates/deployment.yaml
+++ b/kubernetes/aaf/charts/aaf-sms/templates/deployment.yaml
@@ -80,6 +80,9 @@ spec:
subPath: smsconfig.json
- mountPath: /sms/auth
name: {{ include "common.fullname" . }}-auth
+ - mountPath: /sms/certs/aaf-sms.pub
+ name: aaf-sms--ssl-cert
+ subPath: aaf-sms.pub
resources:
{{ include "common.resources" . | indent 10 }}
{{- if .Values.nodeSelector }}
@@ -97,6 +100,9 @@ spec:
- name : {{ include "common.name" . }}
configMap:
name: {{ include "common.fullname" . }}
+ - name : aaf-sms--ssl-cert
+ configMap:
+ name: {{ include "common.fullname" . }}-preload
- name: {{ include "common.fullname" . }}-auth
persistentVolumeClaim:
claimName: {{ include "common.fullname" . }}
