Hi, I managed to fix the issue. OFP actually worked fine on the initial connect if I manually set the controller on OVS even when OVSDB connection was failing. I then looked at the OFP code and the certificate management is handled differently there via OFP lib methods/classes for handling the SSL Context, while OVSDB relies on AAA certificate manager. Looking at that code, the keystore was only being read from when OVSDB was started. I pushed a patch which I tested on the same setup and works: https://git.opendaylight.org/gerrit/#/c/70038/
Tim Rozet Red Hat SDN Team On Tue, Mar 20, 2018 at 4:12 PM, Tim Rozet <[email protected]> wrote: > Also if I use keytool on the truststore I can see the entry: > ()[root@overcloud-controller-0 ssl]# keytool -list -v -keystore > truststore.jks > [34/1923] > Enter keystore password: > > > Keystore type: JKS > > > Keystore provider: SUN > > > > > > Your keystore contains 1 entry > > > > > > Alias name: overcloud-controller-0 > > > Creation date: Mar 20, 2018 > > > Entry type: trustedCertEntry > > > > > > Owner: CN=overcloud-controller-0.internalapi.opnfvlf.org, O=OPNFVLF.ORG > > > Issuer: CN=Certificate Authority, O=OPNFVLF.ORG > > > Serial number: 12b > > > Valid from: Tue Mar 20 09:34:11 UTC 2018 until: Fri Mar 20 09:34:11 UTC > 2020 > > Certificate fingerprints: > > > MD5: 46:1D:AB:A3:66:06:59:3C:1F:E6:83:6F:99:86:14:6A > > > SHA1: EB:0A:5D:8D:4C:BB:DA:D1:76:DD:52:5D:4B:B5:13:4F:09:EE:6B:4E > > > SHA256: CC:13:42:AE:75:9B:00:58:18:05: > 3B:A0:BE:A6:02:CF:98:54:E7:13:3E:27:51:DB:36:76:22:B7:B1:1C:B3:A0 > > > Signature algorithm name: SHA256withRSA > > > Subject Public Key Algorithm: 2048-bit RSA key > > > Version: 3 > > Tim Rozet > Red Hat SDN Team > > On Tue, Mar 20, 2018 at 4:03 PM, Tim Rozet <[email protected]> wrote: > >> Hi Mohamed, >> Just to clarify...we do create the ctl.jks and place it in >> configuration/ssl before ODL boots. The truststore is created on first >> boot, I can see truststore.jks in the same directory. I also believe that >> the certificate for OVS is in the truststore: >> >> [root@overcloud-controller-0 ~]# curl -k -X POST --fail --silent -H >> 'Content-Type: application/json' -H 'Cache-Control: no-cache' -u >> admin:admin -d '{ "aaa-cert-rpc:input": { "aaa-cert-rpc:node-alias": >> "overcloud-controller-0" }}' https://192.0.2.10:8081/restco >> nf/operations/aaa-cert-rpc:getNodeCertifcate >> {"output":{"node-cert":"MIIFGjCCBAKgAwIBAgICASswDQYJKoZIhvcN >> AQELBQAwNjEUMBIGA1UECgwLT1BORlZMRi5PUkcxHjAcBgNVBAMMFUNlcnRp >> ZmljYXRlIEF1dGhvcml0eTAeFw0xODAzMjAwOTM0MTFaFw0yMDAzMjAwOTM0 >> MTFaME8xFDASBgNVBAoMC09QTkZWTEYuT1JHMTcwNQYDVQQDDC5vdmVyY2xv >> dWQtY29udHJvbGxlci0wLmludGVybmFsYXBpLm9wbmZ2bGYub3JnMIIBIjAN >> BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoLDetkUZscbmllZNKc9u1B9e >> DNW4dptIHn8PpUNG6cHnwLhmuynRDwgI0zYPaoZnFy6QzyeCtpVARgqpUDGB >> 0SKyHPoO38YYwXuEUfK27mpfCFsZ0vKwmt4827fqglUQXMNlypyaPKdrcx1w >> 0z++E8x5DUixgL+trjjWtDBlooFkGs/qVoWFgySUvw/+CZ/mfg3G3W7oVX1k >> 3zqf75wxsqzM7ZePGn7W0UPuPjNkkOOI9Hl8PFoLeLwZXuYaAaJmz3O6Uffv >> RZgl3aSgPuCK31fM5tIfeCNnlQMxMdZZiPrbRpAi9O5vpaXxM/ >> d6r7HkTI6N9l0g/hX7P53hearImQIDAQABo4ICFzCCAhMwHwYDVR0jBBgwFo >> AUEGBKlCQqJ05za3ZwFRfuCW0ae0owPQYIKwYBBQUHAQEEMTAvMC0GCCsGAQ >> UFBzABhiFodHRwOi8vaXBhLWNhLm9wbmZ2bGYub3JnL2NhL29jc3AwDgYDVR >> 0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjB2Bg >> NVHR8EbzBtMGugM6Axhi9odHRwOi8vaXBhLWNhLm9wbmZ2bGYub3JnL2lwYS >> 9jcmwvTWFzdGVyQ1JMLmJpbqI0pDIwMDEOMAwGA1UECgwFaXBhY2ExHjAcBg >> NVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAdBgNVHQ4EFgQUcySBEFDXvF >> w45hSfRhLgTlpxcrowgeoGA1UdEQSB4jCB34Iub3ZlcmNsb3VkLWNvbnRyb2 >> xsZXItMC5pbnRlcm5hbGFwaS5vcG5mdmxmLm9yZ6BOBgorBgEEAYI3FAIDoE >> AMPm92cy9vdmVyY2xvdWQtY29udHJvbGxlci0wLmludGVybmFsYXBpLm9wbm >> Z2bGYub3JnQE9QTkZWTEYuT1JHoF0GBisGAQUCAqBTMFGgDRsLT1BORlZMRi >> 5PUkehQDA+oAMCAQGhNzA1GwNvdnMbLm92ZXJjbG91ZC1jb250cm9sbGVyLT >> AuaW50ZXJuYWxhcGkub3BuZnZsZi5vcmcwDQYJKoZIhvcNAQELBQADggEBAF >> +7W//ICAw0i+FxPr9hSG2rlWcauo0Zllk2ff5e9NGbf7uu7aJVINYRVpGW7I >> N0kgsq63QuCGZu7y/zCiPb5hmWuBGHKgqHFXTjLUXGmSQXJ5yEuE7yLTmN5o >> HbXxJgbXgJoy7PDOLziAnWRQkrScBvLNZur7+06FurN2UPtCMfrnn8cmbite >> LBW6sdnzCFEUlrkU6bHOFPJWwAGski6YpdHYKcnw4p3J22phABCSRe8dq7xb >> IZx0wBg6WaSGTlwshqUSFEDX4VrjtVoWE06ipTSbHR3Ra93kh1dEd3VdMPND >> /CdflsIOcsiykD7SZl7oli9c35asqjipOAvI8HCn8="}} >> >> >> Let me know if there are any other debug outputs that could be helpful. >> >> Thanks, >> >> Tim Rozet >> Red Hat SDN Team >> >> On Tue, Mar 20, 2018 at 2:17 PM, Mohamed El-Serngawy < >> [email protected]> wrote: >> >>> Sorry Tim, just confused ignore my previous email the keystore will be >>> created anyway [1]. "didn't maintain this code for a long time" >>> >>> [1] https://github.com/opendaylight/aaa/blob/master/aaa-cert >>> /src/main/java/org/opendaylight/aaa/cert/impl/AaaCertProvider.java#L89 >>> >>> On Tue, Mar 20, 2018 at 2:10 PM, Mohamed El-Serngawy < >>> [email protected]> wrote: >>> >>>> Hi Tim, >>>> >>>> This most properly mean that the ovs certificate is not in the trust >>>> keystore. Just need to clarify things, based on the your previous email >>>> you mentioned >>>> "We create a JKS for the controller keystore. For the trust store I >>>> believe ODL creates it on boot" >>>> >>>> Is this mean you create the ctl.jks file place it under /configuration/ssl/ >>>> ? if yes, so I guess the issue at [0]. The certificate manager at the first >>>> time start check if the keystores files are there, If not create them. Let >>>> me know if my understand is correct ? You may do one more thing to confirm, >>>> just create an empty trust keystore file with the respect to the config at >>>> aaa-cert-config.xml and check if it will gonna work without restart. >>>> >>>> >>>> [0] https://github.com/opendaylight/aaa/blob/master/aaa-cert >>>> /src/main/java/org/opendaylight/aaa/cert/impl/CertificateMan >>>> agerService.java#L70 >>>> >>>> >>>> Thanks >>>> >>>> On Tue, Mar 20, 2018 at 12:19 PM, Tim Rozet <[email protected]> wrote: >>>> >>>>> Hi Mohamed, >>>>> I managed to reproduce the issue. I flipped on debugging and I see: >>>>> >>>>> 2018-03-20 11:07:09,492 | INFO | entLoopGroup-4-1 | LoggingHandler >>>>> | 60 - io.netty.common - 4.1.16.Final | [id: 0xfe0d155e, >>>>> L:/ >>>>> 0.0.0.0:6640] READ COMPLETE >>>>> 2018-03-20 11:07:09,492 | DEBUG | entLoopGroup-5-4 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | New Passive channel created : [id: 0xae5053bf, L:/ >>>>> 192.0.2.8:6640 - R:/192.0.2.8:51224] >>>>> 2018-03-20 11:07:09,593 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | Handshake status NEED_UNWRAP >>>>> 2018-03-20 11:07:09,593 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | handshake not done yet NEED_UNWRAP >>>>> 2018-03-20 11:07:09,693 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | Handshake status NEED_UNWRAP >>>>> 2018-03-20 11:07:09,693 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | handshake not done yet NEED_UNWRAP >>>>> 2018-03-20 11:07:09,793 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | Handshake status NEED_UNWRAP >>>>> 2018-03-20 11:07:09,794 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | handshake not done yet NEED_UNWRAP >>>>> 2018-03-20 11:07:09,894 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | Handshake status NEED_UNWRAP >>>>> 2018-03-20 11:07:09,894 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | handshake not done yet NEED_UNWRAP >>>>> 2018-03-20 11:07:09,894 | DEBUG | assiveConnServ-5 | >>>>> OvsdbConnectionService | 399 - org.opendaylight.ovsdb.library - >>>>> 1.6.0.SNAPSHOT | channel closed [id: 0xae5053bf, L:/192.0.2.8:6640 ! >>>>> R:/192.0.2.8:51224] >>>>> >>>>> Which looks like it is coming from here: >>>>> https://github.com/opendaylight/ovsdb/blob/e6b469e18d5f72402 >>>>> ccb817ce1fb1469dd2a9d6c/library/impl/src/main/java/org/opend >>>>> aylight/ovsdb/lib/impl/OvsdbConnectionService.java#L439 >>>>> >>>>> Tim Rozet >>>>> Red Hat SDN Team >>>>> >>>>> On Mon, Mar 19, 2018 at 11:53 AM, Mohamed El-Serngawy < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Sam >>>>>> >>>>>> I'm not quit sure what could be causing this issue, suppose there is >>>>>> no need to restart. "getServerContext()" will be executed every >>>>>> time a connection established. Will try ti reproduce it today. >>>>>> >>>>>> thanks >>>>>> >>>>>> >>>>>> On Sun, Mar 18, 2018 at 11:01 AM, Sam Hague <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Mohamed, >>>>>>> >>>>>>> Tim is using the JKS - he pushes all that before connecting the OVS >>>>>>> nodes to ODL. >>>>>>> >>>>>>> Do you know if there are any timing with the JKS when ODL starts >>>>>>> compared to when certs are added via rest? ovsdb southbound stats up and >>>>>>> has the certificateManager >>>>>>> which it uses to start the netty listening on 6640. Then client >>>>>>> certs are included to the ODL via rest. Then connections attempted from >>>>>>> the >>>>>>> ovs nodes but they never connect. Reboot ODL and the connections then >>>>>>> work. >>>>>>> >>>>>>> Could there be something in the reboot which actally gets the client >>>>>>> certs applied? >>>>>>> >>>>>>> Or does the server context change when cert are applied? At startup >>>>>>> the ovsdb southbound does certManagerSrv.getServerContext() and >>>>>>> opens the listening channel. That same context is used when the incoming >>>>>>> connections come in - ovsdb does not do another read of that context. >>>>>>> >>>>>>> Thanks, Sam >>>>>>> >>>>>>> On Thu, Mar 15, 2018 at 3:03 PM, Tim Rozet <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Mohamed, >>>>>>>> Right, that is one of the wiki pages I followed. There are several >>>>>>>> that I kind of had to merge the info together to get it to all work. >>>>>>>> The >>>>>>>> read from the trust store should work. I tested it manually and we >>>>>>>> have an >>>>>>>> unless here in puppet so we do not re-add the cert: >>>>>>>> https://github.com/openstack/puppet-neutron/blob/master/mani >>>>>>>> fests/plugins/ovs/opendaylight.pp#L191 >>>>>>>> >>>>>>>> We create a JKS for the controller keystore. For the trust store I >>>>>>>> believe ODL creates it on boot based on this config: >>>>>>>> https://git.opendaylight.org/gerrit/gitweb?p=integration/pac >>>>>>>> kaging/puppet-opendaylight.git;a=blob;f=templates/aaa-cert-c >>>>>>>> onfig.xml.erb;h=d6faa891630cba1c4747f64ea977d07de08c6b65;hb= >>>>>>>> refs/heads/master >>>>>>>> >>>>>>>> >>>>>>>> Tim Rozet >>>>>>>> Red Hat SDN Team >>>>>>>> >>>>>>>> On Thu, Mar 15, 2018 at 2:41 PM, Mohamed El-Serngawy < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> The logs attached with the bug is not really showing errors, Just >>>>>>>>> the aaa-cert service waiting for aaa-encryption service then it starts >>>>>>>>> fine. >>>>>>>>> >>>>>>>>> Tim, >>>>>>>>> >>>>>>>>> I assume you followed the link at [0] to configure the ssl. After >>>>>>>>> you add the OVS certificate using the REST API, can you just confirm >>>>>>>>> that >>>>>>>>> you are able to read the certificate from the trust-store ? are you >>>>>>>>> using >>>>>>>>> MDSAL or java Key Store files ? >>>>>>>>> >>>>>>>>> [0] https://wiki.opendaylight.org/view/OVSDB_Integration:TLS >>>>>>>>> _Communication >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 15, 2018 at 2:27 PM, Luis Gomez <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I do not remember that issue when I tested OF TLS in the past, I >>>>>>>>>> will have to retest to confirm. >>>>>>>>>> >>>>>>>>>> On Mar 15, 2018, at 11:24 AM, Tim Rozet <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Luis, >>>>>>>>>> To clarify we are not talking about SSL configuration here. We >>>>>>>>>> indeed configure the file you mentioned along with other config >>>>>>>>>> files pax >>>>>>>>>> web, ovsdb to only allow SSL/TLS, creating controller and trust >>>>>>>>>> stores. >>>>>>>>>> This is all done prior to ODL starting. The failure here is that ODL >>>>>>>>>> allows a REST implementation to add certificates to the trust store >>>>>>>>>> for OVS >>>>>>>>>> switches (which obviously implies ODL is up and running). At deploy >>>>>>>>>> time, >>>>>>>>>> we generate certificates for OVS and then add them via REST to ODL. >>>>>>>>>> At >>>>>>>>>> that point ODL should trust the switch and allow connections. >>>>>>>>>> However, >>>>>>>>>> OVSDB never seems to read again from the trust store (unless >>>>>>>>>> rebooted) and >>>>>>>>>> does not allow the switch to connect. >>>>>>>>>> >>>>>>>>>> Tim Rozet >>>>>>>>>> Red Hat SDN Team >>>>>>>>>> >>>>>>>>>> On Thu, Mar 15, 2018 at 1:55 PM, Luis Gomez <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> AFAIR for ofp you need to modify this config file: >>>>>>>>>>> >>>>>>>>>>> /etc/opendaylight/datastore/initial/config/default-openflow- >>>>>>>>>>> connection-config.xml >>>>>>>>>>> >>>>>>>>>>> which means you have to reboot the controller after. >>>>>>>>>>> >>>>>>>>>>> BR/Luis >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mar 15, 2018, at 10:42 AM, Sam Hague <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Mo, and ofp devs, >>>>>>>>>>> >>>>>>>>>>> how do you handle openflow connections using ssl? We have the >>>>>>>>>>> bug below where ODL is required to be restarted to pick up >>>>>>>>>>> connections over >>>>>>>>>>> ssl. >>>>>>>>>>> >>>>>>>>>>> Is that a design requirement that ODL has to be restarted or is >>>>>>>>>>> there a different config that can be used? >>>>>>>>>>> >>>>>>>>>>> Thanks, Sam >>>>>>>>>>> >>>>>>>>>>> https://jira.opendaylight.org/browse/OVSDB-449 >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> integration-dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://lists.opendaylight.org/mailman/listinfo/integration-dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Mohamed ElSerngawy >>>>>>>>> >>>>>>>>> +1 438 993 2462 <(438)%20993-2462> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Mohamed ElSerngawy >>>>>> >>>>>> +1 438 993 2462 <(438)%20993-2462> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Mohamed ElSerngawy >>>> >>>> +1 438 993 2462 <(438)%20993-2462> >>>> >>> >>> >>> >>> -- >>> Mohamed ElSerngawy >>> >>> +1 438 993 2462 <(438)%20993-2462> >>> >> >> >
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
