Oops, I guess I responded too late, glad you figured it out :) For what it's worth, I do believe that TR 1.8 handles that case and converts the key to the proper format. I still recommend that you run at least 1.8 if you want HTTPS support.
Thanks, Dave On Tue, Apr 25, 2017 at 3:38 PM, Dave Neuman <[email protected]> wrote: > Hey Oren, > I believe there were some issues with TR support for HTTPS in 1.7. Can > you upgrade to at least the 1.8 version of Traffic Router and let us know > if you are still having issues? > If you are not already, you need to be using a newer version of curl that > supports SNI in order to successfully curl Traffic Router and request > HTTPS, though I cannot remember exactly what the minimum version needs to > be (I am running 7.43 on my laptop). > You might have better luck using the openssl command instead; something > like ```openssl s_client -showcerts -connect tr.<ds>.<cdn>.<domain>:443```. > > Thanks, > Dave > > On Tue, Apr 25, 2017 at 2:57 PM, Oren Shemesh <[email protected]> wrote: > >> Another bit of crucial information: >> >> The tomcat log (/opt/tomcat/logs/catalina.out) show multiple identical >> warning messages like below, for every connection attempt: >> >> Apr 25, 2017 8:32:02 PM >> com.comcast.cdn.traffic_control.traffic_router.secure.KeyManager >> chooseServerAlias >> WARNING: No certificate registry aliases matching >> tr.file-sharing.<cdn-domain> >> >> It also shows the below exceptions, *once every hour*. >> I guess this settles it, there is a problem when converting the data from >> sslkeys.json into a Java certificate. >> >> Another piece of information: This is an externally-created certificate, >> manually pasted into TO. >> It contains the expected wildcard Common Name: >> >> [orens@rd10 ~]$openssl x509 -noout -in wildcard.file-sharing.<cdn-dom >> ain>.crt >> -subject >> subject= /OU=Domain Control Validated/CN=*.file-sharing.<cdn-domain> >> [orens@rd10 ~]$ >> >> This is the first time we are using an externally generated certificate in >> TC. >> >> Any ideas ? >> >> I am sure that I accurately copied the gibberish from the .key, .crt and >> .csr files we got, into the TO dialog box. >> >> Here are the exceptions found, every hour, in catalina.log: >> >> SEVERE: Failed to convert certificate data from traffic ops to handshake >> data! InvalidKeySpecException: java.security.InvalidKeyException: >> IOException : algid parse error, not a sequence >> java.security.spec.InvalidKeySpecException: >> java.security.InvalidKeyException: IOException : algid parse error, not a >> sequence >> at >> sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyF >> actory.java:217) >> at java.security.KeyFactory.generatePrivate(KeyFactory.java:372) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Pkcs.< >> init>(Pkcs.java:34) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Pkcs8. >> <init>(Pkcs8.java:31) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Privat >> eKeyDecoder.decode(PrivateKeyDecoder.java:27) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif >> icateDataConverter.toHandshakeData(CertificateDataConverter.java:34) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif >> icateRegistry.importCertificateDataList(CertificateRegistry.java:61) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif >> icateDataListener.handleNotification(CertificateDataListener.java:44) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$Listen >> erWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754) >> at >> javax.management.NotificationBroadcasterSupport. >> handleNotification(NotificationBroadcasterSupport.java:275) >> at >> javax.management.NotificationBroadcasterSupport$ >> SendNotifJob.run(NotificationBroadcasterSupport.java:352) >> at >> javax.management.NotificationBroadcasterSupport$1.execute(No >> tificationBroadcasterSupport.java:337) >> at >> javax.management.NotificationBroadcasterSupport. >> sendNotification(NotificationBroadcasterSupport.java:248) >> at >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive >> ryServiceCertificates.setCertificateDataList(DeliveryService >> Certificates.java:40) >> at >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive >> ryServiceCertificates.setCertificateDataListString(DeliveryS >> erviceCertificates.java:51) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S >> tandardMBeanIntrospector.java:112) >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S >> tandardMBeanIntrospector.java:46) >> at >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter(MBean >> Introspector.java:267) >> at >> com.sun.jmx.mbeanserver.PerInterface.setAttribute(PerInterface.java:102) >> at >> com.sun.jmx.mbeanserver.MBeanSupport.setAttribute(MBeanSupport.java:230) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAtt >> ribute(DefaultMBeanServerInterceptor.java:746) >> at >> com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBean >> Server.java:739) >> at >> com.comcast.cdn.traffic_control.traffic_router.core.secure. >> CertificatesPublisher.publishCertificateList(CertificatesPub >> lisher.java:61) >> at >> com.comcast.cdn.traffic_control.traffic_router.core.secure. >> CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.security.InvalidKeyException: IOException : algid parse >> error, not a sequence >> at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:351) >> at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:356) >> at >> sun.security.rsa.RSAPrivateCrtKeyImpl.<init>(RSAPrivateCrtKe >> yImpl.java:91) >> at >> sun.security.rsa.RSAPrivateCrtKeyImpl.newKey(RSAPrivateCrtKe >> yImpl.java:75) >> at >> sun.security.rsa.RSAKeyFactory.generatePrivate(RSAKeyFactory.java:316) >> at >> sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyF >> actory.java:213) >> ... 34 more >> >> Apr 25, 2017 7:55:56 PM >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif >> icateDataListener >> handleNotification >> WARNING: Failed importing certificate data list into registry >> NullPointerException >> java.lang.NullPointerException >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif >> icateRegistry.importCertificateDataList(CertificateRegistry.java:62) >> at >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif >> icateDataListener.handleNotification(CertificateDataListener.java:44) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$Listen >> erWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754) >> at >> javax.management.NotificationBroadcasterSupport. >> handleNotification(NotificationBroadcasterSupport.java:275) >> at >> javax.management.NotificationBroadcasterSupport$ >> SendNotifJob.run(NotificationBroadcasterSupport.java:352) >> at >> javax.management.NotificationBroadcasterSupport$1.execute(No >> tificationBroadcasterSupport.java:337) >> at >> javax.management.NotificationBroadcasterSupport. >> sendNotification(NotificationBroadcasterSupport.java:248) >> at >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive >> ryServiceCertificates.setCertificateDataList(DeliveryService >> Certificates.java:40) >> at >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive >> ryServiceCertificates.setCertificateDataListString(DeliveryS >> erviceCertificates.java:51) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S >> tandardMBeanIntrospector.java:112) >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S >> tandardMBeanIntrospector.java:46) >> at >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter(MBean >> Introspector.java:267) >> at >> com.sun.jmx.mbeanserver.PerInterface.setAttribute(PerInterface.java:102) >> at >> com.sun.jmx.mbeanserver.MBeanSupport.setAttribute(MBeanSupport.java:230) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAtt >> ribute(DefaultMBeanServerInterceptor.java:746) >> at >> com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBean >> Server.java:739) >> at >> com.comcast.cdn.traffic_control.traffic_router.core.secure. >> CertificatesPublisher.publishCertificateList(CertificatesPub >> lisher.java:61) >> at >> com.comcast.cdn.traffic_control.traffic_router.core.secure. >> CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> >> I would be greatfull for any advice. >> >> >> >> On Tue, Apr 25, 2017 at 11:00 PM, Oren Shemesh <[email protected]> wrote: >> >> > Hi TC developers and users, >> > >> > I would appreciate any help making a Traffic Router (=TR) respond to >> HTTPS >> > requests. >> > I am setting up one of our TC setups (Version 1.7) with an HTTPS DS for >> > the first time, and the TR misbehaves. >> > I have configured the DS for HTTPS, and setup Riak like I normally do >> > (This is not the first time I have installed TC 1.7 with Riak), and >> loaded >> > a certificate into the DS in Traffic Ops (=TO), and generated CrConfig. >> > >> > However, when attempting to connect to the TR using HTTPS, it responds >> to >> > the "Client Hello" message with an "Alert (21): Handshake Failure (40)" >> > message (Wireshark is cool). >> > This is shown by curl as one of the following messages (Depending on the >> > curl version used): >> > >> > $curl https://tr.file-sharing.<cdn-domain>/some-file >> > curl: (35) Cannot communicate securely with peer: no common encryption >> > algorithm(s). >> > Or: >> > curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 >> alert >> > handshake failure >> > >> > (Throughout this message, I have replaced actual CDN/domain names with >> the >> > strings <cdn-name>, <tc-domain> and <cdn-domain>, as these are >> > potentially customer sensitive data). >> > >> > When using HTTP and not HTTPS, I get the expected 302 response from TR. >> > >> > If you have any experience in dealing with such a situation, I would >> > appreciate any advice, the sooner the better. >> > >> > >> > Here is what I could dig so far: >> > >> > TR responds with an "Alert: Handshake Failure" when the "Client Hello" >> > message contains a host name which does not match any of the >> certificates >> > it has. >> > (I have another TR that responds well to proper HTTPS requests. I opened >> > an HTTPS connection to it, with a "Client Hello" host name which does >> not >> > match any DS, and it responded exactly like the misbehaving TR responds, >> > with an "Alert: Handshake Failure" message.) >> > Which leads me to believe that the problem is that the TR does not have >> > the certificate for this DS"file-sharing". >> > >> > However: >> > >> > According to the TR log files, it attempts to GET the sslkeys.json from >> TO >> > every hour. >> > The TO access log shows that it responds to these GET requests with 200, >> > so I believe that TR has the certificates. >> > After every such hourly GET operation, there is an error message in the >> TR >> > log. To the best of my understanding, this error message is not real, >> there >> > is a bug in the code, causing it to emit an error message for every >> domain >> > being looked at, even when there is no error. >> > >> > Here is such a set of hourly TR log messages : >> > >> > INFO 2017-04-25T17:09:14.411 [pool-2-thread-1] com.comcast.cdn.traffic_ >> > control.traffic_router.core.util.Fetcher - POSTing: https://ops >> > .<tc-domain>/api/1.1/user/login; timeout is 15000 >> > INFO 2017-04-25T17:09:14.439 [pool-2-thread-1] com.comcast.cdn.traffic_ >> > control.traffic_router.core.util.Fetcher - GETing: https://ops >> .<tc-domain> >> > /api/1.2/cdns/name/<cdn-name>/sslkeys.json; timeout is 15000 >> > ERROR 2017-04-25T17:09:14.811 [Thread-5] com.comcast.cdn.traffic_ >> > control.traffic_router.core.config.CertificateChecker - No certificate >> > data for https file-sharing domain <cdn-domain> >> > >> > And here is the respective TO access log message: >> > >> > <some-ip-address> - - [25/Apr/2017:17:09:33 +0000] "GET >> > /api/1.2/cdns/name/<cdn-name>/sslkeys.json HTTP/1.1" 200 5002 353167 >> > "Java/1.8.0_92" >> > >> > The TR log error message is due to a bug in core/src/main/java/com/ >> > comcast/cdn/traffic_control/traffic_router/core/config/ >> > CertificateChecker.java. >> > The function deliveryServiceHasValidCertificates() emits this error for >> > every DS domain name being checked, regardless of the presence (Or not) >> of >> > a certificate for the DS being checked. >> > >> > When I manually get the sslkeys.json file from traffic ops, I see that >> it >> > contains a certificate for the DS called "file-sharing", with a hostname >> > that is exactly what is should be: "hostname":"*.file-sharing.<cd >> n-domain> >> > ". >> > So I have no idea why tomcat behaves as if it does not have a proper >> > certificate. >> > >> > This is as far as I got. >> > >> > If you have any idea how to debug this further, I would appreciate any >> > advice. For example: >> > >> > 1. Is there a way to query the tomcat server, at run time, what >> > certificates it knows about ? >> > 2. Are there any relevant log messages that can be enabled ? If so, how >> do >> > I enable them ? >> > 3. Any idea as to any other thing worth checking ? >> > >> > Thanks a lot in advance, Oren. >> > -- >> > >> > *Oren Shemesh* >> > Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile: >> > +972-50-2281168 <+972%2050-228-1168> | [email protected] <[email protected]> >> > >> >> >> >> -- >> >> *Oren Shemesh* >> Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | [email protected] >> <[email protected]> >> > >
