Thanks Dave, good to know ! On Wed, Apr 26, 2017 at 12:40 AM, Dave Neuman <[email protected]> wrote:
> Oops, I guess I responded too late, glad you figured it out :) > For what it's worth, I do believe that TR 1.8 handles that case and > converts the key to the proper format. I still recommend that you run at > least 1.8 if you want HTTPS support. > > Thanks, > Dave > > On Tue, Apr 25, 2017 at 3:38 PM, Dave Neuman <[email protected]> wrote: > > > Hey Oren, > > I believe there were some issues with TR support for HTTPS in 1.7. Can > > you upgrade to at least the 1.8 version of Traffic Router and let us know > > if you are still having issues? > > If you are not already, you need to be using a newer version of curl that > > supports SNI in order to successfully curl Traffic Router and request > > HTTPS, though I cannot remember exactly what the minimum version needs to > > be (I am running 7.43 on my laptop). > > You might have better luck using the openssl command instead; something > > like ```openssl s_client -showcerts -connect > tr.<ds>.<cdn>.<domain>:443```. > > > > Thanks, > > Dave > > > > On Tue, Apr 25, 2017 at 2:57 PM, Oren Shemesh <[email protected]> wrote: > > > >> Another bit of crucial information: > >> > >> The tomcat log (/opt/tomcat/logs/catalina.out) show multiple identical > >> warning messages like below, for every connection attempt: > >> > >> Apr 25, 2017 8:32:02 PM > >> com.comcast.cdn.traffic_control.traffic_router.secure.KeyManager > >> chooseServerAlias > >> WARNING: No certificate registry aliases matching > >> tr.file-sharing.<cdn-domain> > >> > >> It also shows the below exceptions, *once every hour*. > >> I guess this settles it, there is a problem when converting the data > from > >> sslkeys.json into a Java certificate. > >> > >> Another piece of information: This is an externally-created certificate, > >> manually pasted into TO. > >> It contains the expected wildcard Common Name: > >> > >> [orens@rd10 ~]$openssl x509 -noout -in wildcard.file-sharing.<cdn-dom > >> ain>.crt > >> -subject > >> subject= /OU=Domain Control Validated/CN=*.file-sharing.<cdn-domain> > >> [orens@rd10 ~]$ > >> > >> This is the first time we are using an externally generated certificate > in > >> TC. > >> > >> Any ideas ? > >> > >> I am sure that I accurately copied the gibberish from the .key, .crt and > >> .csr files we got, into the TO dialog box. > >> > >> Here are the exceptions found, every hour, in catalina.log: > >> > >> SEVERE: Failed to convert certificate data from traffic ops to handshake > >> data! InvalidKeySpecException: java.security.InvalidKeyException: > >> IOException : algid parse error, not a sequence > >> java.security.spec.InvalidKeySpecException: > >> java.security.InvalidKeyException: IOException : algid parse error, > not a > >> sequence > >> at > >> sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyF > >> actory.java:217) > >> at java.security.KeyFactory.generatePrivate(KeyFactory. > java:372) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Pkcs.< > >> init>(Pkcs.java:34) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Pkcs8. > >> <init>(Pkcs8.java:31) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Privat > >> eKeyDecoder.decode(PrivateKeyDecoder.java:27) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif > >> icateDataConverter.toHandshakeData(CertificateDataConverter.java:34) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif > >> icateRegistry.importCertificateDataList(CertificateRegistry.java:61) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif > >> icateDataListener.handleNotification(CertificateDataListener.java:44) > >> at > >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$Listen > >> erWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754) > >> at > >> javax.management.NotificationBroadcasterSupport. > >> handleNotification(NotificationBroadcasterSupport.java:275) > >> at > >> javax.management.NotificationBroadcasterSupport$ > >> SendNotifJob.run(NotificationBroadcasterSupport.java:352) > >> at > >> javax.management.NotificationBroadcasterSupport$1.execute(No > >> tificationBroadcasterSupport.java:337) > >> at > >> javax.management.NotificationBroadcasterSupport. > >> sendNotification(NotificationBroadcasterSupport.java:248) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive > >> ryServiceCertificates.setCertificateDataList(DeliveryService > >> Certificates.java:40) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive > >> ryServiceCertificates.setCertificateDataListString(DeliveryS > >> erviceCertificates.java:51) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > >> ssorImpl.java:62) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > >> thodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > >> ssorImpl.java:62) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > >> thodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > >> at > >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S > >> tandardMBeanIntrospector.java:112) > >> at > >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S > >> tandardMBeanIntrospector.java:46) > >> at > >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter(MBean > >> Introspector.java:267) > >> at > >> com.sun.jmx.mbeanserver.PerInterface.setAttribute( > PerInterface.java:102) > >> at > >> com.sun.jmx.mbeanserver.MBeanSupport.setAttribute( > MBeanSupport.java:230) > >> at > >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAtt > >> ribute(DefaultMBeanServerInterceptor.java:746) > >> at > >> com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBean > >> Server.java:739) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.core.secure. > >> CertificatesPublisher.publishCertificateList(CertificatesPub > >> lisher.java:61) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.core.secure. > >> CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) > >> at java.lang.Thread.run(Thread.java:745) > >> Caused by: java.security.InvalidKeyException: IOException : algid parse > >> error, not a sequence > >> at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:351) > >> at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:356) > >> at > >> sun.security.rsa.RSAPrivateCrtKeyImpl.<init>(RSAPrivateCrtKe > >> yImpl.java:91) > >> at > >> sun.security.rsa.RSAPrivateCrtKeyImpl.newKey(RSAPrivateCrtKe > >> yImpl.java:75) > >> at > >> sun.security.rsa.RSAKeyFactory.generatePrivate(RSAKeyFactory.java:316) > >> at > >> sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyF > >> actory.java:213) > >> ... 34 more > >> > >> Apr 25, 2017 7:55:56 PM > >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif > >> icateDataListener > >> handleNotification > >> WARNING: Failed importing certificate data list into registry > >> NullPointerException > >> java.lang.NullPointerException > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif > >> icateRegistry.importCertificateDataList(CertificateRegistry.java:62) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.secure.Certif > >> icateDataListener.handleNotification(CertificateDataListener.java:44) > >> at > >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$Listen > >> erWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754) > >> at > >> javax.management.NotificationBroadcasterSupport. > >> handleNotification(NotificationBroadcasterSupport.java:275) > >> at > >> javax.management.NotificationBroadcasterSupport$ > >> SendNotifJob.run(NotificationBroadcasterSupport.java:352) > >> at > >> javax.management.NotificationBroadcasterSupport$1.execute(No > >> tificationBroadcasterSupport.java:337) > >> at > >> javax.management.NotificationBroadcasterSupport. > >> sendNotification(NotificationBroadcasterSupport.java:248) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive > >> ryServiceCertificates.setCertificateDataList(DeliveryService > >> Certificates.java:40) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.shared.Delive > >> ryServiceCertificates.setCertificateDataListString(DeliveryS > >> erviceCertificates.java:51) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > >> ssorImpl.java:62) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > >> thodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > >> ssorImpl.java:62) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > >> thodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > >> at > >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S > >> tandardMBeanIntrospector.java:112) > >> at > >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(S > >> tandardMBeanIntrospector.java:46) > >> at > >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter(MBean > >> Introspector.java:267) > >> at > >> com.sun.jmx.mbeanserver.PerInterface.setAttribute( > PerInterface.java:102) > >> at > >> com.sun.jmx.mbeanserver.MBeanSupport.setAttribute( > MBeanSupport.java:230) > >> at > >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAtt > >> ribute(DefaultMBeanServerInterceptor.java:746) > >> at > >> com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBean > >> Server.java:739) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.core.secure. > >> CertificatesPublisher.publishCertificateList(CertificatesPub > >> lisher.java:61) > >> at > >> com.comcast.cdn.traffic_control.traffic_router.core.secure. > >> CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) > >> at java.lang.Thread.run(Thread.java:745) > >> > >> > >> > >> > >> I would be greatfull for any advice. > >> > >> > >> > >> On Tue, Apr 25, 2017 at 11:00 PM, Oren Shemesh <[email protected]> wrote: > >> > >> > Hi TC developers and users, > >> > > >> > I would appreciate any help making a Traffic Router (=TR) respond to > >> HTTPS > >> > requests. > >> > I am setting up one of our TC setups (Version 1.7) with an HTTPS DS > for > >> > the first time, and the TR misbehaves. > >> > I have configured the DS for HTTPS, and setup Riak like I normally do > >> > (This is not the first time I have installed TC 1.7 with Riak), and > >> loaded > >> > a certificate into the DS in Traffic Ops (=TO), and generated > CrConfig. > >> > > >> > However, when attempting to connect to the TR using HTTPS, it responds > >> to > >> > the "Client Hello" message with an "Alert (21): Handshake Failure > (40)" > >> > message (Wireshark is cool). > >> > This is shown by curl as one of the following messages (Depending on > the > >> > curl version used): > >> > > >> > $curl https://tr.file-sharing.<cdn-domain>/some-file > >> > curl: (35) Cannot communicate securely with peer: no common encryption > >> > algorithm(s). > >> > Or: > >> > curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 > >> alert > >> > handshake failure > >> > > >> > (Throughout this message, I have replaced actual CDN/domain names with > >> the > >> > strings <cdn-name>, <tc-domain> and <cdn-domain>, as these are > >> > potentially customer sensitive data). > >> > > >> > When using HTTP and not HTTPS, I get the expected 302 response from > TR. > >> > > >> > If you have any experience in dealing with such a situation, I would > >> > appreciate any advice, the sooner the better. > >> > > >> > > >> > Here is what I could dig so far: > >> > > >> > TR responds with an "Alert: Handshake Failure" when the "Client Hello" > >> > message contains a host name which does not match any of the > >> certificates > >> > it has. > >> > (I have another TR that responds well to proper HTTPS requests. I > opened > >> > an HTTPS connection to it, with a "Client Hello" host name which does > >> not > >> > match any DS, and it responded exactly like the misbehaving TR > responds, > >> > with an "Alert: Handshake Failure" message.) > >> > Which leads me to believe that the problem is that the TR does not > have > >> > the certificate for this DS"file-sharing". > >> > > >> > However: > >> > > >> > According to the TR log files, it attempts to GET the sslkeys.json > from > >> TO > >> > every hour. > >> > The TO access log shows that it responds to these GET requests with > 200, > >> > so I believe that TR has the certificates. > >> > After every such hourly GET operation, there is an error message in > the > >> TR > >> > log. To the best of my understanding, this error message is not real, > >> there > >> > is a bug in the code, causing it to emit an error message for every > >> domain > >> > being looked at, even when there is no error. > >> > > >> > Here is such a set of hourly TR log messages : > >> > > >> > INFO 2017-04-25T17:09:14.411 [pool-2-thread-1] > com.comcast.cdn.traffic_ > >> > control.traffic_router.core.util.Fetcher - POSTing: https://ops > >> > .<tc-domain>/api/1.1/user/login; timeout is 15000 > >> > INFO 2017-04-25T17:09:14.439 [pool-2-thread-1] > com.comcast.cdn.traffic_ > >> > control.traffic_router.core.util.Fetcher - GETing: https://ops > >> .<tc-domain> > >> > /api/1.2/cdns/name/<cdn-name>/sslkeys.json; timeout is 15000 > >> > ERROR 2017-04-25T17:09:14.811 [Thread-5] com.comcast.cdn.traffic_ > >> > control.traffic_router.core.config.CertificateChecker - No > certificate > >> > data for https file-sharing domain <cdn-domain> > >> > > >> > And here is the respective TO access log message: > >> > > >> > <some-ip-address> - - [25/Apr/2017:17:09:33 +0000] "GET > >> > /api/1.2/cdns/name/<cdn-name>/sslkeys.json HTTP/1.1" 200 5002 353167 > >> > "Java/1.8.0_92" > >> > > >> > The TR log error message is due to a bug in core/src/main/java/com/ > >> > comcast/cdn/traffic_control/traffic_router/core/config/ > >> > CertificateChecker.java. > >> > The function deliveryServiceHasValidCertificates() emits this error > for > >> > every DS domain name being checked, regardless of the presence (Or > not) > >> of > >> > a certificate for the DS being checked. > >> > > >> > When I manually get the sslkeys.json file from traffic ops, I see that > >> it > >> > contains a certificate for the DS called "file-sharing", with a > hostname > >> > that is exactly what is should be: "hostname":"*.file-sharing.<cd > >> n-domain> > >> > ". > >> > So I have no idea why tomcat behaves as if it does not have a proper > >> > certificate. > >> > > >> > This is as far as I got. > >> > > >> > If you have any idea how to debug this further, I would appreciate any > >> > advice. For example: > >> > > >> > 1. Is there a way to query the tomcat server, at run time, what > >> > certificates it knows about ? > >> > 2. Are there any relevant log messages that can be enabled ? If so, > how > >> do > >> > I enable them ? > >> > 3. Any idea as to any other thing worth checking ? > >> > > >> > Thanks a lot in advance, Oren. > >> > -- > >> > > >> > *Oren Shemesh* > >> > Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile: > >> > +972-50-2281168 <+972%2050-228-1168> | [email protected] < > [email protected]> > >> > > >> > >> > >> > >> -- > >> > >> *Oren Shemesh* > >> Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | > [email protected] > >> <[email protected]> > >> > > > > > -- *Oren Shemesh* Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | [email protected] <[email protected]>
