Hey Oren, I believe there were some issues with TR support for HTTPS in 1.7. Can you upgrade to at least the 1.8 version of Traffic Router and let us know if you are still having issues? If you are not already, you need to be using a newer version of curl that supports SNI in order to successfully curl Traffic Router and request HTTPS, though I cannot remember exactly what the minimum version needs to be (I am running 7.43 on my laptop). You might have better luck using the openssl command instead; something like ```openssl s_client -showcerts -connect tr.<ds>.<cdn>.<domain>:443```.
Thanks, Dave On Tue, Apr 25, 2017 at 2:57 PM, Oren Shemesh <[email protected]> wrote: > Another bit of crucial information: > > The tomcat log (/opt/tomcat/logs/catalina.out) show multiple identical > warning messages like below, for every connection attempt: > > Apr 25, 2017 8:32:02 PM > com.comcast.cdn.traffic_control.traffic_router.secure.KeyManager > chooseServerAlias > WARNING: No certificate registry aliases matching > tr.file-sharing.<cdn-domain> > > It also shows the below exceptions, *once every hour*. > I guess this settles it, there is a problem when converting the data from > sslkeys.json into a Java certificate. > > Another piece of information: This is an externally-created certificate, > manually pasted into TO. > It contains the expected wildcard Common Name: > > [orens@rd10 ~]$openssl x509 -noout -in wildcard.file-sharing.<cdn- > domain>.crt > -subject > subject= /OU=Domain Control Validated/CN=*.file-sharing.<cdn-domain> > [orens@rd10 ~]$ > > This is the first time we are using an externally generated certificate in > TC. > > Any ideas ? > > I am sure that I accurately copied the gibberish from the .key, .crt and > .csr files we got, into the TO dialog box. > > Here are the exceptions found, every hour, in catalina.log: > > SEVERE: Failed to convert certificate data from traffic ops to handshake > data! InvalidKeySpecException: java.security.InvalidKeyException: > IOException : algid parse error, not a sequence > java.security.spec.InvalidKeySpecException: > java.security.InvalidKeyException: IOException : algid parse error, not a > sequence > at > sun.security.rsa.RSAKeyFactory.engineGeneratePrivate( > RSAKeyFactory.java:217) > at java.security.KeyFactory.generatePrivate(KeyFactory.java:372) > at > com.comcast.cdn.traffic_control.traffic_router.secure. > Pkcs.<init>(Pkcs.java:34) > at > com.comcast.cdn.traffic_control.traffic_router.secure. > Pkcs8.<init>(Pkcs8.java:31) > at > com.comcast.cdn.traffic_control.traffic_router.secure. > PrivateKeyDecoder.decode(PrivateKeyDecoder.java:27) > at > com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataConverter.toHandshakeData(CertificateDataConverter.java:34) > at > com.comcast.cdn.traffic_control.traffic_router.secure.CertificateRegistry. > importCertificateDataList(CertificateRegistry.java:61) > at > com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataListener.handleNotification(CertificateDataListener.java: > 44) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper. > handleNotification(DefaultMBeanServerInterceptor.java:1754) > at > javax.management.NotificationBroadcasterSupport.handleNotification( > NotificationBroadcasterSupport.java:275) > at > javax.management.NotificationBroadcasterSupport$SendNotifJob.run( > NotificationBroadcasterSupport.java:352) > at > javax.management.NotificationBroadcasterSupport$1.execute( > NotificationBroadcasterSupport.java:337) > at > javax.management.NotificationBroadcasterSupport.sendNotification( > NotificationBroadcasterSupport.java:248) > at > com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataList( > DeliveryServiceCertificates.java:40) > at > com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataListString( > DeliveryServiceCertificates.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter( > MBeanIntrospector.java:267) > at > com.sun.jmx.mbeanserver.PerInterface.setAttribute(PerInterface.java:102) > at > com.sun.jmx.mbeanserver.MBeanSupport.setAttribute(MBeanSupport.java:230) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAttribute( > DefaultMBeanServerInterceptor.java:746) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute( > JmxMBeanServer.java:739) > at > com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.publishCertificateList( > CertificatesPublisher.java:61) > at > com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.security.InvalidKeyException: IOException : algid parse > error, not a sequence > at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:351) > at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:356) > at > sun.security.rsa.RSAPrivateCrtKeyImpl.<init>(RSAPrivateCrtKeyImpl.java:91) > at > sun.security.rsa.RSAPrivateCrtKeyImpl.newKey(RSAPrivateCrtKeyImpl.java:75) > at > sun.security.rsa.RSAKeyFactory.generatePrivate(RSAKeyFactory.java:316) > at > sun.security.rsa.RSAKeyFactory.engineGeneratePrivate( > RSAKeyFactory.java:213) > ... 34 more > > Apr 25, 2017 7:55:56 PM > com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataListener > handleNotification > WARNING: Failed importing certificate data list into registry > NullPointerException > java.lang.NullPointerException > at > com.comcast.cdn.traffic_control.traffic_router.secure.CertificateRegistry. > importCertificateDataList(CertificateRegistry.java:62) > at > com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataListener.handleNotification(CertificateDataListener.java: > 44) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper. > handleNotification(DefaultMBeanServerInterceptor.java:1754) > at > javax.management.NotificationBroadcasterSupport.handleNotification( > NotificationBroadcasterSupport.java:275) > at > javax.management.NotificationBroadcasterSupport$SendNotifJob.run( > NotificationBroadcasterSupport.java:352) > at > javax.management.NotificationBroadcasterSupport$1.execute( > NotificationBroadcasterSupport.java:337) > at > javax.management.NotificationBroadcasterSupport.sendNotification( > NotificationBroadcasterSupport.java:248) > at > com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataList( > DeliveryServiceCertificates.java:40) > at > com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataListString( > DeliveryServiceCertificates.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter( > MBeanIntrospector.java:267) > at > com.sun.jmx.mbeanserver.PerInterface.setAttribute(PerInterface.java:102) > at > com.sun.jmx.mbeanserver.MBeanSupport.setAttribute(MBeanSupport.java:230) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAttribute( > DefaultMBeanServerInterceptor.java:746) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute( > JmxMBeanServer.java:739) > at > com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.publishCertificateList( > CertificatesPublisher.java:61) > at > com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) > at java.lang.Thread.run(Thread.java:745) > > > > > I would be greatfull for any advice. > > > > On Tue, Apr 25, 2017 at 11:00 PM, Oren Shemesh <[email protected]> wrote: > > > Hi TC developers and users, > > > > I would appreciate any help making a Traffic Router (=TR) respond to > HTTPS > > requests. > > I am setting up one of our TC setups (Version 1.7) with an HTTPS DS for > > the first time, and the TR misbehaves. > > I have configured the DS for HTTPS, and setup Riak like I normally do > > (This is not the first time I have installed TC 1.7 with Riak), and > loaded > > a certificate into the DS in Traffic Ops (=TO), and generated CrConfig. > > > > However, when attempting to connect to the TR using HTTPS, it responds to > > the "Client Hello" message with an "Alert (21): Handshake Failure (40)" > > message (Wireshark is cool). > > This is shown by curl as one of the following messages (Depending on the > > curl version used): > > > > $curl https://tr.file-sharing.<cdn-domain>/some-file > > curl: (35) Cannot communicate securely with peer: no common encryption > > algorithm(s). > > Or: > > curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 > alert > > handshake failure > > > > (Throughout this message, I have replaced actual CDN/domain names with > the > > strings <cdn-name>, <tc-domain> and <cdn-domain>, as these are > > potentially customer sensitive data). > > > > When using HTTP and not HTTPS, I get the expected 302 response from TR. > > > > If you have any experience in dealing with such a situation, I would > > appreciate any advice, the sooner the better. > > > > > > Here is what I could dig so far: > > > > TR responds with an "Alert: Handshake Failure" when the "Client Hello" > > message contains a host name which does not match any of the certificates > > it has. > > (I have another TR that responds well to proper HTTPS requests. I opened > > an HTTPS connection to it, with a "Client Hello" host name which does not > > match any DS, and it responded exactly like the misbehaving TR responds, > > with an "Alert: Handshake Failure" message.) > > Which leads me to believe that the problem is that the TR does not have > > the certificate for this DS"file-sharing". > > > > However: > > > > According to the TR log files, it attempts to GET the sslkeys.json from > TO > > every hour. > > The TO access log shows that it responds to these GET requests with 200, > > so I believe that TR has the certificates. > > After every such hourly GET operation, there is an error message in the > TR > > log. To the best of my understanding, this error message is not real, > there > > is a bug in the code, causing it to emit an error message for every > domain > > being looked at, even when there is no error. > > > > Here is such a set of hourly TR log messages : > > > > INFO 2017-04-25T17:09:14.411 [pool-2-thread-1] com.comcast.cdn.traffic_ > > control.traffic_router.core.util.Fetcher - POSTing: https://ops > > .<tc-domain>/api/1.1/user/login; timeout is 15000 > > INFO 2017-04-25T17:09:14.439 [pool-2-thread-1] com.comcast.cdn.traffic_ > > control.traffic_router.core.util.Fetcher - GETing: https://ops > .<tc-domain> > > /api/1.2/cdns/name/<cdn-name>/sslkeys.json; timeout is 15000 > > ERROR 2017-04-25T17:09:14.811 [Thread-5] com.comcast.cdn.traffic_ > > control.traffic_router.core.config.CertificateChecker - No certificate > > data for https file-sharing domain <cdn-domain> > > > > And here is the respective TO access log message: > > > > <some-ip-address> - - [25/Apr/2017:17:09:33 +0000] "GET > > /api/1.2/cdns/name/<cdn-name>/sslkeys.json HTTP/1.1" 200 5002 353167 > > "Java/1.8.0_92" > > > > The TR log error message is due to a bug in core/src/main/java/com/ > > comcast/cdn/traffic_control/traffic_router/core/config/ > > CertificateChecker.java. > > The function deliveryServiceHasValidCertificates() emits this error for > > every DS domain name being checked, regardless of the presence (Or not) > of > > a certificate for the DS being checked. > > > > When I manually get the sslkeys.json file from traffic ops, I see that it > > contains a certificate for the DS called "file-sharing", with a hostname > > that is exactly what is should be: "hostname":"*.file-sharing.< > cdn-domain> > > ". > > So I have no idea why tomcat behaves as if it does not have a proper > > certificate. > > > > This is as far as I got. > > > > If you have any idea how to debug this further, I would appreciate any > > advice. For example: > > > > 1. Is there a way to query the tomcat server, at run time, what > > certificates it knows about ? > > 2. Are there any relevant log messages that can be enabled ? If so, how > do > > I enable them ? > > 3. Any idea as to any other thing worth checking ? > > > > Thanks a lot in advance, Oren. > > -- > > > > *Oren Shemesh* > > Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile: > > +972-50-2281168 <+972%2050-228-1168> | [email protected] <[email protected]> > > > > > > -- > > *Oren Shemesh* > Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | [email protected] > <[email protected]> >
