Well, Issue solved, I am posting for completeness: The problem was that the .key file I used was not in the format expected by TO. After converting with this utility:
openssl pkcs8 -topk8 -nocrypt -in mykey.key -out mykey.pcks8 The text found in the .pcks8 file is apparently in the proper format, because TR responds properly now. All, Have a good day :-) On Tue, Apr 25, 2017 at 11:57 PM, Oren Shemesh <or...@qwilt.com> wrote: > Another bit of crucial information: > > The tomcat log (/opt/tomcat/logs/catalina.out) show multiple identical > warning messages like below, for every connection attempt: > > Apr 25, 2017 8:32:02 PM com.comcast.cdn.traffic_ > control.traffic_router.secure.KeyManager chooseServerAlias > WARNING: No certificate registry aliases matching > tr.file-sharing.<cdn-domain> > > It also shows the below exceptions, *once every hour*. > I guess this settles it, there is a problem when converting the data from > sslkeys.json into a Java certificate. > > Another piece of information: This is an externally-created certificate, > manually pasted into TO. > It contains the expected wildcard Common Name: > > [orens@rd10 ~]$openssl x509 -noout -in wildcard.file-sharing.<cdn-domain>.crt > -subject > subject= /OU=Domain Control Validated/CN=*.file-sharing.<cdn-domain> > [orens@rd10 ~]$ > > This is the first time we are using an externally generated certificate in > TC. > > Any ideas ? > > I am sure that I accurately copied the gibberish from the .key, .crt and > .csr files we got, into the TO dialog box. > > Here are the exceptions found, every hour, in catalina.log: > > SEVERE: Failed to convert certificate data from traffic ops to handshake > data! InvalidKeySpecException: java.security.InvalidKeyException: > IOException : algid parse error, not a sequence > java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: > IOException : algid parse error, not a sequence > at sun.security.rsa.RSAKeyFactory.engineGeneratePrivate( > RSAKeyFactory.java:217) > at java.security.KeyFactory.generatePrivate(KeyFactory.java:372) > at com.comcast.cdn.traffic_control.traffic_router.secure. > Pkcs.<init>(Pkcs.java:34) > at com.comcast.cdn.traffic_control.traffic_router.secure. > Pkcs8.<init>(Pkcs8.java:31) > at com.comcast.cdn.traffic_control.traffic_router.secure. > PrivateKeyDecoder.decode(PrivateKeyDecoder.java:27) > at com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataConverter.toHandshakeData(CertificateDataConverter.java:34) > at com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateRegistry.importCertificateDataList(CertificateRegistry.java:61) > at com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataListener.handleNotification(CertificateDataListener.java: > 44) > at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ > ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor. > java:1754) > at javax.management.NotificationBroadcasterSupport > .handleNotification(NotificationBroadcasterSupport.java:275) > at javax.management.NotificationBroadcasterSupport > $SendNotifJob.run(NotificationBroadcasterSupport.java:352) > at javax.management.NotificationBroadcasterSupport$1.execute( > NotificationBroadcasterSupport.java:337) > at javax.management.NotificationBroadcasterSupport > .sendNotification(NotificationBroadcasterSupport.java:248) > at com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataList( > DeliveryServiceCertificates.java:40) > at com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataListString( > DeliveryServiceCertificates.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:112) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:46) > at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter( > MBeanIntrospector.java:267) > at com.sun.jmx.mbeanserver.PerInterface.setAttribute( > PerInterface.java:102) > at com.sun.jmx.mbeanserver.MBeanSupport.setAttribute( > MBeanSupport.java:230) > at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor. > setAttribute(DefaultMBeanServerInterceptor.java:746) > at com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute( > JmxMBeanServer.java:739) > at com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.publishCertificateList( > CertificatesPublisher.java:61) > at com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.security.InvalidKeyException: IOException : algid parse > error, not a sequence > at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:351) > at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:356) > at sun.security.rsa.RSAPrivateCrtKeyImpl.<init>( > RSAPrivateCrtKeyImpl.java:91) > at sun.security.rsa.RSAPrivateCrtKeyImpl.newKey( > RSAPrivateCrtKeyImpl.java:75) > at sun.security.rsa.RSAKeyFactory.generatePrivate( > RSAKeyFactory.java:316) > at sun.security.rsa.RSAKeyFactory.engineGeneratePrivate( > RSAKeyFactory.java:213) > ... 34 more > > Apr 25, 2017 7:55:56 PM com.comcast.cdn.traffic_ > control.traffic_router.secure.CertificateDataListener handleNotification > WARNING: Failed importing certificate data list into registry > NullPointerException > java.lang.NullPointerException > at com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateRegistry.importCertificateDataList(CertificateRegistry.java:62) > at com.comcast.cdn.traffic_control.traffic_router.secure. > CertificateDataListener.handleNotification(CertificateDataListener.java: > 44) > at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ > ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor. > java:1754) > at javax.management.NotificationBroadcasterSupport > .handleNotification(NotificationBroadcasterSupport.java:275) > at javax.management.NotificationBroadcasterSupport > $SendNotifJob.run(NotificationBroadcasterSupport.java:352) > at javax.management.NotificationBroadcasterSupport$1.execute( > NotificationBroadcasterSupport.java:337) > at javax.management.NotificationBroadcasterSupport > .sendNotification(NotificationBroadcasterSupport.java:248) > at com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataList( > DeliveryServiceCertificates.java:40) > at com.comcast.cdn.traffic_control.traffic_router.shared. > DeliveryServiceCertificates.setCertificateDataListString( > DeliveryServiceCertificates.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:112) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2( > StandardMBeanIntrospector.java:46) > at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter( > MBeanIntrospector.java:267) > at com.sun.jmx.mbeanserver.PerInterface.setAttribute( > PerInterface.java:102) > at com.sun.jmx.mbeanserver.MBeanSupport.setAttribute( > MBeanSupport.java:230) > at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor. > setAttribute(DefaultMBeanServerInterceptor.java:746) > at com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute( > JmxMBeanServer.java:739) > at com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.publishCertificateList( > CertificatesPublisher.java:61) > at com.comcast.cdn.traffic_control.traffic_router.core. > secure.CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42) > at java.lang.Thread.run(Thread.java:745) > > > > > I would be greatfull for any advice. > > > > On Tue, Apr 25, 2017 at 11:00 PM, Oren Shemesh <or...@qwilt.com> wrote: > >> Hi TC developers and users, >> >> I would appreciate any help making a Traffic Router (=TR) respond to >> HTTPS requests. >> I am setting up one of our TC setups (Version 1.7) with an HTTPS DS for >> the first time, and the TR misbehaves. >> I have configured the DS for HTTPS, and setup Riak like I normally do >> (This is not the first time I have installed TC 1.7 with Riak), and loaded >> a certificate into the DS in Traffic Ops (=TO), and generated CrConfig. >> >> However, when attempting to connect to the TR using HTTPS, it responds to >> the "Client Hello" message with an "Alert (21): Handshake Failure (40)" >> message (Wireshark is cool). >> This is shown by curl as one of the following messages (Depending on the >> curl version used): >> >> $curl https://tr.file-sharing.<cdn-domain>/some-file >> curl: (35) Cannot communicate securely with peer: no common encryption >> algorithm(s). >> Or: >> curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 >> alert handshake failure >> >> (Throughout this message, I have replaced actual CDN/domain names with >> the strings <cdn-name>, <tc-domain> and <cdn-domain>, as these are >> potentially customer sensitive data). >> >> When using HTTP and not HTTPS, I get the expected 302 response from TR. >> >> If you have any experience in dealing with such a situation, I would >> appreciate any advice, the sooner the better. >> >> >> Here is what I could dig so far: >> >> TR responds with an "Alert: Handshake Failure" when the "Client Hello" >> message contains a host name which does not match any of the certificates >> it has. >> (I have another TR that responds well to proper HTTPS requests. I opened >> an HTTPS connection to it, with a "Client Hello" host name which does not >> match any DS, and it responded exactly like the misbehaving TR responds, >> with an "Alert: Handshake Failure" message.) >> Which leads me to believe that the problem is that the TR does not have >> the certificate for this DS"file-sharing". >> >> However: >> >> According to the TR log files, it attempts to GET the sslkeys.json from >> TO every hour. >> The TO access log shows that it responds to these GET requests with 200, >> so I believe that TR has the certificates. >> After every such hourly GET operation, there is an error message in the >> TR log. To the best of my understanding, this error message is not real, >> there is a bug in the code, causing it to emit an error message for every >> domain being looked at, even when there is no error. >> >> Here is such a set of hourly TR log messages : >> >> INFO 2017-04-25T17:09:14.411 [pool-2-thread-1] >> com.comcast.cdn.traffic_control.traffic_router.core.util.Fetcher - >> POSTing: https://ops.<tc-domain>/api/1.1/user/login; timeout is 15000 >> INFO 2017-04-25T17:09:14.439 [pool-2-thread-1] >> com.comcast.cdn.traffic_control.traffic_router.core.util.Fetcher - >> GETing: https://ops.<tc-domain>/api/1.2/cdns/name/<cdn-name>/sslkeys.json; >> timeout is 15000 >> ERROR 2017-04-25T17:09:14.811 [Thread-5] com.comcast.cdn.traffic_contro >> l.traffic_router.core.config.CertificateChecker - No certificate data >> for https file-sharing domain <cdn-domain> >> >> And here is the respective TO access log message: >> >> <some-ip-address> - - [25/Apr/2017:17:09:33 +0000] "GET >> /api/1.2/cdns/name/<cdn-name>/sslkeys.json HTTP/1.1" 200 5002 353167 >> "Java/1.8.0_92" >> >> The TR log error message is due to a bug in core/src/main/java/com/comc >> ast/cdn/traffic_control/traffic_router/core/config/Certifica >> teChecker.java. >> The function deliveryServiceHasValidCertificates() emits this error for >> every DS domain name being checked, regardless of the presence (Or not) of >> a certificate for the DS being checked. >> >> When I manually get the sslkeys.json file from traffic ops, I see that it >> contains a certificate for the DS called "file-sharing", with a hostname >> that is exactly what is should be: "hostname":"*.file-sharing. >> <cdn-domain>". >> So I have no idea why tomcat behaves as if it does not have a proper >> certificate. >> >> This is as far as I got. >> >> If you have any idea how to debug this further, I would appreciate any >> advice. For example: >> >> 1. Is there a way to query the tomcat server, at run time, what >> certificates it knows about ? >> 2. Are there any relevant log messages that can be enabled ? If so, how >> do I enable them ? >> 3. Any idea as to any other thing worth checking ? >> >> Thanks a lot in advance, Oren. >> -- >> >> *Oren Shemesh* >> Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile: >> +972-50-2281168 <+972%2050-228-1168> | or...@qwilt.com <y...@qwilt.com> >> > > > > -- > > *Oren Shemesh* > Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile: > +972-50-2281168 <+972%2050-228-1168> | or...@qwilt.com <y...@qwilt.com> > -- *Oren Shemesh* Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | or...@qwilt.com <y...@qwilt.com>
