Another bit of crucial information:
The tomcat log (/opt/tomcat/logs/catalina.out) show multiple identical
warning messages like below, for every connection attempt:
Apr 25, 2017 8:32:02 PM
com.comcast.cdn.traffic_control.traffic_router.secure.KeyManager
chooseServerAlias
WARNING: No certificate registry aliases matching
tr.file-sharing.<cdn-domain>
It also shows the below exceptions, *once every hour*.
I guess this settles it, there is a problem when converting the data from
sslkeys.json into a Java certificate.
Another piece of information: This is an externally-created certificate,
manually pasted into TO.
It contains the expected wildcard Common Name:
[orens@rd10 ~]$openssl x509 -noout -in wildcard.file-sharing.<cdn-domain>.crt
-subject
subject= /OU=Domain Control Validated/CN=*.file-sharing.<cdn-domain>
[orens@rd10 ~]$
This is the first time we are using an externally generated certificate in
TC.
Any ideas ?
I am sure that I accurately copied the gibberish from the .key, .crt and
.csr files we got, into the TO dialog box.
Here are the exceptions found, every hour, in catalina.log:
SEVERE: Failed to convert certificate data from traffic ops to handshake
data! InvalidKeySpecException: java.security.InvalidKeyException:
IOException : algid parse error, not a sequence
java.security.spec.InvalidKeySpecException:
java.security.InvalidKeyException: IOException : algid parse error, not a
sequence
at
sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyFactory.java:217)
at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)
at
com.comcast.cdn.traffic_control.traffic_router.secure.Pkcs.<init>(Pkcs.java:34)
at
com.comcast.cdn.traffic_control.traffic_router.secure.Pkcs8.<init>(Pkcs8.java:31)
at
com.comcast.cdn.traffic_control.traffic_router.secure.PrivateKeyDecoder.decode(PrivateKeyDecoder.java:27)
at
com.comcast.cdn.traffic_control.traffic_router.secure.CertificateDataConverter.toHandshakeData(CertificateDataConverter.java:34)
at
com.comcast.cdn.traffic_control.traffic_router.secure.CertificateRegistry.importCertificateDataList(CertificateRegistry.java:61)
at
com.comcast.cdn.traffic_control.traffic_router.secure.CertificateDataListener.handleNotification(CertificateDataListener.java:44)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754)
at
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at
com.comcast.cdn.traffic_control.traffic_router.shared.DeliveryServiceCertificates.setCertificateDataList(DeliveryServiceCertificates.java:40)
at
com.comcast.cdn.traffic_control.traffic_router.shared.DeliveryServiceCertificates.setCertificateDataListString(DeliveryServiceCertificates.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter(MBeanIntrospector.java:267)
at
com.sun.jmx.mbeanserver.PerInterface.setAttribute(PerInterface.java:102)
at
com.sun.jmx.mbeanserver.MBeanSupport.setAttribute(MBeanSupport.java:230)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAttribute(DefaultMBeanServerInterceptor.java:746)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBeanServer.java:739)
at
com.comcast.cdn.traffic_control.traffic_router.core.secure.CertificatesPublisher.publishCertificateList(CertificatesPublisher.java:61)
at
com.comcast.cdn.traffic_control.traffic_router.core.secure.CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.InvalidKeyException: IOException : algid parse
error, not a sequence
at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:351)
at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:356)
at
sun.security.rsa.RSAPrivateCrtKeyImpl.<init>(RSAPrivateCrtKeyImpl.java:91)
at
sun.security.rsa.RSAPrivateCrtKeyImpl.newKey(RSAPrivateCrtKeyImpl.java:75)
at
sun.security.rsa.RSAKeyFactory.generatePrivate(RSAKeyFactory.java:316)
at
sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyFactory.java:213)
... 34 more
Apr 25, 2017 7:55:56 PM
com.comcast.cdn.traffic_control.traffic_router.secure.CertificateDataListener
handleNotification
WARNING: Failed importing certificate data list into registry
NullPointerException
java.lang.NullPointerException
at
com.comcast.cdn.traffic_control.traffic_router.secure.CertificateRegistry.importCertificateDataList(CertificateRegistry.java:62)
at
com.comcast.cdn.traffic_control.traffic_router.secure.CertificateDataListener.handleNotification(CertificateDataListener.java:44)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754)
at
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at
com.comcast.cdn.traffic_control.traffic_router.shared.DeliveryServiceCertificates.setCertificateDataList(DeliveryServiceCertificates.java:40)
at
com.comcast.cdn.traffic_control.traffic_router.shared.DeliveryServiceCertificates.setCertificateDataListString(DeliveryServiceCertificates.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeSetter(MBeanIntrospector.java:267)
at
com.sun.jmx.mbeanserver.PerInterface.setAttribute(PerInterface.java:102)
at
com.sun.jmx.mbeanserver.MBeanSupport.setAttribute(MBeanSupport.java:230)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAttribute(DefaultMBeanServerInterceptor.java:746)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBeanServer.java:739)
at
com.comcast.cdn.traffic_control.traffic_router.core.secure.CertificatesPublisher.publishCertificateList(CertificatesPublisher.java:61)
at
com.comcast.cdn.traffic_control.traffic_router.core.secure.CertificatesPublisher.lambda$new$1(CertificatesPublisher.java:42)
at java.lang.Thread.run(Thread.java:745)
I would be greatfull for any advice.
On Tue, Apr 25, 2017 at 11:00 PM, Oren Shemesh <[email protected]> wrote:
> Hi TC developers and users,
>
> I would appreciate any help making a Traffic Router (=TR) respond to HTTPS
> requests.
> I am setting up one of our TC setups (Version 1.7) with an HTTPS DS for
> the first time, and the TR misbehaves.
> I have configured the DS for HTTPS, and setup Riak like I normally do
> (This is not the first time I have installed TC 1.7 with Riak), and loaded
> a certificate into the DS in Traffic Ops (=TO), and generated CrConfig.
>
> However, when attempting to connect to the TR using HTTPS, it responds to
> the "Client Hello" message with an "Alert (21): Handshake Failure (40)"
> message (Wireshark is cool).
> This is shown by curl as one of the following messages (Depending on the
> curl version used):
>
> $curl https://tr.file-sharing.<cdn-domain>/some-file
> curl: (35) Cannot communicate securely with peer: no common encryption
> algorithm(s).
> Or:
> curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert
> handshake failure
>
> (Throughout this message, I have replaced actual CDN/domain names with the
> strings <cdn-name>, <tc-domain> and <cdn-domain>, as these are
> potentially customer sensitive data).
>
> When using HTTP and not HTTPS, I get the expected 302 response from TR.
>
> If you have any experience in dealing with such a situation, I would
> appreciate any advice, the sooner the better.
>
>
> Here is what I could dig so far:
>
> TR responds with an "Alert: Handshake Failure" when the "Client Hello"
> message contains a host name which does not match any of the certificates
> it has.
> (I have another TR that responds well to proper HTTPS requests. I opened
> an HTTPS connection to it, with a "Client Hello" host name which does not
> match any DS, and it responded exactly like the misbehaving TR responds,
> with an "Alert: Handshake Failure" message.)
> Which leads me to believe that the problem is that the TR does not have
> the certificate for this DS"file-sharing".
>
> However:
>
> According to the TR log files, it attempts to GET the sslkeys.json from TO
> every hour.
> The TO access log shows that it responds to these GET requests with 200,
> so I believe that TR has the certificates.
> After every such hourly GET operation, there is an error message in the TR
> log. To the best of my understanding, this error message is not real, there
> is a bug in the code, causing it to emit an error message for every domain
> being looked at, even when there is no error.
>
> Here is such a set of hourly TR log messages :
>
> INFO 2017-04-25T17:09:14.411 [pool-2-thread-1] com.comcast.cdn.traffic_
> control.traffic_router.core.util.Fetcher - POSTing: https://ops
> .<tc-domain>/api/1.1/user/login; timeout is 15000
> INFO 2017-04-25T17:09:14.439 [pool-2-thread-1] com.comcast.cdn.traffic_
> control.traffic_router.core.util.Fetcher - GETing: https://ops.<tc-domain>
> /api/1.2/cdns/name/<cdn-name>/sslkeys.json; timeout is 15000
> ERROR 2017-04-25T17:09:14.811 [Thread-5] com.comcast.cdn.traffic_
> control.traffic_router.core.config.CertificateChecker - No certificate
> data for https file-sharing domain <cdn-domain>
>
> And here is the respective TO access log message:
>
> <some-ip-address> - - [25/Apr/2017:17:09:33 +0000] "GET
> /api/1.2/cdns/name/<cdn-name>/sslkeys.json HTTP/1.1" 200 5002 353167
> "Java/1.8.0_92"
>
> The TR log error message is due to a bug in core/src/main/java/com/
> comcast/cdn/traffic_control/traffic_router/core/config/
> CertificateChecker.java.
> The function deliveryServiceHasValidCertificates() emits this error for
> every DS domain name being checked, regardless of the presence (Or not) of
> a certificate for the DS being checked.
>
> When I manually get the sslkeys.json file from traffic ops, I see that it
> contains a certificate for the DS called "file-sharing", with a hostname
> that is exactly what is should be: "hostname":"*.file-sharing.<cdn-domain>
> ".
> So I have no idea why tomcat behaves as if it does not have a proper
> certificate.
>
> This is as far as I got.
>
> If you have any idea how to debug this further, I would appreciate any
> advice. For example:
>
> 1. Is there a way to query the tomcat server, at run time, what
> certificates it knows about ?
> 2. Are there any relevant log messages that can be enabled ? If so, how do
> I enable them ?
> 3. Any idea as to any other thing worth checking ?
>
> Thanks a lot in advance, Oren.
> --
>
> *Oren Shemesh*
> Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile:
> +972-50-2281168 <+972%2050-228-1168> | [email protected] <[email protected]>
>
--
*Oren Shemesh*
Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | [email protected]
<[email protected]>
