Re: FIXED - Re: Broken SSL cert caused by router?
On 29/03/2015 11:56 PM, John Levine wrote: SSLCertificateChainFile /etc/ssl/certs/gd_bundle-g2-g1.crt I have actually fixed it. Yeah, that's always it. Back in the good aulde days all of the SSL certs one might buy were signed directly by the CA, but now more often than not there are intermediate certs, and a valid cert needs to be accompanied by all of the intermediate certs between it and the CA. What makes debugging hard is that browsers try to be helpful. If a server doesn't provide the intermediate certs, but the browser happens to have them in its cache from some other site, well, close enough and the SSL works. But if some other browser doesn't happen to have them, you lose. So if your SSL is flaky, check those intermediate certs first. R's, John With all this resolved, I'll note that I just reviewed draft-ietf-tls-sslv3-diediedie, which is in IETF Last Call prior to publication as an RFC. It deprecates the use of any version of SSL in favour of TLS 1.2 in the clientHello negotiations. Tom Taylor
Re: FIXED - Re: Broken SSL cert caused by router?
>SSLCertificateChainFile /etc/ssl/certs/gd_bundle-g2-g1.crt > >I have actually fixed it. Yeah, that's always it. Back in the good aulde days all of the SSL certs one might buy were signed directly by the CA, but now more often than not there are intermediate certs, and a valid cert needs to be accompanied by all of the intermediate certs between it and the CA. What makes debugging hard is that browsers try to be helpful. If a server doesn't provide the intermediate certs, but the browser happens to have them in its cache from some other site, well, close enough and the SSL works. But if some other browser doesn't happen to have them, you lose. So if your SSL is flaky, check those intermediate certs first. R's, John
Re: FIXED - Re: Broken SSL cert caused by router?
That's something I suspected at first, it but discounted when your said your laptop also failed at the site. The first intermediate you installed took care of anything with the newer root certificates installed. But for your older 10.4 Mac clients (which presumably haven't had a root certificate bundle update in a while) that wasn't enough - the new root needed to be provided since from their perspective it's an intermediate. M. Original Message From: Mike Sent: Sunday, March 29, 2015 23:29 To: nanog@nanog.org Subject: Re: FIXED - Re: Broken SSL cert caused by router? On 03/28/2015 01:50 PM, Matt Palmer wrote: > On Sat, Mar 28, 2015 at 09:05:38AM -0700, Mike wrote: >> On 03/27/2015 10:34 AM, Frank Bulk wrote: >>> Glad you figured that out. >>> >>> I've used three SSL evaluation websites to help me with intermediate >>> certificate issues: >>> https://www.ssllabs.com/ssltest/analyze.html (will show the names and >>> details of the certs, missing or not >>> https://www.wormly.com/test_ssl (quick SSL tester, will point out if >>> intermediate certificate is missing) >>> https://www.digicert.com/help/ (will show a green chain link between certs >>> when they're all there *and* in order) >> I went back to Frank's list and did some additional testing. I have a >> different server which was set up the same way as the previous one >> discussed, and I thought I would use the above tools and see if my problem >> would have been identified by any of them. I am sorry to report, no, none of >> these either caught the problem either. > Are you able to share the URL of the misconfigured site? It would be > interesting to examine exactly what's going on. > > - Matt > SSLCertificateChainFile /etc/ssl/certs/gd_bundle-g2-g1.crt I have actually fixed it. What was going on seems to be this - I have a new godaddy certificate for *.mydomain.com, and that is what I installed. However, the certificate chain I supplied was missing some intermediate godaddy certificate. Originally, it appeared I was missing 'gdig2.crt', and once installed, that fixed some clients including the ones behind the meraki router. But then there were also some older clients this did not fix (a macos 10.4 something for example). So I went back and installed gd_bundle-g2-g1.crt in it's place, and that seems to have finally done it. I apologize for the diminishing lack of operational content. It just seems that these ssl tests should be tightened up and perhaps some additional tools deployed out there to help us less knowledgeable folks 'get it right'. Mike-
Re: FIXED - Re: Broken SSL cert caused by router?
On 03/28/2015 01:50 PM, Matt Palmer wrote: On Sat, Mar 28, 2015 at 09:05:38AM -0700, Mike wrote: On 03/27/2015 10:34 AM, Frank Bulk wrote: Glad you figured that out. I've used three SSL evaluation websites to help me with intermediate certificate issues: https://www.ssllabs.com/ssltest/analyze.html (will show the names and details of the certs, missing or not https://www.wormly.com/test_ssl (quick SSL tester, will point out if intermediate certificate is missing) https://www.digicert.com/help/ (will show a green chain link between certs when they're all there *and* in order) I went back to Frank's list and did some additional testing. I have a different server which was set up the same way as the previous one discussed, and I thought I would use the above tools and see if my problem would have been identified by any of them. I am sorry to report, no, none of these either caught the problem either. Are you able to share the URL of the misconfigured site? It would be interesting to examine exactly what's going on. - Matt SSLCertificateChainFile /etc/ssl/certs/gd_bundle-g2-g1.crt I have actually fixed it. What was going on seems to be this - I have a new godaddy certificate for *.mydomain.com, and that is what I installed. However, the certificate chain I supplied was missing some intermediate godaddy certificate. Originally, it appeared I was missing 'gdig2.crt', and once installed, that fixed some clients including the ones behind the meraki router. But then there were also some older clients this did not fix (a macos 10.4 something for example). So I went back and installed gd_bundle-g2-g1.crt in it's place, and that seems to have finally done it. I apologize for the diminishing lack of operational content. It just seems that these ssl tests should be tightened up and perhaps some additional tools deployed out there to help us less knowledgeable folks 'get it right'. Mike-
Re: FIXED - Re: Broken SSL cert caused by router?
On Sat, Mar 28, 2015 at 09:05:38AM -0700, Mike wrote: > On 03/27/2015 10:34 AM, Frank Bulk wrote: > >Glad you figured that out. > > > >I've used three SSL evaluation websites to help me with intermediate > >certificate issues: > >https://www.ssllabs.com/ssltest/analyze.html (will show the names and > >details of the certs, missing or not > >https://www.wormly.com/test_ssl (quick SSL tester, will point out if > >intermediate certificate is missing) > >https://www.digicert.com/help/ (will show a green chain link between certs > >when they're all there *and* in order) > > I went back to Frank's list and did some additional testing. I have a > different server which was set up the same way as the previous one > discussed, and I thought I would use the above tools and see if my problem > would have been identified by any of them. I am sorry to report, no, none of > these either caught the problem either. Are you able to share the URL of the misconfigured site? It would be interesting to examine exactly what's going on. - Matt -- The main advantages of Haynes and Chilton manuals are that they cost $15, where the factory manuals cost $100 and up, and that they will tell you how to use two hammers, a block of wood, and a meerkat to replace "special tool no. 2-112-A"-- Matt Roberds in asr.
Re: FIXED - Re: Broken SSL cert caused by router?
On 3/28/15 9:05 AM, Mike wrote: I went back to Frank's list and did some additional testing. I have a different server which was set up the same way as the previous one discussed, and I thought I would use the above tools and see if my problem would have been identified by any of them. I am sorry to report, no, none of these either caught the problem either. Although I still do not fully understand the dependencies involved, it seems that if my server was failing to supply the full certificate chain, and the browser was compensating for it by (attempting?) to load the missing certificate from elsewhere, and this Meraki router was somehow able to confound that process, that would be an issue worthy of exploring more. I certainly don't blame these ssl check sites but clearly theres more checks needed. The Qualsys site (https://www.ssllabs.com/ssltest/analyze.html) will report whether or not the server supplied the intermediate cert. But I agree with you that the other tools should make a bigger deal about it if the server doesn't supply it. FWIW, it's been the CW to do this for some time now, as there are systems like the one you've run into that were designed before intermediate certs were commonplace, and don't know how to handle them. I've also experienced situations where an enterprise purchases a DV certificate to be used on an offline system, and while that system has access to the "root" CA certs, it cannot retrieve the intermediate cert. Having the end system supply the intermediate cert as well solves this issue. The method of supplying the intermediate cert is simple, just append the intermediate certificate to the end of the file with your server certificate (the .crt file). Any reasonably modern software will handle that transparently, and provide the intermediate cert along with the server cert when doing its business. hope this helps, Doug -- I am conducting an experiment in the efficacy of PGP/MIME signatures. This message should be signed. If it is not, or the signature does not validate, please let me know how you received this message (direct, or to a list) and the mail software you use. Thanks! signature.asc Description: OpenPGP digital signature
Re: FIXED - Re: Broken SSL cert caused by router?
On 03/27/2015 10:34 AM, Frank Bulk wrote: Glad you figured that out. I've used three SSL evaluation websites to help me with intermediate certificate issues: https://www.ssllabs.com/ssltest/analyze.html (will show the names and details of the certs, missing or not https://www.wormly.com/test_ssl (quick SSL tester, will point out if intermediate certificate is missing) https://www.digicert.com/help/ (will show a green chain link between certs when they're all there *and* in order) Frank I went back to Frank's list and did some additional testing. I have a different server which was set up the same way as the previous one discussed, and I thought I would use the above tools and see if my problem would have been identified by any of them. I am sorry to report, no, none of these either caught the problem either. Although I still do not fully understand the dependencies involved, it seems that if my server was failing to supply the full certificate chain, and the browser was compensating for it by (attempting?) to load the missing certificate from elsewhere, and this Meraki router was somehow able to confound that process, that would be an issue worthy of exploring more. I certainly don't blame these ssl check sites but clearly theres more checks needed. Mike-
Re: Broken SSL cert caused by router?
It might be filtering the CRL or OCSP verification for the SSL certificate. For GoDaddy I think this would be: http://crl.godaddy.com/ http://ocsp.godaddy.com/ http://certificates.godaddy.com/ We ran into this when OS X changed how it handles SSL a few years back, our captive portal was presenting a splash page in place of Thawte OCSP and crashing the SSL keychain process. The work-around was either to respond with a TCP RST for these requests or to allow them through. On Thu, Mar 26, 2015 at 11:57 PM, Lewis,Mitchell T. wrote: > Meraki Access Points are interesting devices. > > I have found they cause issues with Linux firewalls if the merakis are not > configured "correctly". > > Meraki Access Points do content inspections which I have found can cause > produce symptoms similar to yours, although I have not experienced what you > are describing. Since the MX64W is both an Access Point & security gateway, > it has some additional content inspection/intelligence for it's security > appliance role on top of the functions it performs as an access point, the > same functions which are found in Meraki standalone access points as well. > > I am not sure what the specifics are as I do not use Meraki security > appliances but it is worth checking. I have found with Meraki that items in > the control panel/dashboard are not always labeled the best so I have found > it is usually worth putting in a ticket with them and/or a call to them to > see what they think (1-888-490-0918). > > > > > > > > > > > > Mitchell T. Lewis > mle...@techcompute.net > : www.linkedin.com/in/mlewiscc > Mobile: (203)816-0371 > PGP Fingerprint: 79F2A12BAC77827581C734212AFA805732A1394E Public PGP Key > > > > > A computer will do what you tell it to do, but that may be much different > from what you had in mind. ~Joseph Weizenbaum > > - Original Message - > > From: "Mike" > To: nanog@nanog.org > Sent: Thursday, March 26, 2015 6:38:55 PM > Subject: Broken SSL cert caused by router? > > Hi, > > I have a very odd problem. > > We've recently gotten a 'real' ssl certificate from godaddy to > cover our domain (*.domain.com) and have installed it in several places > where needed for email (imap/starttls and etc) and web. This works > great, seems ok according to various online TLS certificate checkers, > and I get the green lock when testing using my own browsers and such. > > I have a customer however that uses our web mail system now secured > with ssl. I myself and many others use it and get the green lock. But, > whenever any station at the customer tries using it, they get a broken > lock and 'your connection is not private'. The actual error displayed > below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate > Authority - G2". And it gets worse - whenever I go to the location and > use my own laptop, the very one that 'works' when at my office, I ALSO > get the error. AND EVEN WORSE - when I connect to my cell phone provided > hotspot, the error goes away! > > As weird as this all sounds, I got it nailed down to one device - > they have a Cisco/Meraki MX64W as their internet gateway - and when I > remove that device from the chain and go 'straight' out to the internet, > suddenly, the certificate problem goes away entirely. > > How is this possible? Can anyone comment on these devices and tell > me what might be going on here? > > Mike- > -- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
RE: FIXED - Re: Broken SSL cert caused by router?
Glad you figured that out. I've used three SSL evaluation websites to help me with intermediate certificate issues: https://www.ssllabs.com/ssltest/analyze.html (will show the names and details of the certs, missing or not https://www.wormly.com/test_ssl (quick SSL tester, will point out if intermediate certificate is missing) https://www.digicert.com/help/ (will show a green chain link between certs when they're all there *and* in order) Frank -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mike Sent: Friday, March 27, 2015 10:36 AM Cc: nanog@nanog.org Subject: FIXED - Re: Broken SSL cert caused by router? I'd like to thank everyone for their kind responses. One person who responded off list and bothered to look at the returned certificates pointed out, and correctly it seems, that my original setup was missing an intermediate certificate. The site was returning 'valid ssl' and all browsers got the green lock and offsite ssl tests came back ok, but apparently the missing intermediate means it would have had to have been fetched and that was the part that was failing at the customer site. Once I put the intermediate certificate in there, the customer site was able to access https without fail. I have not had an opportunity yet to examine in detail the config of the meraki router there but it's either a routing problem or a DPI problem. If I get an answer I'll post again with my results. Thanks all. Mike-
Re: FIXED - Re: Broken SSL cert caused by router?
When I had the same mistake as you, that toll identified it. That's why I mentioned that one :) Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mar 27, 2015 12:34 PM, "Mike" wrote: > > > On 03/27/2015 08:43 AM, Josh Luthman wrote: > >> FFR you can use this to verify the site itself is good or not: >> >> https://www.sslshopper.com/ssl-checker.html (there are others, this is >> just what I have bookmarked) >> >> > Thanks. Previously while diagnosing this however, I used some others > similar and they all were saying I was ok. For example, > https://www.ssllabs.com/ssltest/analyze.html and one other I forget now. > I am surprised this problem was not being pointed out. > > > >
Re: FIXED - Re: Broken SSL cert caused by router?
I believe the SSLLabs Analyzer should have pointed out an "Extra Download" in the cert chain. That was the hint that there was an intermediate cert that a client would have to go find on it's own because it wasn't included with your server cert. https://community.qualys.com/thread/12831 On 3/27/2015 12:34 PM, Mike wrote: On 03/27/2015 08:43 AM, Josh Luthman wrote: FFR you can use this to verify the site itself is good or not: https://www.sslshopper.com/ssl-checker.html (there are others, this is just what I have bookmarked) Thanks. Previously while diagnosing this however, I used some others similar and they all were saying I was ok. For example, https://www.ssllabs.com/ssltest/analyze.html and one other I forget now. I am surprised this problem was not being pointed out.
Re: FIXED - Re: Broken SSL cert caused by router?
On 03/27/2015 08:43 AM, Josh Luthman wrote: FFR you can use this to verify the site itself is good or not: https://www.sslshopper.com/ssl-checker.html (there are others, this is just what I have bookmarked) Thanks. Previously while diagnosing this however, I used some others similar and they all were saying I was ok. For example, https://www.ssllabs.com/ssltest/analyze.html and one other I forget now. I am surprised this problem was not being pointed out.
Re: FIXED - Re: Broken SSL cert caused by router?
FFR you can use this to verify the site itself is good or not: https://www.sslshopper.com/ssl-checker.html (there are others, this is just what I have bookmarked) Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Mar 27, 2015 at 11:35 AM, Mike wrote: > > I'd like to thank everyone for their kind responses. One person who > responded off list and bothered to look at the returned certificates > pointed out, and correctly it seems, that my original setup was missing an > intermediate certificate. The site was returning 'valid ssl' and all > browsers got the green lock and offsite ssl tests came back ok, but > apparently the missing intermediate means it would have had to have been > fetched and that was the part that was failing at the customer site. Once I > put the intermediate certificate in there, the customer site was able to > access https without fail. I have not had an opportunity yet to examine in > detail the config of the meraki router there but it's either a routing > problem or a DPI problem. If I get an answer I'll post again with my > results. > > Thanks all. > > Mike- > >
FIXED - Re: Broken SSL cert caused by router?
I'd like to thank everyone for their kind responses. One person who responded off list and bothered to look at the returned certificates pointed out, and correctly it seems, that my original setup was missing an intermediate certificate. The site was returning 'valid ssl' and all browsers got the green lock and offsite ssl tests came back ok, but apparently the missing intermediate means it would have had to have been fetched and that was the part that was failing at the customer site. Once I put the intermediate certificate in there, the customer site was able to access https without fail. I have not had an opportunity yet to examine in detail the config of the meraki router there but it's either a routing problem or a DPI problem. If I get an answer I'll post again with my results. Thanks all. Mike-
Re: Broken SSL cert caused by router?
Meraki Access Points are interesting devices. I have found they cause issues with Linux firewalls if the merakis are not configured "correctly". Meraki Access Points do content inspections which I have found can cause produce symptoms similar to yours, although I have not experienced what you are describing. Since the MX64W is both an Access Point & security gateway, it has some additional content inspection/intelligence for it's security appliance role on top of the functions it performs as an access point, the same functions which are found in Meraki standalone access points as well. I am not sure what the specifics are as I do not use Meraki security appliances but it is worth checking. I have found with Meraki that items in the control panel/dashboard are not always labeled the best so I have found it is usually worth putting in a ticket with them and/or a call to them to see what they think (1-888-490-0918). Mitchell T. Lewis mle...@techcompute.net : www.linkedin.com/in/mlewiscc Mobile: (203)816-0371 PGP Fingerprint: 79F2A12BAC77827581C734212AFA805732A1394E Public PGP Key A computer will do what you tell it to do, but that may be much different from what you had in mind. ~Joseph Weizenbaum - Original Message - From: "Mike" To: nanog@nanog.org Sent: Thursday, March 26, 2015 6:38:55 PM Subject: Broken SSL cert caused by router? Hi, I have a very odd problem. We've recently gotten a 'real' ssl certificate from godaddy to cover our domain (*.domain.com) and have installed it in several places where needed for email (imap/starttls and etc) and web. This works great, seems ok according to various online TLS certificate checkers, and I get the green lock when testing using my own browsers and such. I have a customer however that uses our web mail system now secured with ssl. I myself and many others use it and get the green lock. But, whenever any station at the customer tries using it, they get a broken lock and 'your connection is not private'. The actual error displayed below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate Authority - G2". And it gets worse - whenever I go to the location and use my own laptop, the very one that 'works' when at my office, I ALSO get the error. AND EVEN WORSE - when I connect to my cell phone provided hotspot, the error goes away! As weird as this all sounds, I got it nailed down to one device - they have a Cisco/Meraki MX64W as their internet gateway - and when I remove that device from the chain and go 'straight' out to the internet, suddenly, the certificate problem goes away entirely. How is this possible? Can anyone comment on these devices and tell me what might be going on here? Mike-
Re: Broken SSL cert caused by router?
Thu, Mar 26, 2015 at 03:38:55PM -0700, Mike wrote: > I have a customer however that uses our web mail system now secured > with ssl. I myself and many others use it and get the green lock. But, > whenever any station at the customer tries using it, they get a broken > lock and 'your connection is not private'. The actual error displayed > below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate > Authority - G2". And it gets worse - whenever I go to the location and > use my own laptop, the very one that 'works' when at my office, I ALSO > get the error. AND EVEN WORSE - when I connect to my cell phone provided > hotspot, the error goes away! > > As weird as this all sounds, I got it nailed down to one device - > they have a Cisco/Meraki MX64W as their internet gateway - and when I > remove that device from the chain and go 'straight' out to the internet, > suddenly, the certificate problem goes away entirely. > > How is this possible? Can anyone comment on these devices and tell > me what might be going on here? Sounds like deep packet inspection (DPI) with SSL MITM. Reading https://meraki.cisco.com/lib/pdf/meraki_datasheet_mx.pdf makes me believe that this device can do that. Look for it's configuration, DPI for HTTPS must be active. -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
Re: Broken SSL cert caused by router?
You might want to look at some of the documentation on that device. Looks like it might be doing some proxy stuff. Regards, -Joe On Thu, Mar 26, 2015 at 5:38 PM, Mike wrote: > Hi, > > I have a very odd problem. > > We've recently gotten a 'real' ssl certificate from godaddy to cover our > domain (*.domain.com) and have installed it in several places where needed > for email (imap/starttls and etc) and web. This works great, seems ok > according to various online TLS certificate checkers, and I get the green > lock when testing using my own browsers and such. > > I have a customer however that uses our web mail system now secured with > ssl. I myself and many others use it and get the green lock. But, whenever > any station at the customer tries using it, they get a broken lock and 'your > connection is not private'. The actual error displayed below is > 'cert_authority_invalid' and it's "Go Daddy Secure Certificate Authority - > G2". And it gets worse - whenever I go to the location and use my own > laptop, the very one that 'works' when at my office, I ALSO get the error. > AND EVEN WORSE - when I connect to my cell phone provided hotspot, the error > goes away! > > As weird as this all sounds, I got it nailed down to one device - they > have a Cisco/Meraki MX64W as their internet gateway - and when I remove that > device from the chain and go 'straight' out to the internet, suddenly, the > certificate problem goes away entirely. > > How is this possible? Can anyone comment on these devices and tell me > what might be going on here? > > Mike-
Re: Broken SSL cert caused by router?
On 27 Mar 2015, at 5:38, Mike wrote: How is this possible? Can anyone comment on these devices and tell me what might be going on here? It's been compromised and its being used for MITM? Or has some sort of TLS inspection capability built in which is essentially MITM, and which is enabled? Or some kind of content filtering capability which amounts to the same thing? --- Roland Dobbins
Broken SSL cert caused by router?
Hi, I have a very odd problem. We've recently gotten a 'real' ssl certificate from godaddy to cover our domain (*.domain.com) and have installed it in several places where needed for email (imap/starttls and etc) and web. This works great, seems ok according to various online TLS certificate checkers, and I get the green lock when testing using my own browsers and such. I have a customer however that uses our web mail system now secured with ssl. I myself and many others use it and get the green lock. But, whenever any station at the customer tries using it, they get a broken lock and 'your connection is not private'. The actual error displayed below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate Authority - G2". And it gets worse - whenever I go to the location and use my own laptop, the very one that 'works' when at my office, I ALSO get the error. AND EVEN WORSE - when I connect to my cell phone provided hotspot, the error goes away! As weird as this all sounds, I got it nailed down to one device - they have a Cisco/Meraki MX64W as their internet gateway - and when I remove that device from the chain and go 'straight' out to the internet, suddenly, the certificate problem goes away entirely. How is this possible? Can anyone comment on these devices and tell me what might be going on here? Mike-
