Hi again to both of you. Just doing a quick check, as I've heard from another user that was reporting this that apparently Sky made some changes and it now works for them.
Could you confirm if it is fixed on your end as well? Thanks :) On Wednesday, September 2, 2015 at 4:56:21 PM UTC-4, Patrice (Cloud Platform Support) wrote: > > Hi to you two, > > To answer Jon's question : yeah a Wireshark capture should give us enough. > That and the "chrome://net-internals" page would be the best things you can > send up so we continue investigating on this. > > In the meantime, the use of VIP is definitely a workaround this issue, and > as you point out, if other ISPs start having the same behavior, the VIP > will make sure this is stable throughout changes on those fronts. > > Thank you in advance for the information (again, use the "reply privately" > button). > > Cheers! > > On Wednesday, September 2, 2015 at 7:40:01 AM UTC-4, Hugo Visser wrote: >> >> OK, got another one on Sky. I'll ask for the info and will try to reach >> out to the other ones that have contacted me. >> >> Hugo >> >> On Wednesday, September 2, 2015 at 12:16:28 AM UTC+2, Jon Travers wrote: >>> >>> Sorry, in point 2 of my explanation above I should have said the >>> "ClientHello" message, not "ServerHello". >>> >>> On Tuesday, 1 September 2015 23:06:51 UTC+1, Jon Travers wrote: >>>> >>>> Hi Patrice >>>> >>>> I can confirm that our office broadband, where we've consistently seen >>>> this problem, is indeed a Sky connection. Since I last posted, I've done >>>> some further investigation using Wireshark to capture and decode the TSL >>>> protocol traffic being sent by Safari, Chrome, and Curl. My conclusion was >>>> that the connection failure is caused by a very specific combination of >>>> factors: >>>> >>>> 1. There is some specific property of the TLS client setup that >>>> must be present. I'm unable to identify exactly what, but on my machine >>>> it >>>> fails with up-to-date versions of Safari and Chrome, and also with the >>>> built-in OpenSSL (0.9.8zg), but not with the version of Curl that I >>>> have, >>>> nor with an updated version of OpenSSL. I can see differences in the >>>> connection parameters between these clients, but it's unclear which one >>>> is >>>> the cause. >>>> 2. The Initial TLS "ServerHello" message is being modified in a >>>> specific way by some IP routing node between my computer, and the >>>> endpoint >>>> at ghs.googlehosted.com. Based on all the evidence, we believe this >>>> is definitely happening inside our ISP, Sky Broadband, but the same >>>> issue >>>> could also occur on other routes. Whether this is the result of a >>>> configuration error at Sky, or the result of something they're doing >>>> deliberately, and how widespread this problem is, is all unclear. >>>> 3. The server at the destination of the SNI TLS connection, >>>> ghs.googlehosted.com, has been configured in such a way that the >>>> combination of 1 and 2 is regarded as a terminal error, and the >>>> connection >>>> is immediately terminated with an 'Illegal parameter' error. Connecting >>>> to >>>> another SNI SSL server (our own proxy) does not trigger the same error >>>> response, but since I can't see the traffic as it arrives at the >>>> Google, >>>> and I don't have access to any further debug information from the >>>> server, I >>>> can't determine whether the rejection is valid. Most likely it is, and >>>> our >>>> own proxy works simply because it has a less strict security setup. >>>> >>>> Now with these conclusions established, the only feasible solution I >>>> could see was to try running the connection via a dedicated virtual IP >>>> address (which costs $39 per month), rather than using the (free) SNI >>>> setup. I finally gave in, paid our $39, updated our DNS records, and it >>>> worked perfectly, no more connection errors. It's not exactly clear why >>>> this works, but I'm actually very happy with this as a solution (other >>>> than >>>> having to pay), especially since even if Sky fixes their routing error, I >>>> can still be confident that if the same problem crops up somewhere else on >>>> the Internet we won't be affected. >>>> >>>> *Bottom line: I would strongly advise anybody reading this to regard >>>> the Google Apps SNI SSL service as inherently unreliable. Pay the money >>>> and >>>> go with a virtual IP if at all possible, because the last thing you want >>>> is >>>> for some of your customers to be unable to access your app depending on >>>> where they are. I also think this advice should be clearly given in the >>>> Google documentation.* >>>> >>>> Now Patrice, if you still want to investigate the specific issue at >>>> Sky, then I'm very happy to provide you with additional debug information, >>>> once I get into the office tomorrow morning (it's 11pm here now and I'm at >>>> home). Can you be more specific about what you want? When you talk about >>>> tcpdump output, are you looking for a capture of the network packets? So >>>> my >>>> Wireshark capture would also be OK? >>>> >>>> Cheers >>>> Jon >>>> >>>> On Tuesday, 1 September 2015 20:03:50 UTC+1, Patrice (Cloud Platform >>>> Support) wrote: >>>>> >>>>> Hi Hugo, >>>>> >>>>> Exactly what I was expecting. Whoever reports these always seem to be >>>>> on Sky, which could be the problem. >>>>> >>>>> I don't know if you'll be able to get this since it's your users and >>>>> not you directly, but if you could get a tcpdump and a print screen of >>>>> the >>>>> page "chrome://net-internals*" *from your affected user, maybe we'll >>>>> be able to figure something out. >>>>> >>>>> Once you get these, don't send them publicly to the group. You can use >>>>> the arrow pointing down by the reply button and select "reply privately >>>>> to >>>>> author" on one of my messages. >>>>> >>>>> Cheers! >>>>> >>>>> On Tuesday, September 1, 2015 at 2:58:41 PM UTC-4, Hugo Visser wrote: >>>>>> >>>>>> The latest report came from a user on Sky. He also mentioned that >>>>>> sometimes it works, and sometimes it doesn't. I can't verify if that's >>>>>> true >>>>>> though. I've asked both users to open a browser to my custom domain app >>>>>> url. I've now resorted to updating the app and setting the endpoint to >>>>>> the >>>>>> appspot domain, which is not what I'd want ideally, but I don't want to >>>>>> lose any users over it either. >>>>>> >>>>>> The user that contacted me stated that it broke several weeks ago. >>>>>> >>>>>> Hugo >>>>>> >>>>>> On Tuesday, September 1, 2015 at 5:16:34 PM UTC+2, Patrice (Cloud >>>>>> Platform Support) wrote: >>>>>>> >>>>>>> Hi Hugo, >>>>>>> >>>>>>> Continuing to look into this, I realized that a lot of people who >>>>>>> have similar issues are all using a precise ISP. Do you know the ISP >>>>>>> that >>>>>>> your users have? >>>>>>> >>>>>>> Cheers! >>>>>>> >>>>>>> On Saturday, August 29, 2015 at 2:37:01 PM UTC-4, Hugo Visser wrote: >>>>>>>> >>>>>>>> Like I mentioned previously, today I got another user from the UK >>>>>>>> reporting a similar issue. For some reason the SNI SSL served version >>>>>>>> of my >>>>>>>> app isn't reachable or working for them, but going to >>>>>>>> https://myapp.appspot.com works. I don't see a huge drop in usage >>>>>>>> in the app so I can't really tell what is causing it, just that those >>>>>>>> users >>>>>>>> can't establish a SSL connection to my custom domain url from their >>>>>>>> devices. >>>>>>>> >>>>>>>> Hugo >>>>>>>> >>>>>>>> On Wednesday, August 26, 2015 at 9:09:52 PM UTC+2, Patrice (Cloud >>>>>>>> Platform Support) wrote: >>>>>>>>> >>>>>>>>> Hi again Jon, >>>>>>>>> >>>>>>>>> I continued trying to find what is exactly happening here. So here >>>>>>>>> goes : >>>>>>>>> >>>>>>>>> Running this in San Francisco : >>>>>>>>> >>>>>>>>> openssl s_client -debug -msg -servername dashboard.geospock.com >>>>>>>>> -connect dashboard.geospock.com:443 -showcerts >>>>>>>>> >>>>>>>>> works perfectly and consistently. So I think we can all see that >>>>>>>>> SF won't have issues connecting here. >>>>>>>>> >>>>>>>>> I then connected through an European location, and from there, >>>>>>>>> trying this also succeeds: >>>>>>>>> >>>>>>>>> openssl s_client -servername dashboard.geospock.com -connect >>>>>>>>> 64.233.167.121:443 -showcerts >>>>>>>>> curl -k -I --resolve dashboard.geospock.com:443:64.233.167.121 >>>>>>>>> https://dashboard.geospock.com/ >>>>>>>>> >>>>>>>>> From your error and your message, I get you're using TLS 1.0, >>>>>>>>> while I am running with TLS 1.2. I started looking into the version >>>>>>>>> of >>>>>>>>> openSSL we're both using, and while you're using the 0.98zg, I am on >>>>>>>>> 1.0.1f. >>>>>>>>> >>>>>>>>> I believe since 0.9.8j SNI support was part of OpenSSL, so you >>>>>>>>> using 0.98zg should work fine, but as a test, do you mind trying to >>>>>>>>> run the >>>>>>>>> same with 1.01 latest? I believe they are up to 1.0.1p. If I remember >>>>>>>>> correctly, there was a backport patch implemented in 0.9.8j to let >>>>>>>>> openSSL >>>>>>>>> work with SNI ( >>>>>>>>> https://en.wikipedia.org/wiki/Server_Name_Indication ). >>>>>>>>> >>>>>>>>> The basis of the issue we have here is that I'm incapable of >>>>>>>>> reproducing any of the behaviors you're experiencing, so it's hard to >>>>>>>>> investigate further. I'm definitely interested in continuing this, >>>>>>>>> but >>>>>>>>> without a reliable way for me to replicate, it's impossible to send >>>>>>>>> it up >>>>>>>>> the chain to get it looked at and possibly fixed. If you have a >>>>>>>>> specific >>>>>>>>> test case that consistently fail, that I can then reproduce on my >>>>>>>>> side as >>>>>>>>> consistently, I'll be able to get some traction on this. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On Wednesday, August 26, 2015 at 11:16:54 AM UTC-4, Jon Travers >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Here is some more detailed debugging information from the failing >>>>>>>>>> openssl SSL negotiation. Perhaps this would give you a clue what's >>>>>>>>>> actually >>>>>>>>>> going on?: >>>>>>>>>> >>>>>>>>>> $ openssl s_client -debug -msg -servername dashboard.geospock.com >>>>>>>>>> -connect dashboard.geospock.com:443 -showcerts >>>>>>>>>> CONNECTED(00000003) >>>>>>>>>> write to 0x7f96b8d006a0 [0x7f96b9802000] (131 bytes => 131 (0x83)) >>>>>>>>>> 0000 - 16 03 01 00 7e 01 00 00-7a 03 01 55 dd d7 93 c7 >>>>>>>>>> ....~...z..U.... >>>>>>>>>> 0010 - f0 b2 0d ee ea f4 1c 2b-ee 50 b6 ff 0f e6 8f 59 >>>>>>>>>> .......+.P.....Y >>>>>>>>>> 0020 - 8b 81 9e 05 2f 17 84 e2-20 ed b7 00 00 2e 00 39 ..../... >>>>>>>>>> ......9 >>>>>>>>>> 0030 - 00 38 00 35 00 16 00 13-00 0a 00 33 00 32 00 2f >>>>>>>>>> .8.5.......3.2./ >>>>>>>>>> 0040 - 00 9a 00 99 00 96 00 05-00 04 00 15 00 12 00 09 >>>>>>>>>> ................ >>>>>>>>>> 0050 - 00 14 00 11 00 08 00 06-00 03 00 ff 01 00 00 23 >>>>>>>>>> ...............# >>>>>>>>>> 0060 - 00 00 00 1b 00 19 00 00-16 64 61 73 68 62 6f 61 >>>>>>>>>> .........dashboa >>>>>>>>>> 0070 - 72 64 2e 67 65 6f 73 70-6f 63 6b 2e 63 6f 6d 00 >>>>>>>>>> rd.geospock.com. >>>>>>>>>> 0080 - 23 # >>>>>>>>>> 0083 - <SPACES/NULS> >>>>>>>>>> >>> TLS 1.0 Handshake [length 007e], ClientHello >>>>>>>>>> 01 00 00 7a 03 01 55 dd d7 93 c7 f0 b2 0d ee ea >>>>>>>>>> f4 1c 2b ee 50 b6 ff 0f e6 8f 59 8b 81 9e 05 2f >>>>>>>>>> 17 84 e2 20 ed b7 00 00 2e 00 39 00 38 00 35 00 >>>>>>>>>> 16 00 13 00 0a 00 33 00 32 00 2f 00 9a 00 99 00 >>>>>>>>>> 96 00 05 00 04 00 15 00 12 00 09 00 14 00 11 00 >>>>>>>>>> 08 00 06 00 03 00 ff 01 00 00 23 00 00 00 1b 00 >>>>>>>>>> 19 00 00 16 64 61 73 68 62 6f 61 72 64 2e 67 65 >>>>>>>>>> 6f 73 70 6f 63 6b 2e 63 6f 6d 00 23 00 00 >>>>>>>>>> read from 0x7f96b8d006a0 [0x7f96b9807600] (7 bytes => 7 (0x7)) >>>>>>>>>> 0000 - 15 03 01 00 02 02 2f ....../ >>>>>>>>>> <<< TLS 1.0 Alert [length 0002], fatal illegal_parameter >>>>>>>>>> 02 2f >>>>>>>>>> 8175:error:14077417:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 >>>>>>>>>> alert illegal >>>>>>>>>> parameter:/SourceCache/OpenSSL098/OpenSSL098-52.40.1/src/ssl/s23_clnt.c:593: >>>>>>>>>> >>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/43177cd0-bc27-4e55-9489-f19cbac4c7ad%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
