Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
Matt, Sorry for the confusion. In the first (correct) trace the client is BoringSSL (webrtc) and the server hello comes from Janus (OpenSSL). In the bad trace the Client is openSSL and the server is BoringSSL. The error almost always seems to happen in this configuration (when client == openSSL, Server == BoringSSL) So im guessing the client hello is different since it comes from boring SSL. I'm currently testing janus with both 1.0.1m and BoringSSL (same version as webrtc/chrome M43). I'll post my results if it doesnt end up in the same state in a few days as I get some testing on these servers! This has been amazing helpful. OpenSSL seems DTLS handshakes seem so easy :) I cant believe theres a SSL_do_handshake() function! Thank you! On Fri, May 1, 2015 at 3:41 PM Matt Caswell wrote: > > > On 01/05/15 02:11, faraz khan wrote: > > Hi everyone, > > This is my first time posting to this list - so if theres a better place > > for this question please let me know. > > > > The problem I'm trying to fix applies to the Janus webrtc gateway > > (https://github.com/meetecho/janus-gateway) and my application which is > > using native C++ webrtc. > > > > What happens is that after hundreds of successful connections, sometimes > > the Janus server is unable to negotiate a DTLS handshake and after a key > > exchange the webrtc client replied with a DTLS Alert: Decrypt failed > > message. I'm attaching a wireshark trace of the issue happening and one > > for the correct negotiation. > > Can you confirm that the trace you attached for the correct negotiation > also came from Janus? I spotted some odd things about the extensions in > the ClientHello: > > It has the "extended master secret" extension. That extension is not > supported by any released version of OpenSSL (we have it in the dev > version of 1.1.0 - but that is unreleased). > > It correctly adds the ec_point_formats and elliptic_curves extensions. > However it so happens that OpenSSL 1.0.1f has a bug in it (not in the > latest version) which suppresses those extensions for DTLS (this is a > significant enough bug in itself that it is worthwhile upgrading your > OpenSSL) > > OpenSSL always adds extensions in the same order - and these extensions > are in a different order. > > Due to the above it looks to me like the OpenSSL DTLS stack was not used > to generate that ClientHello. The failed one *does* look like an OpenSSL > generated ClientHello. > > Matt > > ___ > openssl-users mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users > ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
On 01/05/15 02:11, faraz khan wrote: > Hi everyone, > This is my first time posting to this list - so if theres a better place > for this question please let me know. > > The problem I'm trying to fix applies to the Janus webrtc gateway > (https://github.com/meetecho/janus-gateway) and my application which is > using native C++ webrtc. > > What happens is that after hundreds of successful connections, sometimes > the Janus server is unable to negotiate a DTLS handshake and after a key > exchange the webrtc client replied with a DTLS Alert: Decrypt failed > message. I'm attaching a wireshark trace of the issue happening and one > for the correct negotiation. Can you confirm that the trace you attached for the correct negotiation also came from Janus? I spotted some odd things about the extensions in the ClientHello: It has the "extended master secret" extension. That extension is not supported by any released version of OpenSSL (we have it in the dev version of 1.1.0 - but that is unreleased). It correctly adds the ec_point_formats and elliptic_curves extensions. However it so happens that OpenSSL 1.0.1f has a bug in it (not in the latest version) which suppresses those extensions for DTLS (this is a significant enough bug in itself that it is worthwhile upgrading your OpenSSL) OpenSSL always adds extensions in the same order - and these extensions are in a different order. Due to the above it looks to me like the OpenSSL DTLS stack was not used to generate that ClientHello. The failed one *does* look like an OpenSSL generated ClientHello. Matt ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
On Fri, May 01, 2015 at 09:01:47PM +0100, Matt Caswell wrote: > > > On 01/05/15 20:09, faraz khan wrote: > > Matt, > > Thanks again! To be precise webrtc is using boringssl (Google's fork of > > openssl). From the commits it seems VERY recent but I'm unable to figure > > out the last openssl merge-in. You think this could be the issue? Should > > I try both with boringSSL (its kinda hard to compile webrtc with openSSL > > now since after the move to boringSSL) > > Ahhh. BoringSSL. That means a slightly different list of potential > causes (they forked OpenSSL 1.0.2, and have continued to make changes > since then). I've just checked the latest dev version, and that means > possible causes: > > - digest check failed processing the Finished message > - CRL signature failure processing the client Certificate > - Overly short RSA key > - Bad RSA decrypt/signature processing the CertificateVerify > > To be honest I am more inclined to believe it is a problem on the client > side rather than the WebRTC side. The fact that it goes away after you > restart the client suggests to me that the client is getting itself into > a strange state. Perhaps some kind of memory corruption (either through > client application or openssl bug)? > > Anyway I'd suggest two things as next steps: > - Try and figure out how to get more logging out of WebRTC to find out > more specifically what it is complaining about > - Upgrade your OpenSSL version to 1.0.1m He could also try to use something like valgrind to see if memory gets corrupted, or use build it using address sanitizer. Kurt ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
On 01/05/15 20:09, faraz khan wrote: > Matt, > Thanks again! To be precise webrtc is using boringssl (Google's fork of > openssl). From the commits it seems VERY recent but I'm unable to figure > out the last openssl merge-in. You think this could be the issue? Should > I try both with boringSSL (its kinda hard to compile webrtc with openSSL > now since after the move to boringSSL) Ahhh. BoringSSL. That means a slightly different list of potential causes (they forked OpenSSL 1.0.2, and have continued to make changes since then). I've just checked the latest dev version, and that means possible causes: - digest check failed processing the Finished message - CRL signature failure processing the client Certificate - Overly short RSA key - Bad RSA decrypt/signature processing the CertificateVerify To be honest I am more inclined to believe it is a problem on the client side rather than the WebRTC side. The fact that it goes away after you restart the client suggests to me that the client is getting itself into a strange state. Perhaps some kind of memory corruption (either through client application or openssl bug)? Anyway I'd suggest two things as next steps: - Try and figure out how to get more logging out of WebRTC to find out more specifically what it is complaining about - Upgrade your OpenSSL version to 1.0.1m Matt ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
Matt, Thanks again! To be precise webrtc is using boringssl (Google's fork of openssl). From the commits it seems VERY recent but I'm unable to figure out the last openssl merge-in. You think this could be the issue? Should I try both with boringSSL (its kinda hard to compile webrtc with openSSL now since after the move to boringSSL) On Fri, May 1, 2015 at 9:33 AM Matt Caswell wrote: > > > On 01/05/15 16:51, faraz khan wrote: > > Matt, > > Thanks for taking the time to go through this. Unfortunately this error > > only happens when WebRTC is acting as the server in the dtls handshake > > and no logs (at all) seem to be printed but I think that's because > > openssl logging is done in a different way in native WebRTC which I'm > > not handling. I'll try increasing logs and / or printing openssl errors. > > > > The janus server gets the alert packet which I'm guessing is an error > > produced by the server in the transaction above. > > Yes. You can see the following exchanges in the traces: > > ClientServer > ---- > > ClientHello > > ServerHello > Certificate > CertificateRequest > < ServerHelloDone > Certificate > ClientKeyExchange > CertificateVerify > ChangeCipherSpec > [Finished (Encrypted)] --> > < Alert (Decrypt Error) > > If I understand your setup correctly (I'm not familiar with Janus or > WebRTC), Janus is acting as the client here and WebRTC is the server, > and both are using OpenSSL? > > > > > I'm assuming the problem is with the client reply that janus sent. Any > > idea what it could have sent wrong that WebRTC didn't like? Could it > > have encrypted the message with a wrong crypto or something or is this > > negotiation wholly handled by openssl? I'm trying to understand if janus > > is doing something wrong or is the dtls stuff completely handled by > > libssl - in which case this might be a dtls bug? > > The server doesn't like something the Client sent in the last flight of > messages, i.e. the Certificate, ClientKeyExchange, CertificateVerify, > ChangeCipherSpec or Finished. It's unlikely to be the Certificate as, as > far as I can tell, its the same Certificate in the "good" exchange > versus the bad. So that means either the ClientKeyExchange, > CertificateVerify or Finished messages (CCS is super simple so I don't > think its that). > > Is WebRTC using 1.0.1 too (and if so which letter version)? Looking at > the 1.0.1 code there are only a few places where a decrypt error can > come from. Possible causes (in 1.0.1 code base): > > - digest check failed processing the Finished message > - CRL signature failure processing the client Certificate > - Bad RSA decrypt processing the CertificateVerify > - Bad RSA signature processing the CertificateVerify > > There are a few other places that emit Decrypt Error alerts but none of > them seem to apply here. Getting some additional logging might help us > figure out which one of these is the cause. > > These are all quite low level issues though...deep in OpenSSL internals. > > Matt > > > > ___ > openssl-users mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users > ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
On 01/05/15 16:51, faraz khan wrote: > Matt, > Thanks for taking the time to go through this. Unfortunately this error > only happens when WebRTC is acting as the server in the dtls handshake > and no logs (at all) seem to be printed but I think that's because > openssl logging is done in a different way in native WebRTC which I'm > not handling. I'll try increasing logs and / or printing openssl errors. > > The janus server gets the alert packet which I'm guessing is an error > produced by the server in the transaction above. Yes. You can see the following exchanges in the traces: ClientServer ---- ClientHello > ServerHello Certificate CertificateRequest < ServerHelloDone Certificate ClientKeyExchange CertificateVerify ChangeCipherSpec [Finished (Encrypted)] --> < Alert (Decrypt Error) If I understand your setup correctly (I'm not familiar with Janus or WebRTC), Janus is acting as the client here and WebRTC is the server, and both are using OpenSSL? > > I'm assuming the problem is with the client reply that janus sent. Any > idea what it could have sent wrong that WebRTC didn't like? Could it > have encrypted the message with a wrong crypto or something or is this > negotiation wholly handled by openssl? I'm trying to understand if janus > is doing something wrong or is the dtls stuff completely handled by > libssl - in which case this might be a dtls bug? The server doesn't like something the Client sent in the last flight of messages, i.e. the Certificate, ClientKeyExchange, CertificateVerify, ChangeCipherSpec or Finished. It's unlikely to be the Certificate as, as far as I can tell, its the same Certificate in the "good" exchange versus the bad. So that means either the ClientKeyExchange, CertificateVerify or Finished messages (CCS is super simple so I don't think its that). Is WebRTC using 1.0.1 too (and if so which letter version)? Looking at the 1.0.1 code there are only a few places where a decrypt error can come from. Possible causes (in 1.0.1 code base): - digest check failed processing the Finished message - CRL signature failure processing the client Certificate - Bad RSA decrypt processing the CertificateVerify - Bad RSA signature processing the CertificateVerify There are a few other places that emit Decrypt Error alerts but none of them seem to apply here. Getting some additional logging might help us figure out which one of these is the cause. These are all quite low level issues though...deep in OpenSSL internals. Matt ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
Matt, Thanks for taking the time to go through this. Unfortunately this error only happens when WebRTC is acting as the server in the dtls handshake and no logs (at all) seem to be printed but I think that's because openssl logging is done in a different way in native WebRTC which I'm not handling. I'll try increasing logs and / or printing openssl errors. The janus server gets the alert packet which I'm guessing is an error produced by the server in the transaction above. I'm assuming the problem is with the client reply that janus sent. Any idea what it could have sent wrong that WebRTC didn't like? Could it have encrypted the message with a wrong crypto or something or is this negotiation wholly handled by openssl? I'm trying to understand if janus is doing something wrong or is the dtls stuff completely handled by libssl - in which case this might be a dtls bug? I'm a newbie to openssl and dtls so dont mind the stupid questions. I however do know C and socket stuff so can get in to check with some guidance :) On Fri, May 1, 2015, 2:37 AM Matt Caswell wrote: > > > On 01/05/15 02:11, faraz khan wrote: > > Hi everyone, > > This is my first time posting to this list - so if theres a better place > > for this question please let me know. > > > > The problem I'm trying to fix applies to the Janus webrtc gateway > > (https://github.com/meetecho/janus-gateway) and my application which is > > using native C++ webrtc. > > > > What happens is that after hundreds of successful connections, sometimes > > the Janus server is unable to negotiate a DTLS handshake and after a key > > exchange the webrtc client replied with a DTLS Alert: Decrypt failed > > message. I'm attaching a wireshark trace of the issue happening and one > > for the correct negotiation. > > > > The problem refuses to fix itself till Janus is restarted. > > > > Both installations are using Openssl. Janus is compiled with version > 1.0.1f > > > > If someone can help explain how DTLS key exchange works and whats going > > wrong in the above trace it would be great! I'm completely at a loss as > > far as this is concerned! > > > > Thanks all! > > H. I can't see anything obviously wrong with the above traces. The > handshake seems to proceed as normal and then fail near the end. > > A couple of things of note: > * A client cert is being sent, but it has expired. I don't think this is > the problem though because it is the same cert in the "good" trace and > the "bad" trace. > Validity > Not Before: Feb 9 16:18:45 2007 GMT > Not After : Feb 8 16:18:45 2009 GMT > > * A different ciphersuite is being negotiated between the "good" version > and the "bad" version. "Good" is TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, > whilst "Bad" is TLS_RSA_WITH_AES_256_CBC_SHA. I'm not sure if that is > significant, but I can't see why a server restart would make any > difference if it were. > > Are there any server logs which might indicate why it is sending the > alert? Looking at the code there are only a few places in the code which > generate a decrypt error alert. It would probably help diagnose the > problem if we could narrow down which of those places this is coming > from. OpenSSL adds an error to its error queue for each of those places. > > The other point of note is that there have been quite a lot of DTLS > related defect fixes in the OpenSSL code since 1.0.1f. An upgrade would > be a really good idea. > > Matt > > ___ > openssl-users mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users > ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Trying to understand DTLS (as it applies to webrtc)
On 01/05/15 02:11, faraz khan wrote: > Hi everyone, > This is my first time posting to this list - so if theres a better place > for this question please let me know. > > The problem I'm trying to fix applies to the Janus webrtc gateway > (https://github.com/meetecho/janus-gateway) and my application which is > using native C++ webrtc. > > What happens is that after hundreds of successful connections, sometimes > the Janus server is unable to negotiate a DTLS handshake and after a key > exchange the webrtc client replied with a DTLS Alert: Decrypt failed > message. I'm attaching a wireshark trace of the issue happening and one > for the correct negotiation. > > The problem refuses to fix itself till Janus is restarted. > > Both installations are using Openssl. Janus is compiled with version 1.0.1f > > If someone can help explain how DTLS key exchange works and whats going > wrong in the above trace it would be great! I'm completely at a loss as > far as this is concerned! > > Thanks all! H. I can't see anything obviously wrong with the above traces. The handshake seems to proceed as normal and then fail near the end. A couple of things of note: * A client cert is being sent, but it has expired. I don't think this is the problem though because it is the same cert in the "good" trace and the "bad" trace. Validity Not Before: Feb 9 16:18:45 2007 GMT Not After : Feb 8 16:18:45 2009 GMT * A different ciphersuite is being negotiated between the "good" version and the "bad" version. "Good" is TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, whilst "Bad" is TLS_RSA_WITH_AES_256_CBC_SHA. I'm not sure if that is significant, but I can't see why a server restart would make any difference if it were. Are there any server logs which might indicate why it is sending the alert? Looking at the code there are only a few places in the code which generate a decrypt error alert. It would probably help diagnose the problem if we could narrow down which of those places this is coming from. OpenSSL adds an error to its error queue for each of those places. The other point of note is that there have been quite a lot of DTLS related defect fixes in the OpenSSL code since 1.0.1f. An upgrade would be a really good idea. Matt ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] Trying to understand DTLS (as it applies to webrtc)
Hi everyone, This is my first time posting to this list - so if theres a better place for this question please let me know. The problem I'm trying to fix applies to the Janus webrtc gateway ( https://github.com/meetecho/janus-gateway) and my application which is using native C++ webrtc. What happens is that after hundreds of successful connections, sometimes the Janus server is unable to negotiate a DTLS handshake and after a key exchange the webrtc client replied with a DTLS Alert: Decrypt failed message. I'm attaching a wireshark trace of the issue happening and one for the correct negotiation. The problem refuses to fix itself till Janus is restarted. Both installations are using Openssl. Janus is compiled with version 1.0.1f If someone can help explain how DTLS key exchange works and whats going wrong in the above trace it would be great! I'm completely at a loss as far as this is concerned! Thanks all! WebrtcDTLSNegotation.pcapng Description: Binary data janusbaddtlsnegotiation.pcapng Description: Binary data ___ openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
