On Fri, 2008-09-26 at 13:46 -0700, David Woodhouse wrote: > At the worst, I should be able to reverse-engineer the library I have.
The first failure seems to have been a discrepancy in epoch numbers. Comparing behaviour of their library and 0.9.8e, I find that theirs is adding '00 01 00 00 00 00 00 00' to a digest at some point, while 0.9.8e adds '00 00 00 00 00 00 00 00'. This is called from tls1_mac(), when it's adding the 8 bytes of ssl->s3->read_sequence to the MAC. The 0.9.8e library then rejects the Server Hello because of the MAC failure, which was the original failure mode I was observing. If I hack EVP_DigestUpdate() to "fix" that single byte for that one call, then the MAC check in dtls1_process_record() succeeds, although fairly unsurprisingly I get a later failure -- in ssl3_get_finished() when s->s3->tmp.peer_finish_md doesn't contain what it should: 12778:error:1408C095:SSL routines:SSL3_GET_FINISHED:digest check failed:s3_both.c:235: I'm still entirely clueless about the protocol, but it seems the top 16 bits of ssl->s3->read_sequence are supposed to be an epoch number. But it's getting set to all zeroes in dtls1_reset_seq_numbers() even when the epoch is non-zero. Having narrowed it down that far, does anyone remember a change which might have caused this? I tried removing my hack from EVP_DigestUpdate and instead hacking dtls1_reset_seq_numbers() to call s2n(epoch, seq) to put the epoch in place after the memset. That makes no difference -- I still get the same later failure. Which I'll now investigate, but it's probably going to turn out to be due to the wrongness of my 'fix' for the epoch thing. As I said, I'm fairly clueless. I've converted the OpenSSL CVS history into git so that I can try to look through it, but I don't see anything which jumps out as being relevant. There's a commit entitled '"Liberate" dtls from BN dependency. Fix bug in replay/update.' which helpfully hides an unspecified bug fix in amongst 300-odd lines of more cosmetic changes in one commit, but that doesn't seem to be it. As before, my test case is http://david.woodhou.se/dtls-test.c -- and needs to be run against a version of OpenSSL which still uses 0x100 for DTLS1_VERSION. -- dwmw2 ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
