[openssl.org #2462] [PATCH] DTLS Retransmission Buffer Bug

[Robin Seggelmann via RT]

DTLS clears the handshake retransmission buffer every time a message has been 
received, assuming no retransmissions are necessary anymore. This fails when 
the last flight of the server was lost twice. The first time DTLS bundles the 
SessionTicket, ChangeCipherSpec and Finished messages into one packet. If this 
gets lost, the client repeats its last flight. The server will then retransmit 
the mentioned messages each in a single packet. If any of them is lost again, 
the flight is not complete but the client clears its retransmission buffer as 
soon as the first message of the flight arrives. The server assumes that the 
handshake has been completed but the client still awaits a retransmission of 
the last flight. The client retransmits its empty buffer, that is nothing, and 
will give up when it reached its max number of timeouts. This can be solved 
easily by clearing the buffer when the retransmission timer is stopped. That 
happens when nothing has to be retransmitted anymore whic
 h is exactly when the buffer is supposed to be cleared.

Thanks to Robert Story for providing hints!

-Robin

--- ssl/d1_both.c       3 May 2010 13:01:50 -0000       1.14.2.21
+++ ssl/d1_both.c       28 Feb 2011 15:12:07 -0000
@@ -465,19 +465,6 @@
        memset(msg_hdr, 0x00, sizeof(struct hm_header_st));
 
        s->d1->handshake_read_seq++;
-       /* we just read a handshake message from the other side:
-        * this means that we don't need to retransmit of the
-        * buffered messages.  
-        * XDTLS: may be able clear out this
-        * buffer a little sooner (i.e if an out-of-order
-        * handshake message/record is received at the record
-        * layer.  
-        * XDTLS: exception is that the server needs to
-        * know that change cipher spec and finished messages
-        * have been received by the client before clearing this
-        * buffer.  this can simply be done by waiting for the
-        * first data  segment, but is there a better way?  */
-       dtls1_clear_record_buffer(s);
 
        s->init_msg = s->init_buf->data + DTLS1_HM_HEADER_LENGTH;
        return s->init_num;

--- ssl/d1_lib.c        6 Apr 2010 12:29:21 -0000       1.8.2.13
+++ ssl/d1_lib.c        28 Feb 2011 15:12:07 -0000
@@ -330,6 +330,8 @@
        memset(&(s->d1->next_timeout), 0, sizeof(struct timeval));
        s->d1->timeout_duration = 1;
        BIO_ctrl(SSL_get_rbio(s), BIO_CTRL_DGRAM_SET_NEXT_TIMEOUT, 0, 
&(s->d1->next_timeout));
+       /* Clear retransmission buffer */
+       dtls1_clear_record_buffer(s);
        }
 
 int dtls1_handle_timeout(SSL *s)

dtls-rtx-buffer-bug-0.9.8.patch
Description: Binary data

dtls-rtx-buffer-bug-1.0.0.patch
Description: Binary data

dtls-rtx-buffer-bug-1.0.1.patch
Description: Binary data