Hi Hans,

I will update txprob -> "tx probation"

The kEnabled, it means for a state of a tipc portid only. There's another @is_fctrl_enabled, that's for the feature whether mds has flow control enabled/disabled.



On 14/8/19 5:48 pm, Hans Nordebäck wrote:
Hi Minh,

ack, some minor comments below/Thanks Hans

On 2019-08-14 08:38, Minh Chau wrote:
   src/mds/README | 221 
   1 file changed, 221 insertions(+)
   create mode 100644 src/mds/README

diff --git a/src/mds/README b/src/mds/README
new file mode 100644
index 0000000..1b94632
--- /dev/null
+++ b/src/mds/README
@@ -0,0 +1,221 @@
+/*      -*- OpenSAF  -*-
+ *
+ * (C) Copyright 2019 The OpenSAF Foundation
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE. This file and program are licensed
+ * under the GNU Lesser General Public License Version 2.1, February 1999.
+ * The complete license can be accessed from the following location:
+ * http://opensource.org/licenses/lgpl-license.php
+ * See the Copying file included with the OpenSAF distribution for full
+ * licensing terms.
+ *
+ * Author(s): Ericsson AB
+ *
+ */
+If OpenSAF configures TIPC as transport, the MDS library today will use
+TIPC SOCK_RDM socket for message distribution in the cluster. The SOCK_RDM
+datagram socket possibly encounters buffer overflow at receiver ends which
+has been documented in tipc.io[1]. A temporary solution for this buffer
+overflow issue is that the socket buffer size can be increased to a larger
+number. However, if the cluster continues either scaling out or adding more
+components, the system will be under dimensioned, thus the TIPC buffer
+overflow can occur again.
+MDS's solution for TIPC buffer overflow
+If MDS disables TIPC_DEST_DROPPABLE, TIPC will return the ancillary message
+when the original message is failed to deliver. By this event, if the message
+has been saved in queue, MDS at sender sides can search and retransmit this
+message to the receivers.
+Once the messages in the sender's queue has been delivered successfully, MDS
+needs to remove them. MDS introduces its internal ACK message as an
+acknowledgment from receivers so that the senders can remove the messages
+out of the queue.
+Also, as such situation of buffer overflow at receivers, the retransmission may
+not succeed or even become worse at receiver ends (the more retransmission,
+the more overflow to occur). MDS imitates the sliding window in TCP[2] to
+control the flow of data message towards the receivers.
+Legacy MDS data message, new (data + ACK) MDS message, and upgradability
+Below is the MDS legacy message format that has been used till OpenSAF 5.19.07
+oct 0  message length
+oct 1
+oct 2  sequence number: incremented for every message sent out to all destined
+...       tipc portid.
+oct 5
+oct 6  fragment number: a message with same sequence number can be fragmented,
+oct 7  identified by this fragment number.
+oct 8  length check: cross check with message length(oct0,1), NOT USED.
+oct 9
+oct 10 protocol version: (MDS_PROT:0xA0 | MDS_VERSION:0x08) = 0xA8, NOT USED
+oct 11 mds length: length of mds header and mds data, starting from oct13
+oct 12
+oct 13 mds header and data
+The current sequence number/fragment number are being used in MDS for all
+messages sent to all discovered tipc portid(s), meaning that every message is 
+to any tipc portid, the sequence/fragment number is increased. The flow control
+needs its own sequence number sliding between two tipc porid(s) so that 
+can detect message drop due to buffer overload. Therefore, the oct8 and oct9 
+now reused as flow control sequence number. The oct10, protocol version, has 
+value of 0xB8. The format of new data message as below:
+oct 0  same
+oct 7
+oct 8  flow control sequence number
+oct 9
+oct 10 protocol version: (MDS_PROT_TIPC_FCTRL:0xB0 | MDS_VERSION:0x08) = 0xB8
+oct 11 same
+The ACK message is introduced to acknowledge one data message or a chunk of
+accumulative data message. The ACK message format:
+oct 0  message length
+oct 1
+oct 2  8 bytes, NOT USED
+oct 9
+oct 10 protocol version: (MDS_PROT_TIPC_FCTRL:0xB0 | MDS_VERSION:0x08) = 0xB8
+oct 11 protocol identifier: MDS_PROT_FCTRL_ID
+oct 14
+oct 15 flow control message type: CHUNKACK
+oct 16 service id: service id of data messages to be acknowledged
+oct 17
+oct 18 acknowledged sequence
+oct 19
+oct 20 chunk size
+oct 21
+Psuedo code illustrates the data message handling at MDS.
+if protocol version is 0xB8 then
+  if protocol identifier is MDS_PROT_FCTRL_ID then
+    this is mds flow control message.
+    if message type is CHUNKACK, then
+      this is ACK message for successfully delivered data message(s).
+  else
+    this is data message within flow control.
+    decode oct8,9 as flow control sequence number.
+  this is legacy data message.
+Because the legacy MDS does not use oct8,9,10, so it can communicate 
+to the new MDS in regard to the presence of flow control. Therefore, the 
+will not be affected.
+In case that the receiver end is at legacy MDS version, the new MDS has a timer
+mechanism to recognize if the receiver has no flow control supported. This 
+is implemented at the sender so that MDS will stop message queuing for a 
+-control MDS at receivers, namely tx-probation timer.
+MDS's sliding window
+One important factor that needs to be documented in the implementation of MDS's
+sliding window, is that "TIPC's link layer delivery guarantee, the only 
+factor for datagram delivery is the socket receive buffer size" [1][3][4]. 
+MDS at sender side does not have to implement the retransmission timer. Also, 
+MDS at sender side anticipates the buffer overflow at receiver ends, or 
+the first ancillary message, MDS starts queuing messages till the buffer 
+is resolved to resume data message transmission.
+(1) Sender sequence window
+acked_: last sequence has been acked by receiver
+send_:  next sequence to be sent
+nacked_space_: total bytes are sent but not acked
+   1     2     3     4     5     6     7     8
+            acked_                  send_
+If acked_:3, send_:7, then
+The message with sequence 1,2,3 have been acked.
+The 4,5,6 are sent but not acked, and are still in queue.
+The 7 is not sent yet.
+The nacked_space_: bytes_of(4,5,6)
+(2) Receiver sequence window
+acked_: last sequence has been acked to sender
+rcv_: last sequence has been received
+nacked_space_: total bytes has not been acked
+   1     2     3     4     5     6     7     8     9     10
+            acked_                        rcv_
+If acked_:3, rcv_:8
+The message with sequence 1,2,3: has been acked.
+The 4,5,6,7,8 are received by not acked, still in sender's queue.
+The 9,10 are not received yet
+The nacked_space_: bytes_of(4,5,6,7,8)
+TIPC portid state machine and its transition
+kDisabled, // no flow control support at this state
+kStartup,  // a newly published portid starts at this state
+kTxProb,   // txprob timer is running to confirm if the flow control is 

+kTxProb,   // the tx probation timer is running to confirm if the flow control 
is supported

kFlowControlEnabled or kFCEnabled ?
+kRcvBuffOverflow // anticipating (or experienced) the receiver's buffer 
+  kDisabled <--- kStartup --------
+     /|\             |           |
+      |              |           |
+      |              V           V
+      -----------kTxProb ---> kEnabled <---> kRcvBuffOverflow
+At the kRcvBuffOverflow state, the messages are being requested to send by 
+users will be enqueued at sender sides. When the state returns back to 
+the queued messages will be transmitted, the transmission is open for MDS's 
+At this version, MDS changes to kRcvBuffOverflow state if the TIPC_RETDATA 
+is returned, which is known as loss-based buffer overflow detection. Another
+approach is that MDS can utilize the TIPC_USED_RCV_BUFF TIPC socket option
+so that the senders can periodically get update of the receiver's TIPC sock 
+utilization. In that way, the senders can anticipate the buffer overflow in 
+which is called in MDS's context as a loss-less detection.
+ChunkAckTimeout timer: the receivers send the ACK message if this timer 
+If this ChunkAckTimeout is set too large, the round trip of data message
+acknowledgment increases, data message stays too long in the queue.
+ChunkAckSize: The number of message to be acknowledged in an ACK message. If
+this ChunkAckSize is too small, there will be a plentiful number of ACK 
+sent across two ends, which causes the overhead cost to MDS's message handling.
 1.3.1. Datagram Messaging
+[2] https://tools.ietf.org/html/rfc793, page 20.
 7.2.5. Sequence Control and Retransmission
+[4] http://tipc.sourceforge.net/protocol.html, 4.2. Link

Opensaf-devel mailing list

Reply via email to