Hi Anders Widell,

On 9/16/2016 2:03 PM, Anders Widell wrote:
> Another approach we could consider is that MDS retransmits the message 
> transparently without informing the sender. This would require MDS to 
> internally store sent messages for a while, so that they can be 
> retransmitted. It would also require the receiver to re-order received 
> messages, since a retransmitted message will be received out of sequence.

This is nothing but flow control , this is a long pending enhancement .

-AVM

On 9/16/2016 2:03 PM, Anders Widell wrote:
>
> I don't think we need (or even should) inform the sender when MDS 
> receives an error information message from TIPC. Note that these error 
> information messages are received asynchronously, when the sender has 
> already received an OK return code from the MDS send call. The idea 
> was to just log reception of error info messages, for trouble-shooting 
> purposes. We already have a mechanism in MDS that informs the receiver 
> about lost MDS messages. If we wish to inform the sender we would need 
> to introduce a second mechanism in MDS, and at this point I don't 
> think it is needed. Another approach we could consider is that MDS 
> retransmits the message transparently without informing the sender. 
> This would require MDS to internally store sent messages for a while, 
> so that they can be retransmitted. It would also require the receiver 
> to re-order received messages, since a retransmitted message will be 
> received out of sequence.
>
> regards,
>
> Anders Widell
>
>
> On 09/16/2016 06:40 AM, A V Mahesh wrote:
>> Hi HansN,
>>
>> I managed to create TIPC_ERRINFO/TIPC_RETDATA  error cases ( not  
>> TIPC_ERR_OVERLOAD error )  with normal messages
>> and It is observed that  TIPC_DEST_DROPPABLE set to true even error 
>> TIPC_ERRINFO is NOT notified ( it means TIPC_ERR_OVERLOAD ) ,
>> if TIPC_DEST_DROPPABLE set to false TIPC_ERRINFO/TIPC_RETDATA errors 
>> are notified.
>>
>> Now I will also check implication of TIPC_DEST_DROPPABLE set to false 
>> on multicast and broadcast  messages, based on that
>> we can re-arrange the TIPC_DEST_DROPPABLE setting to false 
>> conditions  based on agent `i_msg_loss_indication = true` condition
>> mds can return to agent the same error  TIPC_ERR_OVERLOAD.
>>
>> TIPC_DEST_DROPPABLE to false:
>>
>> ==================================================================
>>
>> Sep 15 16:10:39 SC-1 osafimmnd[32051]: NO Implementer disconnected 13 
>> <0, 2040f> (MsgQueueService132111)
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: NO MDS event from svc_id 25 
>> (change:4, dest:567413369208836)
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafimmd[32040]:  777 MDTM: undelivered message 
>> condition ancillary data: TIPC_ERRINFO abort err : 2
>> Sep 15 16:10:39 SC-1 osafimmd[32040]: 7777 MDTM: undelivered message 
>> condition ancillary data: TIPC_RETDATA
>> Sep 15 16:10:39 SC-1 osafamfd[32114]: NO Node 'PL-4' left the cluster
>>
>> ==================================================================
>>
>> TIPC_DEST_DROPPABLE to true:
>>
>> ==================================================================
>>
>> Sep 15 15:59:55 SC-1 osafimmnd[26461]: NO Implementer disconnected 13 
>> <0, 2040f> (MsgQueueService132111)
>> Sep 15 15:59:55 SC-1 osafimmd[26450]: NO MDS event from svc_id 25 
>> (change:4, dest:567412923957252)
>> Sep 15 15:59:55 SC-1 osafimmnd[26461]: NO Global discard node 
>> received for nodeId:2040f pid:410
>> Sep 15 15:59:55 SC-1 osafamfd[28810]: NO Node 'PL-4' left the cluster
>> Sep 15 15:59:58 SC-1 kernel: [ 5147.648737] tipc: Resetting link 
>> <1.1.1:eth0-1.1.4:eth0>, peer not responding
>> Sep 15 15:59:58 SC-1 kernel: [ 5147.648756] tipc: Lost link 
>> <1.1.1:eth0-1.1.4:eth0> on network plane A
>> Sep 15 15:59:58 SC-1 kernel: [ 5147.648771] tipc: Lost contact with 
>> <1.1.4>
>>
>> ==================================================================
>>
>> -AVM
>>
>>
>> On 9/1/2016 10:59 AM, Hans Nordebäck wrote:
>>> Hi Mahesh,
>>>
>>> I have not tested this, but the following should work:
>>>
>>> - Set BSRsock TIPC_IMPORTANCE to TIPC_LOW_IMPORTANCE
>>>
>>> - set socket receive buffer to a small value:
>>>
>>>   optval = "small socket recieive buffer size" , 5000 ?
>>>
>>>   setsockopt(tipc_cb.BSRsock, SOL_SOCKET, SO_RCVBUF, &optval, optlen)
>>>
>>> -  sysctl -w net.tipc.tipc_rmem="5000 40000000 68240400" (or smaller 
>>> values)
>>>
>>> - add some delays when processing messages in 
>>> mdtm_process_recv_events(), to provoke overloading the socket 
>>> receive buffer.
>>>
>>> We experience dropped packages in a 75 node system, and as a 
>>> workaround increasing the default so receive buffer size it seems 
>>> working for that setup.
>>>
>>> /Thanks HansN
>>>
>>> On 09/01/2016 05:50 AM, A V Mahesh wrote:
>>>> Hi HansN,
>>>>
>>>> Do you have any tips to created overload case,
>>>>
>>>> I would like test and observe TIPC_DEST_DROPPABLE enabled & 
>>>> disabled cases.
>>>>
>>>> -AVM
>>>>
>>>>
>>>> On 9/1/2016 9:12 AM, A V Mahesh wrote:
>>>>> Hi HansN,
>>>>>
>>>>> Sorry for the delay.
>>>>>
>>>>> I will test it and get back to you soon.
>>>>>
>>>>> -AVM
>>>>>
>>>>>
>>>>> On 8/31/2016 4:29 PM, Hans Nordebäck wrote:
>>>>>> Hi Mahesh,
>>>>>> Any updates on this?
>>>>>>
>>>>>> /Regards HansN
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Anders Widell
>>>>>> Sent: den 25 augusti 2016 13:11
>>>>>> To: A V Mahesh <mahesh.va...@oracle.com>; Hans Nordebäck 
>>>>>> <hans.nordeb...@ericsson.com>; mathi.naic...@oracle.com
>>>>>> Cc: opensaf-devel@lists.sourceforge.net
>>>>>> Subject: Re: [PATCH 1 of 1] MDS: Log TIPC dropped messages [#1957]
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> This is what the TIPC user documentation says about 
>>>>>> TIPC_DEST_DROPPABLE:
>>>>>> "This option governs the handling of messages sent by the socket 
>>>>>> if the message cannot be delivered to its destination, either 
>>>>>> because the receiver is congested or because the specified 
>>>>>> receiver does not exist.
>>>>>> If enabled, the message is discarded; otherwise the message is 
>>>>>> returned to the sender."
>>>>>>
>>>>>> This is what the TIPC user documentation says about the return 
>>>>>> value from the recvmsg() system call: "When used with a 
>>>>>> connectionless socket, a return value of 0 indicates the arrival 
>>>>>> of a returned data message that was originally sent by this socket."
>>>>>>
>>>>>> I think the documentation is pretty clear. If you set 
>>>>>> TIPC_DEST_DROPPABLE to true, the receiver can discard messages 
>>>>>> e.g. when the receive buffer is full. The sender will not be 
>>>>>> notified in this case. If TIPC_DEST_DROPPABLE is set to false, 
>>>>>> the message will be returned to the sender in case of a full 
>>>>>> receive buffer. The sender knows that it has received such a 
>>>>>> returned message when the recvmsg() call returns zero.
>>>>>>
>>>>>> regards,
>>>>>> Anders Widell
>>>>>>
>>>>>> On 08/25/2016 11:30 AM, A V Mahesh wrote:
>>>>>>> Hi HansN,
>>>>>>>
>>>>>>>
>>>>>>> On 8/23/2016 5:22 PM, Hans Nordebäck wrote:
>>>>>>>
>>>>>>>> Hi Mahesh,
>>>>>>>>
>>>>>>>> Yes, this is my understanding too, if TIPC_DROPPABLE = true 
>>>>>>>> tipc may
>>>>>>>> drop messages silently,  at receive sock buffer full 
>>>>>>>> condition,  but
>>>>>>>> do not return any ancillary message.
>>>>>>>> If TIPC_DROPPABLE = false tipc may drop message but will send an
>>>>>>>> ancillary message to inform about TIPC_ERR_OVERLOAD.
>>>>>>> [AVM]
>>>>>>>
>>>>>>> My observation are understanding is different, based on TIPC 
>>>>>>> code and
>>>>>>> Linux TIPC 2.0 Programmer's Guide , that the TIPC_ERR_OVERLOAD 
>>>>>>> error
>>>>>>> returned when TIPC is unable to enqueue an incoming message on the
>>>>>>> receiving socket's receive queue irrelevant of TIPC_DEST_DROPPABLE
>>>>>>> enabled or disabled.
>>>>>>>
>>>>>>> The only difference between TIPC_DEST_DROPPABLE enabled or 
>>>>>>> disabled is
>>>>>>> , If  TIPC_DEST_DROPPABLE enabled, the message is discarded and
>>>>>>> recvmsg() returned size is ZERO and application will get errors, if
>>>>>>> TIPC_DEST_DROPPABLE disabled  the message is returned to the 
>>>>>>> sender it
>>>>>>> means the recvmsg() returned size is user send data size and
>>>>>>> application will get errors .
>>>>>>>
>>>>>>> I did check the TIPC code and documentations  and I haven't get any
>>>>>>> evidences that  TIPC_ERR_OVERLOAD error code will be send only If
>>>>>>> TIPC_DEST_DROPPABLE = false.
>>>>>>>
>>>>>>> Even while testing #1227
>>>>>>> (https://sourceforge.net/p/opensaf/mailman/message/33207717/) my
>>>>>>> observations and understanding was, an individual TIPC socket is 
>>>>>>> only
>>>>>>> allowed to queue up
>>>>>>> OVERLOAD_LIMIT_BASE/2 messages of the lowest importance level 
>>>>>>> before
>>>>>>> it starts rejecting them.
>>>>>>> Once a socket receiving queue length exceeds the maximum limit 
>>>>>>> value,
>>>>>>> the receiving socket will send out a reject message with
>>>>>>> TIPC_ERR_OVERLOAD error code with cmsg_type as
>>>>>>> TIPC_ERRINFO/TIPC_RETDATA, and the tipc code and Linux TIPC 2.0
>>>>>>> Programmer's Guide  confirmed the same .
>>>>>>>
>>>>>>> tipc/socket.c
>>>>>>> =======================================================
>>>>>>> /* Reject message if there isn't room to queue it */
>>>>>>>
>>>>>>> recv_q_len = (u32)atomic_read(&tipc_queue_size);
>>>>>>> if (unlikely(recv_q_len >= OVERLOAD_LIMIT_BASE)) {
>>>>>>>      if (rx_queue_full(msg, recv_q_len, OVERLOAD_LIMIT_BASE))
>>>>>>>          return TIPC_ERR_OVERLOAD;
>>>>>>> }
>>>>>>> recv_q_len = skb_queue_len(&sk->sk_receive_queue);
>>>>>>> if (unlikely(recv_q_len >= (OVERLOAD_LIMIT_BASE / 2))) {
>>>>>>>      if (rx_queue_full(msg, recv_q_len, OVERLOAD_LIMIT_BASE / 2))
>>>>>>>          return TIPC_ERR_OVERLOAD;
>>>>>>> }
>>>>>>> =======================================================
>>>>>>>
>>>>>>>
>>>>>>> 2.1.17. setsockopt() of  TIPC 2.0 Programmer's Guide
>>>>>>> =======================================================
>>>>>>> TIPC_DEST_DROPPABLE
>>>>>>> This option governs the handling of messages sent by the socket 
>>>>>>> if the
>>>>>>> message cannot be delivered to its destination, either because the
>>>>>>> receiver is congested or because the specified receiver does not
>>>>>>> exist. If enabled, the message is discarded; otherwise the 
>>>>>>> message is
>>>>>>> returned to the sender.
>>>>>>>
>>>>>>> By default, this option is disabled for SOCK_SEQPACKET and 
>>>>>>> SOCK_STREAM
>>>>>>> socket types, and enabled for SOCK_RDM and SOCK_DGRAM, This
>>>>>>> arrangement ensures proper teardown of failed connections when
>>>>>>> connection-oriented data transfer is used, without increasing the
>>>>>>> complexity of connectionless data transfer.
>>>>>>>
>>>>>>> TIPC_SRC_DROPPABLE
>>>>>>> This option governs the handling of messages sent by the socket if
>>>>>>> link congestion occurs. If enabled, the message is discarded;
>>>>>>> otherwise the system queues the message for later transmission.
>>>>>>> By default, this option is disabled for SOCK_SEQPACKET, 
>>>>>>> SOCK_STREAM,
>>>>>>> and SOCK_RDM socket types (resulting in "reliable" data 
>>>>>>> transfer), and
>>>>>>> enabled for SOCK_DGRAM (resulting in "unreliable" data transfer).
>>>>>>> =======================================================
>>>>>>>
>>>>>>> Now I will try to create OVERLOAD case and update you soon my 
>>>>>>> latest
>>>>>>> observations.
>>>>>>>
>>>>>>> -AVM
>>>>>>>
>>>>>>>> Correcting this and adding an abort is not backward compatible as
>>>>>>>> some service already handle flow control in some way, only log 
>>>>>>>> when
>>>>>>>> packages are dropped.
>>>>>>>> Regarding ticket #1960 there are other solutions than introducing
>>>>>>>> flow control in MDS, e.g. expose an option to the service to 
>>>>>>>> choose
>>>>>>>> connection oriented or connection less.
>>>>>>>> The problem with dropped messages seems in one case related to, 
>>>>>>>> (by
>>>>>>>> MDS), intensive MDS logging.
>>>>>>>>
>>>>>>>> /Thanks HansN
>>>>>>>> -----Original Message-----
>>>>>>>> From: A V Mahesh [mailto:mahesh.va...@oracle.com]
>>>>>>>> Sent: den 23 augusti 2016 11:27
>>>>>>>> To: Hans Nordebäck <hans.nordeb...@ericsson.com>; Anders Widell
>>>>>>>> <anders.wid...@ericsson.com>; mathi.naic...@oracle.com
>>>>>>>> Cc: opensaf-devel@lists.sourceforge.net
>>>>>>>> Subject: Re: [PATCH 1 of 1] MDS: Log TIPC dropped messages [#1957]
>>>>>>>>
>>>>>>>> Hi HansN,
>>>>>>>>
>>>>>>>> It seems I am missing some thing , please allow me to under stand
>>>>>>>>
>>>>>>>> If I currently understand you observation :
>>>>>>>>
>>>>>>>> With current Opensaf code ( this #1957 patch NOT applied ) , by
>>>>>>>> default TIPC_DROPPABLE=true ,while running Opensaf with that 
>>>>>>>> binary
>>>>>>>> when TIPC_ERR_OVERLOAD  occurring, TIPC is not given errors
>>>>>>>> TIPC_ERRINFO or  TIPC_RETDATA and following code is not being 
>>>>>>>> get hit
>>>>>>>> of function recvfrom_connectionless(), is my understanding right ?
>>>>>>>>
>>>>>>>> ===================================================================== 
>>>>>>>>
>>>>>>>> ========================================
>>>>>>>>
>>>>>>>>
>>>>>>>> *if (anc->cmsg_type == TIPC_ERRINFO) {*
>>>>>>>>        /* TIPC_ERRINFO - TIPC error code associated with a 
>>>>>>>> returned
>>>>>>>> data message or a connection termination message  so abort */
>>>>>>>>        m_MDS_LOG_CRITICAL("MDTM: undelivered message condition
>>>>>>>> ancillary
>>>>>>>> data: TIPC_ERRINFO abort err :%s", strerror(errno) );
>>>>>>>> *abort();*
>>>>>>>> *} else if (anc->cmsg_type == TIPC_RETDATA) {*
>>>>>>>>        /* If we set TIPC_DEST_DROPPABLE off messge (configure 
>>>>>>>> TIPC to
>>>>>>>> return rejected messages to the sender )
>>>>>>>>           we will hit this when we implement MDS retransmit lost
>>>>>>>> messages abort can be replaced with flow control logic*/
>>>>>>>>        for (i = anc->cmsg_len - sizeof(*anc); i > 0; i--) {
>>>>>>>>            m_MDS_LOG_DBG("MDTM: returned byte 0x%02x\n", *cptr);
>>>>>>>>            cptr++;
>>>>>>>>        }
>>>>>>>>        /* TIPC_RETDATA -The contents of a returned data message so
>>>>>>>> abort */
>>>>>>>>        m_MDS_LOG_CRITICAL("MDTM: undelivered message condition
>>>>>>>> ancillary
>>>>>>>> data: TIPC_RETDATA abort err :%s", strerror(errno) );
>>>>>>>> *abort();*
>>>>>>>> }
>>>>>>>>
>>>>>>>> ===================================================================== 
>>>>>>>>
>>>>>>>> ========================================
>>>>>>>>
>>>>>>>>
>>>>>>>> -AVM
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/23/2016 1:08 PM, Hans Nordebäck wrote:
>>>>>>>>> Hi Mahesh,
>>>>>>>>>
>>>>>>>>> Please see response below with [HansN] /Thanks HansN
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: A V Mahesh [mailto:mahesh.va...@oracle.com]
>>>>>>>>> Sent: den 23 augusti 2016 08:25
>>>>>>>>> To: Hans Nordebäck <hans.nordeb...@ericsson.com>; Anders Widell
>>>>>>>>> <anders.wid...@ericsson.com>; mathi.naic...@oracle.com
>>>>>>>>> Cc: opensaf-devel@lists.sourceforge.net
>>>>>>>>> Subject: Re: [PATCH 1 of 1] MDS: Log TIPC dropped messages 
>>>>>>>>> [#1957]
>>>>>>>>>
>>>>>>>>> Hi HansN
>>>>>>>>>
>>>>>>>>> Please see response below with [AVM]
>>>>>>>>>
>>>>>>>>> -AVM
>>>>>>>>>
>>>>>>>>> On 8/23/2016 11:41 AM, Hans Nordebäck wrote:
>>>>>>>>>> Hi Mahesh,
>>>>>>>>>>
>>>>>>>>>> please see comments below.
>>>>>>>>>>
>>>>>>>>>> /Thanks HansN
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08/23/2016 07:21 AM, A V Mahesh wrote:
>>>>>>>>>>> Hi HansN,
>>>>>>>>>>>
>>>>>>>>>>> Let us fist discuss the error handling and abort, then we 
>>>>>>>>>>> can come
>>>>>>>>>>> back to interpretation of  TIPC currently does permit  OR does
>>>>>>>>>>> not permit an application to send a multicast message with the
>>>>>>>>>>> "destination droppable" setting disabled.
>>>>>>>>>>>
>>>>>>>>>>> Let us disable TIPC_DEST_DROPPABLE, so that TIPC will try to
>>>>>>>>>>> return an undelivered multicast message to its sender and we 
>>>>>>>>>>> can
>>>>>>>>>>> determine issue is  because of TIPC_ERR_OVERLOAD, this helps in
>>>>>>>>>>> debugging , so that application may increased 
>>>>>>>>>>> SO_SNDBUF/SO_RCVBUF
>>>>>>>>>>> to reduce the problem.
>>>>>>>>>>>
>>>>>>>>>>> But still we need to abort(), the reason for that is current 
>>>>>>>>>>> MDS
>>>>>>>>>>> implementations doesn't have flow control logic ( no retry 
>>>>>>>>>>> because
>>>>>>>>>>> of error ) , so Application like AMF can go wrong and 
>>>>>>>>>>> cluster will
>>>>>>>>>>> go into unstable/recoverble state.
>>>>>>>>>>>
>>>>>>>>>> [HansN] In the current implementation messages are dropped 
>>>>>>>>>> silently
>>>>>>>>>> and no abort is done.
>>>>>>>>> [AVM]  I can see  abort(); in current code , you mean abort(); is
>>>>>>>>> not working and application(amf) is not existing ?
>>>>>>>>> [HansN] In case of TIPC_DROPPABLE=true and messages are dropped,
>>>>>>>>> (TIPC_ERR_OVERLOAD)  no abort is be performed, e.g amfd 
>>>>>>>>> detects this
>>>>>>>>> in the msg sanity chk and logs "invalid msg id ..."
>>>>>>>>> ==================================================================== 
>>>>>>>>>
>>>>>>>>> ==
>>>>>>>>> ======
>>>>>>>>> if (anc->cmsg_type == TIPC_ERRINFO) {
>>>>>>>>>         /* TIPC_ERRINFO - TIPC error code associated with a 
>>>>>>>>> returned
>>>>>>>>> data message or a connection termination message so abort */
>>>>>>>>>         m_MDS_LOG_CRITICAL("MDTM: undelivered message condition
>>>>>>>>> ancillary
>>>>>>>>> data: TIPC_ERRINFO abort err :%s", strerror(errno) );
>>>>>>>>> *abort();*
>>>>>>>>> } else if (anc->cmsg_type == TIPC_RETDATA) {
>>>>>>>>>         /* If we set TIPC_DEST_DROPPABLE off messge (configure 
>>>>>>>>> TIPC
>>>>>>>>> to return rejected messages to the sender )
>>>>>>>>>            we will hit this when we implement MDS retransmit lost
>>>>>>>>> messages abort can be replaced with flow control logic*/
>>>>>>>>>         for (i = anc->cmsg_len - sizeof(*anc); i > 0; i--) {
>>>>>>>>>             m_MDS_LOG_DBG("MDTM: returned byte 0x%02x\n", *cptr);
>>>>>>>>>             cptr++;
>>>>>>>>>         }
>>>>>>>>>         /* TIPC_RETDATA -The contents of a returned data 
>>>>>>>>> message  so
>>>>>>>>> abort */
>>>>>>>>>         m_MDS_LOG_CRITICAL("MDTM: undelivered message condition
>>>>>>>>> ancillary
>>>>>>>>> data: TIPC_RETDATA abort err :%s", strerror(errno) );
>>>>>>>>> *abort();*
>>>>>>>>> }
>>>>>>>>> ==================================================================== 
>>>>>>>>>
>>>>>>>>> ==
>>>>>>>>> ======
>>>>>>>>>> This patch enables logging
>>>>>>>>>> when packages are dropped to help in debugging. I don't agree 
>>>>>>>>>> that
>>>>>>>>>> we should also introduce abort, but instead:
>>>>>>>>>> 1) Implement a solution to handle dropped packages, ticket #1960
>>>>>>>>> [AVM]  This is nothing but flow control implementation in MDS, 
>>>>>>>>> this
>>>>>>>>> is future enhancement
>>>>>>>>>
>>>>>>>>>> 2) Investigate why packages may be dropped, the receiving MDS
>>>>>>>>>> thread is a real time thread and should be able to consume a 
>>>>>>>>>> large
>>>>>>>>>> amount of incoming messages.
>>>>>>>>>> E.g. is the receiving MDS thread "live hanging" due to locks, 
>>>>>>>>>> file
>>>>>>>>>> I/O etc?
>>>>>>>>>>> This was the reason we haven't gone for it while addressing 
>>>>>>>>>>> Ticket
>>>>>>>>>>> #1227
>>>>>>>>>>> (https://sourceforge.net/p/opensaf/mailman/message/33207717/)
>>>>>>>>>>> So currently we don't have any advantage of disabling
>>>>>>>>>>> TIPC_DEST_DROPPABLE and not allowing multicast messages.
>>>>>>>>>>>
>>>>>>>>>>> -AVM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 8/18/2016 2:43 PM, Hans Nordeback wrote:
>>>>>>>>>>>> osaf/libs/core/mds/mds_dt_tipc.c |  32
>>>>>>>>>>>> +++++++++++++++++++++++++-------
>>>>>>>>>>>>      1 files changed, 25 insertions(+), 7 deletions(-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/osaf/libs/core/mds/mds_dt_tipc.c
>>>>>>>>>>>> b/osaf/libs/core/mds/mds_dt_tipc.c
>>>>>>>>>>>> --- a/osaf/libs/core/mds/mds_dt_tipc.c
>>>>>>>>>>>> +++ b/osaf/libs/core/mds/mds_dt_tipc.c
>>>>>>>>>>>> @@ -320,6 +320,15 @@ uint32_t mdtm_tipc_init(NODE_ID nodeid,
>>>>>>>>>>>>                      m_MDS_LOG_INFO("MDTM: Successfully set
>>>>>>>>>>>> default socket option TIPC_IMP = %d", TIPCIMPORTANCE);
>>>>>>>>>>>>              }
>>>>>>>>>>>>      +        int droppable = 0;
>>>>>>>>>>>> +        if (setsockopt(tipc_cb.BSRsock, SOL_TIPC,
>>>>>>>>>>>> TIPC_DEST_DROPPABLE, &droppable, sizeof(droppable)) != 0) {
>>>>>>>>>>>> +                LOG_ER("MDTM: Can't set 
>>>>>>>>>>>> TIPC_DEST_DROPPABLE to
>>>>>>>>>>>> + zero
>>>>>>>>>>>> err :%s\n", strerror(errno));
>>>>>>>>>>>> +                m_MDS_LOG_ERR("MDTM: Can't set
>>>>>>>>>>>> + TIPC_DEST_DROPPABLE
>>>>>>>>>>>> to zero err :%s\n", strerror(errno));
>>>>>>>>>>>> +                osafassert(0);
>>>>>>>>>>>> +        } else {
>>>>>>>>>>>> +                m_MDS_LOG_NOTIFY("MDTM: Successfully set
>>>>>>>>>>>> TIPC_DEST_DROPPABLE to zero");
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +
>>>>>>>>>>>>          return NCSCC_RC_SUCCESS;
>>>>>>>>>>>>      }
>>>>>>>>>>>>      @@ -563,6 +572,8 @@ ssize_t recvfrom_connectionless 
>>>>>>>>>>>> (int sd,
>>>>>>>>>>>>          unsigned char *cptr;
>>>>>>>>>>>>          int i;
>>>>>>>>>>>>          int has_addr;
>>>>>>>>>>>> +    int anc_data[2];
>>>>>>>>>>>> +
>>>>>>>>>>>>          ssize_t sz;
>>>>>>>>>>>>            has_addr = (from != NULL) && (addrlen != NULL); @@
>>>>>>>>>>>> -591,19
>>>>>>>>>>>> +602,26 @@ ssize_t recvfrom_connectionless (int sd,
>>>>>>>>>>>>                     if the message was sent using a TIPC 
>>>>>>>>>>>> name or
>>>>>>>>>>>> name sequence as the
>>>>>>>>>>>>                     destination rather than a TIPC port ID So
>>>>>>>>>>>> abort for TIPC_ERRINFO and TIPC_RETDATA*/
>>>>>>>>>>>>                  if (anc->cmsg_type == TIPC_ERRINFO) {
>>>>>>>>>>>> -                /* TIPC_ERRINFO - TIPC error code 
>>>>>>>>>>>> associated with a
>>>>>>>>>>>> returned data message or a connection termination message  so
>>>>>>>>>>>> abort */
>>>>>>>>>>>> -                m_MDS_LOG_CRITICAL("MDTM: undelivered message
>>>>>>>>>>>> condition ancillary data: TIPC_ERRINFO abort err :%s",
>>>>>>>>>>>> strerror(errno) );
>>>>>>>>>>>> -                abort();
>>>>>>>>>>>> +                anc_data[0] = *((unsigned 
>>>>>>>>>>>> int*)(CMSG_DATA(anc) +
>>>>>>>>>>>> 0));
>>>>>>>>>>>> +                if (anc_data[0] == TIPC_ERR_OVERLOAD) {
>>>>>>>>>>>> +                    LOG_CR("MDTM: undelivered message 
>>>>>>>>>>>> condition
>>>>>>>>>>>> ancillary data: TIPC_ERR_OVERLOAD");
>>>>>>>>>>>> + m_MDS_LOG_CRITICAL("MDTM: undelivered
>>>>>>>>>>>> + message
>>>>>>>>>>>> condition ancillary data: TIPC_ERR_OVERLOAD");
>>>>>>>>>>>> +                } else {
>>>>>>>>>>>> +                    /* TIPC_ERRINFO - TIPC error code 
>>>>>>>>>>>> associated
>>>>>>>>>>>> with a returned data message or a connection termination 
>>>>>>>>>>>> message
>>>>>>>>>>>> so abort */
>>>>>>>>>>>> +                    LOG_CR("MDTM: undelivered message 
>>>>>>>>>>>> condition
>>>>>>>>>>>> ancillary data: TIPC_ERRINFO abort err : %d", anc_data[0]);
>>>>>>>>>>>> + m_MDS_LOG_CRITICAL("MDTM: undelivered
>>>>>>>>>>>> + message
>>>>>>>>>>>> condition ancillary data: TIPC_ERRINFO abort err : %d",
>>>>>>>>>>>> anc_data[0]);
>>>>>>>>>>>> +                }
>>>>>>>>>>>>                  } else if (anc->cmsg_type == TIPC_RETDATA) {
>>>>>>>>>>>> -                /* If we set TIPC_DEST_DROPPABLE off messge
>>>>>>>>>>>> (configure TIPC to return rejected messages to the sender )
>>>>>>>>>>>> +                /* If we set TIPC_DEST_DROPPABLE off message
>>>>>>>>>>>> (configure TIPC to return rejected messages to the sender )
>>>>>>>>>>>>                         we will hit this when we implement MDS
>>>>>>>>>>>> retransmit lost messages  abort can be replaced with flow 
>>>>>>>>>>>> control
>>>>>>>>>>>> logic*/
>>>>>>>>>>>>                      for (i = anc->cmsg_len - sizeof(*anc); 
>>>>>>>>>>>> i > 0;
>>>>>>>>>>>> i--) {
>>>>>>>>>>>> -                    m_MDS_LOG_DBG("MDTM: returned byte 
>>>>>>>>>>>> 0x%02x\n",
>>>>>>>>>>>> *cptr);
>>>>>>>>>>>> +                    LOG_CR("MDTM: returned byte 0x%02x\n", 
>>>>>>>>>>>> *cptr);
>>>>>>>>>>>> + m_MDS_LOG_CRITICAL("MDTM: returned byte
>>>>>>>>>>>> 0x%02x\n", *cptr);
>>>>>>>>>>>>                          cptr++;
>>>>>>>>>>>>                      }
>>>>>>>>>>>>                      /* TIPC_RETDATA -The contents of a 
>>>>>>>>>>>> returned
>>>>>>>>>>>> data message  so abort */
>>>>>>>>>>>> -                m_MDS_LOG_CRITICAL("MDTM: undelivered message
>>>>>>>>>>>> condition ancillary data: TIPC_RETDATA abort err :%s",
>>>>>>>>>>>> strerror(errno) );
>>>>>>>>>>>> -                abort();
>>>>>>>>>>>> +                LOG_CR("MDTM: undelivered message condition
>>>>>>>>>>>> ancillary data: TIPC_RETDATA");
>>>>>>>>>>>> +                m_MDS_LOG_CRITICAL("MDTM: undelivered message
>>>>>>>>>>>> condition ancillary data: TIPC_RETDATA");
>>>>>>>>>>>>                  } else if (anc->cmsg_type == TIPC_DESTNAME) {
>>>>>>>>>>>>                      if (sz == 0) {
>>>>>>>>>>>> m_MDS_LOG_DBG("MDTM: recd bytes=0 on
>>>>>>>>>>>> received on sock, abnormal/unknown condition. Ignoring");
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>


------------------------------------------------------------------------------
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel

Reply via email to