Hi Steven,
I understood your opinion by mistake.
We do not have simple test case.
The phenomenon generated in our environment is the following thing.
Step 1) corosync constitutes a cluster in 12 nodes.
* begin communication in TOKEN
Step 2) One node raises [FAILED TO RECEIVE].
Step 3) 12 nodes begin the reconfiguration of the cluster again.
Step 4) The node that occurred fails([FAILED TO RECEIVE]) in an consensus of
the JOIN communication.
* Because the node failed in an consensus, node make contents of faildlist and
proclist same.
* And this node compares faildlist with proclist and assert-fail happened.
When the node that made a cluster stood alone, I think that assert() is
unnecessary.
Because the reason is because there is the next processing.
static void memb_join_process (
struct totemsrp_instance *instance,
const struct memb_join *memb_join)
{
struct srp_addr *proc_list;
struct srp_addr *failed_list;
(snip)
instance->failed_to_recv = 0;
srp_addr_copy (&instance->my_proc_list[0],
&instance->my_id);
instance->my_proc_list_entries = 1;
instance->my_failed_list_entries = 0;
memb_state_commit_token_create (instance);
memb_state_commit_enter (instance);
return;
(snip)
Best Regards,
Hideo Yamauchi.
--- [email protected] wrote:
> Hi Steven,
>
> > Hideo,
> >
> > If you have a test case, I can make a patch for you to try.
> >
>
> All right.
>
> We use corosync.1.3.0.
>
> Please send me patch.
>
> Best Regards,
> Hideo Yamauchi.
>
> --- Steven Dake <[email protected]> wrote:
>
> > On 02/06/2011 09:16 PM, [email protected] wrote:
> > > Hi Steven,
> > > Hi Dejan,
> > >
> > >>>>> This code never got a chance to run because on failed_to_recv
> > >>>>> the two sets (my_process_list and my_failed_list) are equal which
> > >>>>> makes the assert fail in memb_consensus_agreed():
> > >
> > > The same problem occurs, and we are troubled, too.
> > >
> > > How did this argument turn out?
> > >
> > > Best Regards,
> > > Hideo Yamauchi.
> > >
> >
> > Hideo,
> >
> > If you have a test case, I can make a patch for you to try.
> >
> > Regards
> > -steve
> >
> > >
> > > --- Dejan Muhamedagic <[email protected]> wrote:
> > >
> > >> nudge, nudge
> > >>
> > >> On Wed, Jan 05, 2011 at 02:05:55PM +0100, Dejan Muhamedagic wrote:
> > >>> On Tue, Jan 04, 2011 at 01:53:00PM -0700, Steven Dake wrote:
> > >>>> On 12/23/2010 06:14 AM, Dejan Muhamedagic wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> On Wed, Dec 01, 2010 at 05:30:44PM +0200, Vladislav Bogdanov wrote:
> > >>>>>> 01.12.2010 16:32, Dejan Muhamedagic wrote:
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> On Tue, Nov 23, 2010 at 12:53:42PM +0200, Vladislav Bogdanov wrote:
> > >>>>>>>> Hi Steven, hi all.
> > >>>>>>>>
> > >>>>>>>> I often see this assert on one of nodes after I stop corosync on
> > >>>>>>>> some
> > >>>>>>>> another node in newly-setup 4-node cluster.
> > >>>>>>>
> > >>>>>>> Does the assert happen on a node lost event? Or once new
> > >>>>>>> partition is formed?
> > >>>>>>
> > >>>>>> I first noticed it when I rebooted another node, just after console
> > >>>>>> said
> > >>>>>> that OpenAIS is stopped.
> > >>>>>>
> > >>>>>> Can't say right now, what exactly event did it follow, I'm actually
> > >>>>>> fighting with several problems with corosync, pacemaker, NFS4 and
> > >>>>>> phantom uncorrectable ECC errors simultaneously and I'm a bit lost
> > >>>>>> with
> > >>>>>> all of them.
> > >>>>>>
> > >>>>>>>
> > >>>>>>>> #0 0x00007f51953e49a5 in raise () from /lib64/libc.so.6
> > >>>>>>>> #1 0x00007f51953e6185 in abort () from /lib64/libc.so.6
> > >>>>>>>> #2 0x00007f51953dd935 in __assert_fail () from /lib64/libc.so.6
> > >>>>>>>> #3 0x00007f5196176406 in memb_consensus_agreed
> > >>>>>>>> (instance=0x7f5196554010) at totemsrp.c:1194
> > >>>>>>>> #4 0x00007f519617b2f3 in memb_join_process
> > >>>>>>>> (instance=0x7f5196554010,
> > >>>>>>>> memb_join=0x262f628) at totemsrp.c:3918
> > >>>>>>>> #5 0x00007f519617b619 in message_handler_memb_join
> > >>>>>>>> (instance=0x7f5196554010, msg=<value optimized out>, msg_len=<value
> > >>>>>>>> optimized out>, endian_conversion_needed=<value optimized out>)
> > >>>>>>>> at totemsrp.c:4161
> > >>>>>>>> #6 0x00007f5196173ba7 in passive_mcast_recv
> > >>>>>>>> (rrp_instance=0x2603030,
> > >>>>>>>> iface_no=0, context=<value optimized out>, msg=<value optimized
> > >>>>>>>> out>,
> > >>>>>>>> msg_len=<value optimized out>) at totemrrp.c:720
> > >>>>>>>> #7 0x00007f5196172b44 in rrp_deliver_fn (context=<value optimized
> > >>>>>>>> out>,
> > >>>>>>>> msg=0x262f628, msg_len=420) at totemrrp.c:1404
> > >>>>>>>> #8 0x00007f5196171a76 in net_deliver_fn (handle=<value optimized
> > >>>>>>>> out>,
> > >>>>>>>> fd=<value optimized out>, revents=<value optimized out>,
> > >>>>>>>> data=0x262ef80)
> > >>>>>>>> at totemudp.c:1244
> > >>>>>>>> #9 0x00007f519616d7f2 in poll_run (handle=4858364909567606784) at
> > >>>>>>>> coropoll.c:510
> > >>>>>>>> #10 0x0000000000406add in main (argc=<value optimized out>,
> > >>>>>>>> argv=<value
> > >>>>>>>> optimized out>, envp=<value optimized out>) at main.c:1680
> > >>>>>>>>
> > >>>>>>>> Last fplay lines are:
> > >>>>>>>>
> > >>>>>>>> rec=[36124] Log Message=Delivering MCAST message with seq 1366 to
> > >>>>>>>> pending delivery queue
> > >>>>>>>> rec=[36125] Log Message=Delivering MCAST message with seq 1367 to
> > >>>>>>>> pending delivery queue
> > >>>>>>>> rec=[36126] Log Message=Received ringid(10.5.4.52:12660) seq 1366
> > >>>>>>>> rec=[36127] Log Message=Received ringid(10.5.4.52:12660) seq 1367
> > >>>>>>>> rec=[36128] Log Message=Received ringid(10.5.4.52:12660) seq 1366
> > >>>>>>>> rec=[36129] Log Message=Received ringid(10.5.4.52:12660) seq 1367
> > >>>>>>>> rec=[36130] Log Message=releasing messages up to and including 1367
> > >>>>>>>> rec=[36131] Log Message=FAILED TO RECEIVE
> > >>>>>>>> rec=[36132] Log Message=entering GATHER state from 6.
> > >>>>>>>> rec=[36133] Log Message=entering GATHER state from 0.
> > >>>>>>>> Finishing replay: records found [33993]
> > >>>>>>>>
> > >>>>>>>> What could be the reason for this? Bug, switches, memory errors?
> > >>>>>>>
> > >>>>>>> The assertion fails because corosync finds out that
> > >>>>>>> instance->my_proc_list and instance->my_failed_list are
> > >>>>>>> equal. That happens immediately after the "FAILED TO RECEIVE"
> > >>>>>>> message which is issued when fail_recv_const token rotations
> > >>>>>>> happened without any multicast packet received (defaults to 50).
> > >>>>>
> > >>>>> I took a look at the code and the protocol specification again
> > >>>>> and it seems like that assert is not valid since Steve patched
> > >>>>> the part dealing with the "FAILED TO RECEIVE" condition. The
> > >>>>> patch is from 2010-06-03 posted to the list here
> > >>>>> http://marc.info/?l=openais&m=127559807608484&w=2
> > >>>>>
> > >>>>> The last hunk of the patch contains this code (exec/totemsrp.c):
> > >>>>>
> > >>>>> 3933 if (memb_consensus_agreed (instance) &&
> > >>>>> instance->failed_to_recv == 1) {
>
> > >>
> > >>>>> 3934 instance->failed_to_recv = 0;
> > >>>>> 3935 srp_addr_copy (&instance->my_proc_list[0],
> > >>>>> 3936 &instance->my_id);
> > >>>>> 3937 instance->my_proc_list_entries = 1;
> > >>>>> 3938 instance->my_failed_list_entries = 0;
> > >>>>> 3939
> > >>>>> 3940 memb_state_commit_token_create (instance);
> > >>>>> 3941
> > >>>>> 3942 memb_state_commit_enter (instance);
> > >>>>> 3943 return;
> > >>>>> 3944 }
> > >>>>>
> > >>>>> This code never got a chance to run because on failed_to_recv
> > >>>>> the two sets (my_process_list and my_failed_list) are equal which
> > >>>>> makes the assert fail in memb_consensus_agreed():
> > >>>>>
> > >>>>> 1185 memb_set_subtract (token_memb, &token_memb_entries,
> > >>>>> 1186 instance->my_proc_list, instance->my_proc_list_entries,
> > >>>>> 1187 instance->my_failed_list,
> > >>>>> instance->my_failed_list_entries);
> > >>>>> ...
> > >>>>> 1195 assert (token_memb_entries >= 1);
> > >>>>>
> > >>>>> In other words, it's something like this:
> > >>>>>
> > >>>>> if A:
> > >>>>> if memb_consensus_agreed() and failed_to_recv:
> > >>>>> form a single node ring and try to recover
> > >>>>>
> > >>>>> memb_consensus_agreed():
> > >>>>> assert(!A)
> > >>>>>
> > >>>>> Steve, can you take a look and confirm that this holds.
> > >>>>>
> > >>>>> Cheers,
> > >>>>>
> > >>>>
> > >>>> Dejan,
> > >>>>
> > >>>> sorry for delay in response - big backlog which is mostly cleared out
> > >>>> :)
> > >>>
> > >>> No problem.
> > >>>
> > >>>> The assert definitely isn't correct, but removing it without addressing
> > >>>> the contents of the proc and fail lists is also not right. That would
> > >>>> cause the logic in the if statement at line 3933 not to be executed
> > >>>> (because the first part of the if would evaluate to false)
> > >>>
> > >>> Actually it wouldn't. The agreed variable is set to 1 and it
> > >>> is going to be returned unchanged.
> > >>>
> > >>>> I believe
> > >>>> what we should do is check the "failed_to_recv" value in
> > >>>> memb_consensus_agreed instead of at line 3933.
> > >>>>
> > >>>> The issue with this is memb_state_consensus_timeout_expired which also
> > >>>> executes some 'then' logic where we may not want to execute the
> > >>>> failed_to_recv logic.
> > >>>
> > >>> Perhaps we should just
> > >>>
> > >>> 3933 if (instance->failed_to_recv == 1) {
> > >>>
> > >>> ? In case failed_to_recv both proc and fail lists are equal so
> > >>> checking for memb_consensus_agreed won't make sense, right?
> > >>>
> > >>>> If anyone has a reliable reproducer and can forward to me, I'll test
> > >>>> out
> > >>>> a change to address this problem. Really hesitant to change anything
> > >>>> in
> > >>>> totemsrp without a test case for this problem - its almost perfect ;-)
> > >>>
> > >>> Since the tester upgraded the switch firmware they couldn't
> > >>> reproduce it anymore.
> > >>>
> > >>> Would compiling with these help?
> > >>>
> > >>> /*
> > >>> * These can be used to test the error recovery algorithms
> > >>> * #define TEST_DROP_ORF_TOKEN_PERCENTAGE 30
> > >>> * #define TEST_DROP_COMMIT_TOKEN_PERCENTAGE 30
> > >>> * #define TEST_DROP_MCAST_PERCENTAGE 50
> > >>> * #define TEST_RECOVERY_MSG_COUNT 300
> > >>> */
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Dejan
> > >>>
> > >>>> Regards
> > >>>> -steve
> > >>>>
> > >>>>> Dejan
> > >>>>> _______________________________________________
> > >>>>> Openais mailing list
> > >>>>> [email protected]
> > >>>>> https://lists.linux-foundation.org/mailman/listinfo/openais
> > >>>>
> > >>>> _______________________________________________
> > >>>> Openais mailing list
> > >>>> [email protected]
> > >>>> https://lists.linux-foundation.org/mailman/listinfo/openais
> > >>> _______________________________________________
> > >>> Openais mailing list
> > >>> [email protected]
> > >>> https://lists.linux-foundation.org/mailman/listinfo/openais
> > >> _______________________________________________
> > >> Openais mailing list
> > >> [email protected]
> > >> https://lists.linux-foundation.org/mailman/listinfo/openais
> > >>
> > >
> > > _______________________________________________
> > > Openais mailing list
> > > [email protected]
> > > https://lists.linux-foundation.org/mailman/listinfo/openais
> >
> >
>
> _______________________________________________
> Openais mailing list
> [email protected]
> https://lists.linux-foundation.org/mailman/listinfo/openais
>
_______________________________________________
Openais mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/openais
