Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-17 Thread Balaji Venkatraman via lists.fd.io
Hi Satya, Inline.. From: on behalf of Satya Murthy Date: Friday, July 17, 2020 at 3:58 AM To: "vpp-dev@lists.fd.io" Subject: Re: [vpp-dev] Crash in ACL code while applying rulematch Hi Andrew, We could not try the same test with the change you suggested. We will try i

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-17 Thread Satya Murthy
Hi Andrew, We could not try the same test with the change you suggested. We will try it for sure in this week. Regarding our scenario for example: we have 4 workers. All the 4 workers can create ACLs and also can process data. During high load of data and create ACLs, we are seeing this crash.

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-16 Thread Andrew Yourtchenko
Hi Satya, > On 16 Jul 2020, at 18:44, Satya Murthy wrote: > > Hi Andrew, > > Thanks for your suggestions on this. > > We did few more tests today, and here are some observations. > Please let us know your thoughts on this. > > 1) We added a lock on the datapath also while doing rulematch

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-16 Thread Balaji Venkatraman via lists.fd.io
. From: on behalf of Satya Murthy Date: Thursday, July 16, 2020 at 9:44 AM To: "vpp-dev@lists.fd.io" Subject: Re: [vpp-dev] Crash in ACL code while applying rulematch Hi Andrew, Thanks for your suggestions on this. We did few more tests today, and here are some observations. Please l

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-16 Thread Satya Murthy
Hi Andrew, Thanks for your suggestions on this. We did few more tests today, and here are some observations. Please let us know your thoughts on this. 1) We added a lock on the datapath also while doing rulematch using acl apis. The same lock is getting acquired while doing acl adds from other

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-15 Thread Andrew Yourtchenko
Hi Satya, Yeah so I suspect since the updates were never MP-safe, they required the workers getting the barrier; so the condition I speculated about in my other mail won’t have occurred... (I am still puzzled by why you see no problem if updating for only one worker, so not sure if that

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-15 Thread Andrew Yourtchenko
That said, my theory is that you are in the loop of bihash searches in one worker (that eventually crashes), while deleting the ACLs from another one, and with larger amount of data the probability is higher that at the moment of the hash lookup the vector is still there, but at the moment of

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-15 Thread Satya Murthy
Hi Andrew, Thanks for the quick response. All the activity that we are performing with ACLs, like , add and delete of ACLs from multiple workers, we are doing by taking a lock. This will ensure that no two workers will be able to operate on ACLs at the same time. Is there any specific

Re: [vpp-dev] Crash in ACL code while applying rulematch

2020-07-15 Thread Andrew Yourtchenko
“ One more thing is this crash is happening only when we add acl rules and lookup contexts from multiple workers.” - you should not be doing that from any workers, let alone multiple. The manipulations with ACLs and lookup contexts are considered to be control plane activity, thus should be

[vpp-dev] Crash in ACL code while applying rulematch

2020-07-15 Thread Satya Murthy
Hi , We are seeing a crash when we are trying to do ACL rulematch within a lookup context. #0  0x7f7786498ca0 in multi_acl_match_get_applied_ace_index (am=0x7f77866caa00 , is_ip6=0, match=0x7f7791fff610) at src/plugins/acl/public_inlines.h:625 #1  0x7f7786498f27 in