Luis, I wonder, how many CPU cores/available hardware threads (taking into account HyperThreading and all that—so just the number of CPUs in /proc/cpuinfo) are available here? It almost sounds like something which would occur with maybe 1 or 2 CPUs being contended over. Perhaps the simplest thing for this kind of call volume is to run Kamailio on a host with 8 or 12+ CPU cores?
— Alex — Sent from my iPad > On Apr 10, 2020, at 2:20 AM, Luis Rojas G. <luis.ro...@sixbell.com> wrote: > > > Hello, > > I have a lot of experience developing mutithreaded applications, and I don't > see it so unlikely at all that a process loses cpu just after recvfrom(). > It's just as probable as to lose it just before, or when writing on a cache > or just before of after sendto(). If there are many messages going through, > some of them will fall in this scenario. if I try sending a burst of 100 > messages, I see two or three presenting the scenario. > > Just forward() with a single process does not give the capacity. I'm getting > almost 1000caps. More than that and start getting errores, retransmissions, > etc. And this is just one way. I need to receive the call to go back to the > network (our application is a B2BUA), so I will be down to 500caps, with a > simple scenario, with no reliable responses, reinvites, updates, etc. I will > end up having as many standalone kamailio processes as the current servers I > do have now. > > I really think the simplest way would be to add a small delay to 200 OK. Very > small, like 10ms, should be enough. Simple and it should work. As Alex > Balashov commented he did for the case with ACK-Re-Invite. > > I have to figure out how to make async_ms_sleep() work in reply_route(). > > Thanks for all the comments and ideas > > Best regards, > > Luis > > > > . On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote: >> >> >> mico...@gmail.com appears similar to someone who previously sent you email, >> but may not be that person. Learn why this could be a risk >> Feedback >> Hello, >> >> then the overtaking is in between reading from the socket and getting to >> parsing the call-id value -- the cpu is lost by first reader after >> recvfrom() and the second process get enough cpu time to go ahead further. I >> haven't encountered this case, but as I said previously, it is very >> unlikely, but still possible. I added the route_locks_size because in the >> past I had cases when processing of some messages took longer executing >> config (e.g., due to authentication, accounting, ..) and I needed to be sure >> they are processed in the order they enter config execution. >> >> Then the option is to see if a single process with stateless sending out >> (using forward()) gives the capacity, if you don't do any other complex >> processing. Or if you do more complex processing, use a dispatcher process >> with forwarding to local host or in a similar manner try to use >> mqueue+rtimer for dispatching using shared memory queues. >> >> Of course, it is open source and there is also the C coding way, to add a >> synchronizing mechanism to protect against parallel execution of the code >> from recvfrom() till call-id lock is acquired. >> >> Cheers, >> Daniel >> >> On 09.04.20 17:37, Luis Rojas G. wrote: >>> Hello, >>> >>> Well, it did not work at all. Exactly same behavior, with random out of >>> order messages. >>> >>> Best regards, >>> >>> Luis >>> >>> On 4/9/20 11:28 AM, Daniel-Constantin Mierla wrote: >>>> >>>> >>>> mico...@gmail.com appears similar to someone who previously sent you >>>> email, but may not be that person. Learn why this could be a risk >>>> Feedback >>>> Hello, >>>> >>>> the sip messages belonging to the same dialog have the same value for >>>> Call-Id header. The locking is done based on hashing the Call-Id, so it >>>> doesn't need at all the dialog module for its purpose. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 09.04.20 14:19, Luis Rojas G. wrote: >>>>> Hello, Daniel, >>>>> >>>>> I am not so sure. I first tried adding that parameter, but it did not >>>>> work at all. Same behavior. Then I read the documentation more carefully >>>>> : >>>>> >>>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size >>>>> route_locks_size >>>>> >>>>> Set the number of mutex locks to be used for synchronizing the execution >>>>> of messages sharing the same Call-Id. In other words, enables Kamailio to >>>>> execute the config script sequentially for the requests and replies >>>>> received within the same dialog – a new message received within the same >>>>> dialog waits until the previous one is routed out. >>>>> >>>>> Locks to execute sequentially messages belonging to same dialog. How will >>>>> Kamailio be aware that messages belong to same dialog, without the dialog >>>>> module?. With just stateless proxy it has no idea about dialogs, it just >>>>> forwards messages. I guess that's why just adding that parameter did not >>>>> work. >>>>> >>>>> Am I wrong? >>>>> >>>>> Luis >>>>> >>>>> >>>>> >>>>> >>>>> On 4/9/20 3:47 AM, Daniel-Constantin Mierla wrote: >>>>>> Hello, >>>>>> >>>>>> On 08.04.20 23:03, Luis Rojas G. wrote: >>>>>>> Hello, Daniel, >>>>>>> >>>>>>> I looked into that parameter, but I need to use with the dialog >>>>>>> module, and I'm pretty afraid to use that. >>>>>> who said or where is written than you need to load the dialog module? >>>>>> You definitely don't. >>>>>> >>>>>> Cheers, >>>>>> Daniel >>>>>> >>>>>> >>>>>> >>>>>>> I was looking more into the stateless proxy, because I need to process >>>>>>> a lot of traffic. >>>>>>> >>>>>>> My target is 4200CAPS. with duration between 90s and 210. Let's say, >>>>>>> 150 seconds. That would mean 630.000 simultaneous dialogs. I don't >>>>>>> think the solution can go that way. >>>>>>> >>>>>>> it would really help me to be able to use completely stateless proxy >>>>>>> plus Async in reply_route(), to introduce an artificial delay before >>>>>>> forwarding 200 OK to Invite.. As someone mentioned, it would help me on >>>>>>> request_route(), for race conditions between ACK and Re-Invite. >>>>>>> >>>>>>> Any idea why Async is not allowed in reply_route()? >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Luis >>>>>>> >>>>>>> >>>>>>> On 4/8/20 1:07 PM, Daniel-Constantin Mierla wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> you have to keep in mind that Kamailio is a SIP packet router, not a >>>>>>>> telephony engine. If 180 and 200 replies are part of a call is not >>>>>>>> something that Kamailio recognize at its core. Its main goal is to >>>>>>>> route out as fast as possible what is received, by executing the >>>>>>>> configuration file script. Now, a matter of your configuration file, >>>>>>>> processing of some SIP messages can take longer than processing other. >>>>>>>> And the processing is done in parallel, a matter of children parameter >>>>>>>> (and tcp_children, sctp_children). >>>>>>>> >>>>>>>> With that in mind, a way to try to cope better with the issue you face >>>>>>>> is to set route_locks_size parameter, see: >>>>>>>> >>>>>>>> * https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size >>>>>>>> >>>>>>>> Probably is what you look for. >>>>>>>> >>>>>>>> But if you want more tight constraints, like when receiving a 180 >>>>>>>> after a 200ok and not route it out, you have to make the logic in >>>>>>>> configuration file by combining modules such as dialog or htable (as >>>>>>>> already suggested). >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Daniel >>>>>>>> >>>>>>>> On 08.04.20 16:04, Luis Rojas G. wrote: >>>>>>>>> Hi, Henning, >>>>>>>>> >>>>>>>>> No need to be ironic. As I mentioned on my first post, I tried >>>>>>>>> stateful proxy and I observed the same behavior. >>>>>>>>> >>>>>>>>> "I tried using stateful proxy and I obtained the same result." >>>>>>>>> >>>>>>>>> The asynchronous sleep seems promising. I will look into it. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Luis >>>>>>>>> >>>>>>>>> >>>>>>>>> On 4/8/20 9:30 AM, Henning Westerholt wrote: >>>>>>>>>> Hi Luis, >>>>>>>>>> >>>>>>>>>> I see. Well, you want to use Kamailio as a stateless proxy, on the >>>>>>>>>> other hand it should do things that are inherently stateful. 😉 >>>>>>>>>> >>>>>>>>>> As mentioned, have a look to the dialog module to track the state of >>>>>>>>>> dialogs that you process. This will not work in a stateless mode, >>>>>>>>>> though. >>>>>>>>>> >>>>>>>>>> You can also use the htable module to just store some data about the >>>>>>>>>> processed messages in a shared memory table and use this to enforce >>>>>>>>>> your ordering. There is also the option to do an asynchronous sleep >>>>>>>>>> (with the async) module on the message that you want to delay but >>>>>>>>>> still processing other messages during it. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> Henning >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Henning Westerholt – https://skalatan.de/blog/ >>>>>>>>>> Kamailio services – https://gilawa.com >>>>>>>>>> >>>>>>>>>> From: Luis Rojas G. <luis.ro...@sixbell.com> >>>>>>>>>> Sent: Wednesday, April 8, 2020 3:00 PM >>>>>>>>>> To: Henning Westerholt <h...@skalatan.de>; Kamailio (SER) - Users >>>>>>>>>> Mailing List <sr-users@lists.kamailio.org> >>>>>>>>>> Subject: Re: [SR-Users] Kamailio propagates 180 and 200 OK OUT OF >>>>>>>>>> ORDER >>>>>>>>>> >>>>>>>>>> Hello, Henning, >>>>>>>>>> >>>>>>>>>> I am worried about this scenario, because it's a symptom of what may >>>>>>>>>> happen in other cases. For instance, I've seen that this operator >>>>>>>>>> usually sends re-invites immediate after sending ACK. This may >>>>>>>>>> create race conditions like 3.1.5 of RFC5407 >>>>>>>>>> >>>>>>>>>> https://tools.ietf.org/html/rfc5407#page-22 >>>>>>>>>> >>>>>>>>>> I'd understand that one happens because of packet loss, as it's in >>>>>>>>>> UDP's nature, but in this case it would be artificially created by >>>>>>>>>> Kamailio. if there was no problem at network level (packet loss, >>>>>>>>>> packets following different path on the network and arriving out of >>>>>>>>>> order), why Kamailio creates it? >>>>>>>>>> >>>>>>>>>> I'd expect that the shared memory is used precisely for this. If an >>>>>>>>>> instance of kamailio receives a 200 OK, it could check on the shm >>>>>>>>>> and say "hey, another instance is processing a 180 for this call. >>>>>>>>>> Let's wait for it to finish" (*). I know there could still be a >>>>>>>>>> problem, the instance processing the 180 undergoes a context switch >>>>>>>>>> just after it receives the message, but before writing to shm, but >>>>>>>>>> it would greatly reduce the chance. >>>>>>>>>> >>>>>>>>>> In our applications we use a SIP stack that always sends messages to >>>>>>>>>> the application in the same order it receives them, even though is >>>>>>>>>> multi-threaded and messages from the network are received by >>>>>>>>>> different threads. So, they really syncronize between them. Why >>>>>>>>>> Kamailio instances don't? >>>>>>>>>> >>>>>>>>>> I am evaluating kamailio to use it as a dispatcher to balance load >>>>>>>>>> against our several Application Servers, to present to the operator >>>>>>>>>> just a couple of entrance points to our platform (they don't want to >>>>>>>>>> establish connections to each one of our servers). This operator is >>>>>>>>>> very difficult to deal with. I am sure they will complain something >>>>>>>>>> like "why are you sending messages out of order? Fix that". The >>>>>>>>>> operator will be able to see traces and check that messages entered >>>>>>>>>> the Kamailio nodes in order and left out of order. They will not >>>>>>>>>> accept it. >>>>>>>>>> >>>>>>>>>> (*) Not really "wait", as it would introduce a delay in processing >>>>>>>>>> all messages. it should be like putting it on a queue, continue >>>>>>>>>> processing other messages, and go back to the queue later. >>>>>>>>>> >>>>>>>>>> Well, thanks for your answer. >>>>>>>>>> >>>>>>>>>> Luis >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 4/8/20 3:01 AM, Henning Westerholt wrote: >>>>>>>>>> Hello Luis, >>>>>>>>>> >>>>>>>>>> as the 1xx responses are usually send unreliable (unless you use >>>>>>>>>> PRACK), you should not make any assumption on the order or even the >>>>>>>>>> arrival of this messages. It can also happens on a network level, if >>>>>>>>>> send by UDP. >>>>>>>>>> >>>>>>>>>> Can you elaborate why you think this re-ordering is a problem for >>>>>>>>>> you? >>>>>>>>>> >>>>>>>>>> One idea to enforce some ordering would be to use the dialog module >>>>>>>>>> in combination with reply routes and the textops(x) module. >>>>>>>>>> >>>>>>>>>> About the shared memory question – Kamailio implement its own memory >>>>>>>>>> manager (private memory and shared memory pool). >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> Henning >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Henning Westerholt – https://skalatan.de/blog/ >>>>>>>>>> Kamailio services – https://gilawa.com >>>>>>>>>> >>>>>>>>>> From: sr-users <sr-users-boun...@lists.kamailio.org> On Behalf Of >>>>>>>>>> Luis Rojas G. >>>>>>>>>> Sent: Tuesday, April 7, 2020 10:43 PM >>>>>>>>>> To: sr-users@lists.kamailio.org >>>>>>>>>> Subject: [SR-Users] Kamailio propagates 180 and 200 OK OUT OF ORDER >>>>>>>>>> >>>>>>>>>> Good day, >>>>>>>>>> >>>>>>>>>> I am testing the dispatcher module, using Kamailio as stateless >>>>>>>>>> proxy. I have a pool of UAC (scripts in SIPP) and a pool of UAS >>>>>>>>>> (also scripts in SIPP) for the destinations. Kamailio version is >>>>>>>>>> kamailio-5.3.3-4.1.x86_64. >>>>>>>>>> >>>>>>>>>> Problem I have is, if UAS responds 180 and 200 OK to Invite >>>>>>>>>> immediately, sometimes they are propagated out of order. 200 OK >>>>>>>>>> before 180, like this : >>>>>>>>>> >>>>>>>>>> <image001.png> >>>>>>>>>> >>>>>>>>>> UAS is 172.30.4.195:5061. UAC is 172.30.4.195:5080. Kamailio is >>>>>>>>>> 192.168.253.4:5070 >>>>>>>>>> >>>>>>>>>> Difference between 180 and 200 is just about 50 microseconds. >>>>>>>>>> >>>>>>>>>> My guess is that both messages are received by different instances >>>>>>>>>> of Kamailio, and then because of context switches, even though the >>>>>>>>>> 180 is received before, that process ends after the processing of >>>>>>>>>> 200. However, I had the idea that in order to avoid these problems >>>>>>>>>> the kamailio processes synchronized with each other using a shared >>>>>>>>>> memory. I tried using stateful proxy and I obtained the same result. >>>>>>>>>> >>>>>>>>>> By the way, anyone has any idea about how Kamailio's share memory is >>>>>>>>>> implemented? It clearly does not use the typical system calls >>>>>>>>>> shmget(), shmat(), because they are not shown by ipcs command. >>>>>>>>>> >>>>>>>>>> Before posting here I googled, but I couldn't find anything related >>>>>>>>>> to this. I can't believe I am the only one who ever had this >>>>>>>>>> problem, so I guess I am doing something wrong... >>>>>>>>>> >>>>>>>>>> Please, any help. I'm really stuck on this. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Luis Rojas >>>>>>>>>> Software Architect >>>>>>>>>> Sixbell >>>>>>>>>> Los Leones 1200 >>>>>>>>>> Providencia >>>>>>>>>> Santiago, Chile >>>>>>>>>> Phone: (+56-2) 22001288 >>>>>>>>>> mailto:luis.ro...@sixbell.com >>>>>>>>>> http://www.sixbell.com >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Luis Rojas >>>>>>>>> Software Architect >>>>>>>>> Sixbell >>>>>>>>> Los Leones 1200 >>>>>>>>> Providencia >>>>>>>>> Santiago, Chile >>>>>>>>> Phone: (+56-2) 22001288 >>>>>>>>> mailto:luis.ro...@sixbell.com >>>>>>>>> http://www.sixbell.com >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Kamailio (SER) - Users Mailing List >>>>>>>>> sr-users@lists.kamailio.org >>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>>>> -- >>>>>>>> Daniel-Constantin Mierla -- www.asipto.com >>>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>>>>>> >>>>>>> -- >>>>>>> Luis Rojas >>>>>>> Software Architect >>>>>>> Sixbell >>>>>>> Los Leones 1200 >>>>>>> Providencia >>>>>>> Santiago, Chile >>>>>>> Phone: (+56-2) 22001288 >>>>>>> mailto:luis.ro...@sixbell.com >>>>>>> http://www.sixbell.com >>>>>> -- >>>>>> Daniel-Constantin Mierla -- www.asipto.com >>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>>>> >>>>> -- >>>>> Luis Rojas >>>>> Software Architect >>>>> Sixbell >>>>> Los Leones 1200 >>>>> Providencia >>>>> Santiago, Chile >>>>> Phone: (+56-2) 22001288 >>>>> mailto:luis.ro...@sixbell.com >>>>> http://www.sixbell.com >>>> -- >>>> Daniel-Constantin Mierla -- www.asipto.com >>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>> >>> -- >>> Luis Rojas >>> Software Architect >>> Sixbell >>> Los Leones 1200 >>> Providencia >>> Santiago, Chile >>> Phone: (+56-2) 22001288 >>> mailto:luis.ro...@sixbell.com >>> http://www.sixbell.com >> -- >> Daniel-Constantin Mierla -- www.asipto.com >> www.twitter.com/miconda -- www.linkedin.com/in/miconda > > -- > Luis Rojas > Software Architect > Sixbell > Los Leones 1200 > Providencia > Santiago, Chile > Phone: (+56-2) 22001288 > mailto:luis.ro...@sixbell.com > http://www.sixbell.com > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
