Re: FreeRadius Losing packets in Multi-Threads mode
Hello, On Tue, 25 Oct 2011 21:09:31 +0100, Alan Buxey a.l.m.bu...@lboro.ac.uk however, as Alan said. in single thread mode, you only have one process dealing with requests.so one single open connection to SQL, one single sesion to LDAP etc etc (whatever you use) - eg even a local file with PERL. with multithread mode, you have many threads - all of which can be hitting your SQL or LDAP at same time...or trying to write to the same file in a PERL moduleso you have to look at the speed/ability of your backendthe jump from a single query ata time to concurrent queries may have tipped your balance alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html Here is what I had thought about: FreeRadius correctly treats the requests (answer are always received), but there is a locking problem with the log files, meaning that basically, only one thread can write inside. Is there any solution about that? As stated before, there is no overloading problem on the server. Running in single thread could be a solution, but it's really slow compared to multi-thread ... Regards, Pierre - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Did the very same test here, with very same results. I find this a little bit scary to imagine that some accounting packets are lost (meaning I have no proof the requests was answered and how. regards Le 26/10/2011 10:21, Pierre Rondou a écrit : Hello, On Tue, 25 Oct 2011 21:09:31 +0100, Alan Buxeya.l.m.bu...@lboro.ac.uk however, as Alan said. in single thread mode, you only have one process dealing with requests.so one single open connection to SQL, one single sesion to LDAP etc etc (whatever you use) - eg even a local file with PERL. with multithread mode, you have many threads - all of which can be hitting your SQL or LDAP at same time...or trying to write to the same file in a PERL moduleso you have to look at the speed/ability of your backendthe jump from a single query ata time to concurrent queries may have tipped your balance alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html Here is what I had thought about: FreeRadius correctly treats the requests (answer are always received), but there is a locking problem with the log files, meaning that basically, only one thread can write inside. Is there any solution about that? As stated before, there is no overloading problem on the server. Running in single thread could be a solution, but it's really slow compared to multi-thread ... Regards, Pierre - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html -- http://www.horoa.net Alexandre Chapellon Ingénierie des systèmes open sources et réseaux. Follow me on twitter: @alxgomz http://www.twitter.com/alxgomz - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Pierre Rondou wrote: Here is what I had thought about: FreeRadius correctly treats the requests (answer are always received), but there is a locking problem with the log files, meaning that basically, only one thread can write inside. Edit raddb/detail, and add locking = yes Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Alexandre Chapellon wrote: Did the very same test here, with very same results. I find this a little bit scary to imagine that some accounting packets are lost (meaning I have no proof the requests was answered and how. Ah... after thinking about it some more, there is no problem. The test is wrong. Accounting packets have *nothing* which makes them unique, other than the packet contents. If you're sending the same packet over and over, it's a duplicate, and gets a duplicate response. In single threaded mode, the server receives a packet, sends a reply, and *deletes* all knowledge about the packet. When it receives the next one, it gets processed as if it was a new packet. In multi-threaded mode, the server receives a packet, hands it to a thread, which processes it and sends a reply. Crucially, the packet isn't deleted until later, because the server avoids thread locks. While the packet is cached, the main thread receives a new packet, which looks *exactly* the same as a packet in the cache. So... the main thread re-sends the same reply. If you look at the statistics from the server, you'll see the duplicate request counter being very high. It SHOULD be zero for accounting packets. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Pierre Rondou wrote: The problem is that the server seems to lose accounting packets in Multithread mode It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them. Logging with module detail or linelog logs only a very little amount (0.5%) of the transmitted packets. The client receive all the answer, but my problem is that I rely on those logs for an other program. It should handle a lot more than .5% of the packets. It does on my system. This only happen when using FreeRadius in Multithread mode, while using it in single-thread (freeradius -s), all the packets are logged correctly. Anyone has a hint on this problem? What *else* is the server doing? Databases? Exec'd programs? Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Hello Alan, See answers inline Regards, Pierre Alan DeKok a écrit : Pierre Rondou wrote: The problem is that the server seems to lose accounting packets in Multithread mode It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them. Well, then, why is this only happening in the multi-thread mode? If it was a kernel issue, wouldn't it drop the packets in the same way for the single-thread version? Logging with module detail or linelog logs only a very little amount (0.5%) of the transmitted packets. The client receive all the answer, but my problem is that I rely on those logs for an other program. It should handle a lot more than .5% of the packets. It does on my system. This only happen when using FreeRadius in Multithread mode, while using it in single-thread (freeradius -s), all the packets are logged correctly. Anyone has a hint on this problem? What *else* is the server doing? Databases? Exec'd programs? Actually, it's a test server (HP Proliant 360 with 16 Gb ram and quad-core proc), so nothing is really running on it. MongoDB is installed (but nothing is using it), monit and ssh are installed as well, and that's it. The debian was installed with basically nothing. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Pierre Rondou wrote: It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them. Well, then, why is this only happening in the multi-thread mode? If it was a kernel issue, wouldn't it drop the packets in the same way for the single-thread version? No. I mean that the threading code has overhead which doesn't exist in the non-threaded mode. Lock contention is a *major* source of lost CPU time in threaded applications. If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not. Actually, it's a test server (HP Proliant 360 with 16 Gb ram and quad-core proc), so nothing is really running on it. MongoDB is installed (but nothing is using it), monit and ssh are installed as well, and that's it. The debian was installed with basically nothing. Well... that doesn't sound right. Accounting works for me... Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Alan DeKok al...@deployingradius.com writes: Pierre Rondou wrote: It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them. Well, then, why is this only happening in the multi-thread mode? If it was a kernel issue, wouldn't it drop the packets in the same way for the single-thread version? No. I mean that the threading code has overhead which doesn't exist in the non-threaded mode. Lock contention is a *major* source of lost CPU time in threaded applications. If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not. But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged. File locking problem when multiple threads are attempting to update the same file? Bjørn - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Bjørn Mork wrote: File locking problem when multiple threads are attempting to update the same file? Maybe... the detail file is marked not thread-safe, which means that the server core takes care of locking it. i.e. only one thread at a time will be inside of the detail module. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
On 26 October 2011 04:48, Bjørn Mork bj...@mork.no wrote: If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not. But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged. Maybe my whole understanding of UDP is flawed, I thought the whole point of using UDP instead of TCP is not to have ACKS and retransmits? Cheers, Andrej - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Andrej andrej.gro...@gmail.com writes: On 26 October 2011 04:48, Bjørn Mork bj...@mork.no wrote: If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not. But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged. Maybe my whole understanding of UDP is flawed, I thought the whole point of using UDP instead of TCP is not to have ACKS and retransmits? Not nessarily. The point is that UDP leaves that up to the application. For RADIUS accounting this is described in http://tools.ietf.org/html/rfc2866#section-2 quote It is recommended that the client continue attempting to send the Accounting-Request packet until it receives an acknowledgement, using some form of backoff. If no response is returned within a length of time, the request is re- sent a number of times. The client can also forward requests to an alternate server or servers in the event that the primary server is down or unreachable. An alternate server can be used either after a number of tries to the primary server fail, or in a round-robin fashion. Retry and fallback algorithms are the topic of current research and are not specified in detail in this document. /quote I don't know if all clients follow this recommendation, but a lot of them do. Bjørn - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: FreeRadius Losing packets in Multi-Threads mode
Hi, But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged. Maybe my whole understanding of UDP is flawed, I thought the whole point of using UDP instead of TCP is not to have ACKS and retransmits? at the service level - the software has do deal with problems instead of the TCPIP stack. however, as Alan said. in single thread mode, you only have one process dealing with requests.so one single open connection to SQL, one single sesion to LDAP etc etc (whatever you use) - eg even a local file with PERL. with multithread mode, you have many threads - all of which can be hitting your SQL or LDAP at same time...or trying to write to the same file in a PERL moduleso you have to look at the speed/ability of your backendthe jump from a single query ata time to concurrent queries may have tipped your balance alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
