Re: Freeradius 1.1.1: 'sqlippools' concurrency problems
Alan DeKok [EMAIL PROTECTED] wrote: A simple solution would be to go to the bottom of rlm_sqlippool.c, and change it from RLM_TYPE_THREAD_SAFE to RLM_TYPE_THREAD_UNSAFE. Then re-compile re-install. That would be the case if I only had one single radiusd running, however as already mentioned this solution encompasses a number of radius servers each with its own mysql server adn the same storage (NDB cluster) behind the scenes i.e. virtually sharing data storage. By implementing the suggestion (which by the way didn't know it was possible so thanks for the tip) I would only solve it for one radius server (at the expense of a lower performance) but not wen several radius servers are at play. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Freeradius 1.1.1: 'radclient' incorrect Authenticator
Alan DeKok [EMAIL PROTECTED] wrote:
I'm using freeradius 1.1.1 and trying to performance test it using
as client the 'radclient' utility program. I have been experiencing
some problems when load testing it since freeradius was responding
in some cases responding to the client with old (already sent)
responses when in fact the request was different (Attribute-wise at
least).
The server shouldn't do that. The request authenticators are
different, so the requests should be treated as different.
I have traced (ethereal) the requests I'm sending and only 256 are
actually unique. Thereafter they are repeated over and over throughout
the requests provided in the input file. It seems like the IDs are being
reused.
I might be mistaken but one odd thing is that I see a comment in the
code saying that if a request has already been used then it needs to
make sure that the Authenticator value is unique. However I don't
see that happening in the code actually ('radclient.c', function
'request_free'):
Look at the rest of the code. See send_one_packet(), which creates
a unique vector.
I agree there it's were the vector is set (by invoking the MD5 library)
however,
I don't see where that vector is made random for every request. Again I might
be
mistaken but the end result is what I described above, that the same request ID
gets
the exact same Authenticator value.
The reasons why in many instances I don't get a Re-send of an old reply is
simply
because it might no longer be stored in the cache hence a brand new one
generated. The
latter behavior is dependent on the number of messages allowed in the cache and
the
time before they stale. I had this time set to 5 seconds and given the
performance of
a litle over 50 req/sec it was causing it to be right in the limit:
(256req / 50req/sec ~= 5 sec).
Thanks for the help.
Regards,
Alex.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Freeradius 1.1.1: 'sqlippools' concurrency problems
Hi,
I'm performance-testing freeradius with rlm_sqlippools support
(an MySQL 5.0 Cluster) and have set up my SQLs in 'sqlipool.sqls'
file as follows (regarding IP address allocation):
-- sqlippool.sqls --
#
# This series of queries allocates an IP number
#
allocate_begin = BEGIN
allocate-clear = UPDATE radippool \
SET SGSNIPV4Address = '', \
NASIPAddress = '', \
username = '', \
calling_station_id = '', \
expiry_time = NOW() - INTERVAL 1 SECOND \
WHERE calling_station_id = '%{Calling-Station-Id}'
allocate-find = SELECT ip_address FROM radippool \
WHERE pool_name = '%P' AND expiry_time NOW() \
LIMIT 1 \
FOR UPDATE
allocate-update = UPDATE radippool \
SET calling_station_id = '%i', \
username = '%u', \
NASIPAddress = '%{NAS-IP-Address}', \
expiry_time = NOW() + INTERVAL '%J' SECOND, \
SGSNIPV4Address = '%{3GPP-SGSN-Address}' \
WHERE ip_address = '%I'
allocate_commit = COMMIT
allocate_rollback = ROLLBACK
I'm using two 'radclient' that concurrently send 1000 Access-Requests
each. All of the requests get answered positively and an IP returned.
The problem is that in many instances these IP addresses are the same
i.e. the 'allocate-find' doesn't seem to lock the returned row properly.
I have read that MySQL with Cluster Storage Engine uses 'READ COMMITED'
isolation level so I think it should be ok for my purposes (SQLs above).
However the locks don't seem to work.
Has anyone noticed this behavior with similar SQL setup on MySQL or similar
Relational Database with similar isolation level and locking mechanisms? If so
do you have any advice on how to prevent the above described problem?
Thanks!
Cheers,
Alex.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Freeradius 1.1.1: 'radclient' incorrect Authenticator value
Hi,
I'm using freeradius 1.1.1 and trying to performance test it using as client
the 'radclient' utility program. I have been experiencing some problems when
load testing it since freeradius was responding in some cases responding to the
client with old (already sent) responses when in fact the request was different
(Attribute-wise at least).
...
rad_recv: Access-Request packet from host 10.66.20.20:36426, id=235, length=110
Sending duplicate reply to client localtest:36426 - ID: 235
Re-sending Access-Accept of id 235 to 10.66.20.20 port 36426
...
After struggling a bit I noticed that the Access-Requests sent by
'radclient' where all having the same Authenticator value for the same request
ID (0-255). In some cases the cached reply was gone before the next request
with the same Authenticator value came hence a brand new reply was generated
however it became a problem otherwise (reply still in the cache).
Have any of you guys noticed this problem and do you have a solution to
it (besides changing the code)? My test files contain at least 1000
Access-Requests each therefore the IDs will be reused.
I might be mistaken but one odd thing is that I see a comment in the
code saying that if a request has already been used then it needs to make sure
that the Authenticator value is unique. However I don't see that happening in
the code actually ('radclient.c', function 'request_free'):
/*
* If we've already sent a packet, free up the old one,
* and ensure that the next packet has a unique
* authentication vector.
*/
if (radclient-request-data) {
free(radclient-request-data);
radclient-request-data = NULL;
}
Thank you!
Regards,
Alex.
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Solaris 8/SPARC - MySQL 5.0 NDB Cluster - Freeradius 1.1.1 withrlm_sqlippool module: 'radiusd' segmentation fault
Hi,
The purpose of this mail is to give you an insight on some stuff I've been
trying at work (playing some could argue) that I'd like to share in case it
could be useful to any of you out there.
I won't describe all the issues I've had during the compilation,
configuration and functional/performance testing nor ask you for help but
rather just describe what I've done and document one of the last problems I had
which kept me awake a few nights (segmentation fault).
I have for the past 4 weeks been trying to evaluate if FreeRadius can
be used as a AAA in an UMTS network with a large amount of subscribers for the
GPRS Data services. With if it can be used I mean essentially if it can
handle:
(1) Functionality: basic Authentication/Authorization/Accounting, IP Address
allocation and some GPRS attribute to IP Address mapping storage.
(2) High Availability (no single point of failure HW/SW)
(3) Distributed Architecture (performance target of 250 requests/second peak
hour at a reasonable HW/SW cost)
For the purpose of this test I have decided to use (32 bit due to
problems getting it to compile with 64 bit on SPARC with the distributed
binaries from MySQL):
(a) Solaris 8 on SPARC (selected due to the fact that these machines were
pretty much idle at my company similar tests were run on x86 PCs based on
Fedora Linux Core 4).
(b) MySQL 5.0.21 (MAX version) 32 bit SPARC binary distribution.
(c) Freeradius 1.1.1 (originally with 1.1.0 but due to bugs on the Dictionary
and thanks to recommendation (mail archives) from Alan DeKok I upgraded.
(d) For IP allocation I'm using the rlm_sqlippool module (hard to tell its
version because it's not version controlled as far as I could see, I got it
from a Russian website) as per Alan DeKok's recommendation (mail archives). It
will require some customization as I'm looking into being able to define IP
pools as being comprised of several (not just one) start/end IP ranges.
The test bed is basically two physical nodes each running the same
software i.e. radiusd, mysqld and ndbd (MySQL clustered storage engine
process). The NAS (in UMTS these are called GGSN) will load-balance the
requests (directly or through an IP Load Balancer or even a freeradius proxy
haven't decided yet which).
This configuration allows vertical (bigger machines) and horizontal
(more machines) scalability by adding more CPU:s or extra nodes to the cluster
respectively for improved performance. I have tested the vertical scalability
and it's linear with the CPU utilization. The horizontal will be tested in the
coming days (hard to get hold of the required HW for the tests). I will publish
some results (more quantitative than this email) then.
Last but not least (and in connection to the subject of this email) one
bug I found on the rlm_sqlippool that I have (as I mentioned hard to tell its
version) is that during load testing and given the right circumstances
(multiple NAS, Solaris architecture, MySQL Cluster storage engine only and high
CPU utilization) I was getting a core dump of the 'radiusd' process.
The problem was during the post-authorization phase of the sqlippool
module on the 'allocate-find' SQL statement result retrieval due to the fact
that the expected result row (just one expected with just one field containing
the IP address to allocate) had invalid memory references (a row is modelled as
an array of references to result columns and the only reference was invalid and
therefore causing a segmentation fault to happen).
Looking at the code and debugging it for a while I noticed that the
memory holding the result set was being released before it was being used
(though previously a reference to the first and only row had been kept) hence
causing unpredictable results.
Anyhow the code changes to fix this was to simply move the
'sql_finish_select_query' function call (which indirectly calls the MySQL
function 'mysql_free_result' to release memory allocated to the result set) a
few lines down the 'sqlippool_query1' function which is the one retrieving the
IP Address to be allocated in 'rlm_sqlippool.c' file. See below for details:
1 /*
2* Query the database expecting a single result row
3*/
4 static int sqlippool_query1(char * out, int outlen, const char * fmt,
SQLSOCK * sqlsocket, void * instance, REQU
5 EST * request, char * param, int param_len)
6 {
7 rlm_sqlippool_t * data = (rlm_sqlippool_t *) instance;
8 char expansion[MAX_STRING_LEN * 4];
9 char query[MAX_STRING_LEN * 4];
10 SQL_ROW row;
11 int r;
12
13 sqlippool_expand(expansion, sizeof(expansion), fmt, instance,
param, param_len);
14
15 /*
16 * Do an xlat on the provided string
17 */
18 if (request) {
19
Re: Solaris 8/SPARC - MySQL 5.0 NDB Cluster - Freeradius 1.1.1 with rlm_sqlippool module: 'radiusd' segmentation fault
Title: Re: Solaris 8/SPARC - MySQL 5.0 NDB Cluster - Freeradius 1.1.1 with rlm_sqlippool module: 'radiusd' segmentation fault Alan DeKok [EMAIL PROTECTED] wrote: I have for the past 4 weeks been trying to evaluate if FreeRadius can be used as a AAA in an UMTS network with a large amount of subscribers for the GPRS Data services. I believe others are doing this today. FreeRADIUS scales very well. 10 million subscribers shouldn't be a problem in a carefully designed system. Well, I'm wondering if this is in a clustered configuration i.e. multiple nodes handling the load and cooperating (sharing data such as IP pools). If this is the case I'd be interested to know a few accounts to find out more about their architecture (behind freeradius such as database servers, etc). Any clues? I'd really like o create some sort of standard architecture for freeradius that can scale and is reliable and have it in a real environment for a while for others to have confidence when making this same decision that I'm about to make. I'm done for now more details will come later meanwhile I have a question: is the rlm_sqlippool module going to be part of a freeradius release in the near future and if not, what would it be the procedure to follow for it to happen? An uopdated version of the module is in CVS head. If there's interest, it could be back-ported to 1.1.x. I have compared my version with that of the CVS root and apart form some small differences that I'll investigate further I noticed that it has the same bug that I found. The problem is that I don't know how to report it. Can you help? Also I think it'd be a good idea to back-port it to 1.1.x. Who decides this and how do I express my interest? Thank you! Cheers, Alejandro Robles. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
