Bug#603445: libapache2-mod-auth-mysql: The module crashes when the web service, is under heavy concurrent load

2011-04-14 Thread Rotem Ringel

Hi Sumit,

Yes. I am using the same patch in my production system.

Regards,
Rotem.

Sumit Madan wrote:

Hello Rotem,

I have the same problem here. I have build this package with your 
patch and installed it successfully.


I have one important question: do you think your patch is well suited 
for a productive system?


Are you using the same patch for your production system?






--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org



Bug#603445: libapache2-mod-auth-mysql: The module crashes when the web service is under heavy concurrent load

2010-11-13 Thread Rotem Ringel


Package: libapache2-mod-auth-mysql
Version: 4.3.9-13+b1
Severity: normal

I wrote a web service that is configured with AuthMySQL.
I used Jmeter to simulate a heavy load on the web service by sending about 50
requests simultaneously. The module crashes after responding to some hundreds
requests. See attached backtrace taken from
/var/log/apache2/error.log.

I am using the package apache2-mpm-worker and not apache2-mpm-prefork.
It seems like two threads tries to send a query to the MySQL server at
the same time on the same connection.
Suggested solution: Add a mutex lock around the pair of mysql_query() and
mysql_store_result() calls. Once mysql_store_result() is ready, the lock is
released and other threads may query the same connection.
See attached patch.

Packages information:
libapache2-mod-auth-mysql4.3.9.1-13
apache2   2.2.9-10+lenny8
apache2-mpm-worker  2.2.9-10+lenny8
libmysqlclient15-dev 5.0.51a-24+lenny4



-- System Information:
Debian Release: squeeze/sid
 APT prefers testing
 APT policy: (500, 'testing')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-2-686 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages libapache2-mod-auth-mysql depends on:
ii  apache2.2-common  2.2.16-3   Apache HTTP Server common files
ii  libc6 2.10.2-2   GNU C Library: Shared libraries
ii  libmysqlclient16  5.1.37-2   MySQL database client library

libapache2-mod-auth-mysql recommends no packages.

libapache2-mod-auth-mysql suggests no packages.

-- no debconf information


*** mod-auth-mysql-4.3.9/mod_auth_mysql.c	2004-12-23 15:43:14.0 +0200
--- mod_auth_mysql.c	2010-11-11 12:58:21.0 +0200
***
*** 55,60 
--- 55,64 
  #include 
  #endif
  
+ #if APR_HAS_THREADS
+ #include 
+ #endif
+ 
  #include 
  #include 
  #include 
***
*** 287,292 
--- 291,300 
  	
  	MYSQL *dbh;
  
+ #if APR_HAS_THREADS	
+ 	apr_thread_mutex_t *lock;   /* Lock for this config */
+ #endif
+ 
  	/* Boolean options */
  	unsigned char persistent;
  	unsigned char enable_mysql_auth;
***
*** 393,398 
--- 401,410 
  	ap_register_cleanup(p, sec, auth_mysql_cleanup, ap_null_cleanup);
  #endif
  
+ #if APR_HAS_THREADS	
+ 	apr_thread_mutex_create(&sec->lock, APR_THREAD_MUTEX_DEFAULT, p);
+ #endif
+ 
  	sec->dir = d;
  	
  	sec->user_table = sec->group_table = NULL;
***
*** 1347,1352 
--- 1359,1368 
  		return -1;
  	}
  
+ #if APR_HAS_THREADS
+ apr_thread_mutex_lock(sec->lock);
+ #endif
+ 
  	if ((rv = safe_mysql_query(r, query, sec))) {
  		if (sec->dbh)
  		{
***
*** 1355,1369 
  		}
  
  		APACHELOG(APLOG_DEBUG, r, "Failed query was: [%s]", query);
! 		return -1;
  	}
  
  	result = safe_mysql_store_result(r->pool, sec);
  	if (!result) {
  		APACHELOG(APLOG_ERR, r,
  			"Failed to get MySQL result structure : %s", mysql_error(sec->dbh));
! 		return -1;
  	}
  	switch (mysql_num_rows(result)) {
  		case 0:
  			APACHELOG(APLOG_INFO, r, "User not found");
--- 1371,1390 
  		}
  
  		APACHELOG(APLOG_DEBUG, r, "Failed query was: [%s]", query);
! 		goto Error;
  	}
  
  	result = safe_mysql_store_result(r->pool, sec);
  	if (!result) {
  		APACHELOG(APLOG_ERR, r,
  			"Failed to get MySQL result structure : %s", mysql_error(sec->dbh));
! 		goto Error;
  	}
+ 
+ #if APR_HAS_THREADS	
+ apr_thread_mutex_unlock(sec->lock);
+ #endif
+ 
  	switch (mysql_num_rows(result)) {
  		case 0:
  			APACHELOG(APLOG_INFO, r, "User not found");
***
*** 1396,1401 
--- 1417,1428 
  
  	APACHELOG(APLOG_CRIT, r, "Can't happen - dropped out of switch!");
  	return -1;
+ 
+ Error:
+ #if APR_HAS_THREADS	
+ apr_thread_mutex_unlock(sec->lock);
+ #endif
+ 	return -1;
  }
  
  /* Has a look to see if the given user is a member of the named group.

*** glibc detected *** /usr/sbin/apache2: double free or corruption (!prev): 0x08270fb0 ***
=== Backtrace: =
/lib/i686/cmov/libc.so.6[0xb7639764]
/lib/i686/cmov/libc.so.6(cfree+0x96)[0xb763b966]
/usr/lib/libmysqlclient_r.so.15(my_no_flags_free+0x21)[0xb7285e01]
/usr/lib/libmysqlclient_r.so.15(vio_delete+0x30)[0xb72b2090]
/usr/lib/libmysqlclient_r.so.15(end_server+0x35)[0xb72add95]
/usr/lib/libmysqlclient_r.so.15(cli_safe_read+0xd0)[0xb72adfb0]
/usr/lib/libmysqlclient_r.so.15[0xb72ae5c5]
/usr/lib/libmysqlclient_r.so.15(mysql_real_query+0x44)[0xb72aca64]
/usr/lib/libmysqlclient_r.so.15(mysql_query+0x35)[0xb7283655]
/usr/lib/apache2/modules/mod_auth_mysql.so[0xb6d38711]
/usr/lib/apache2/modules/mod_auth_mysql.so[0xb6d38c11]
/usr/lib/apache2/modules/mod_auth_mysql.so(mysql_authenticate_basic_user+0x1eb)[0xb6d39455]
/usr/sbin/apache2(ap_run_check_user_id+0x59)[0x80756a9]
/usr/sbin/apache2(ap_process_request_internal+0x2b0)[0x8077810]
/usr/sbin/apache2(ap_process_request+0x180)