HTTP retries and queue delay

[Juan Fernández]

Hi I'm running a Kannel server that since a couple of days is getting a big volume of traffic (around 20K daily). I use it with an HTTP service to route all incoming messages to a PHP application with a mysql database and so. This morning, the PHP server got overloaded and could not process about 2500 requests in an hour, so those requests got queued on the smsbox http queue. When the server lowered its stress, kannel started retrying all of that queued requests. This is an extract of the smsbox log, showing that retries are being performed: 2005-07-07 10:27:09 [18321] [6] DEBUG: HTTP: Queue contains 2538 outstanding requests 2005-07-07 10:27:39 [18321] [6] DEBUG: HTTP: Queue contains 2537 outstanding requests 2005-07-07 10:28:09 [18321] [6] DEBUG: HTTP: Queue contains 2536 outstanding requests 2005-07-07 10:28:39 [18321] [6] DEBUG: HTTP: Queue contains 2535 outstanding requests 2005-07-07 10:29:09 [18321] [6] DEBUG: HTTP: Queue contains 2534 outstanding requests 2005-07-07 10:29:39 [18321] [6] DEBUG: HTTP: Queue contains 2533 outstanding requests 2005-07-07 10:30:09 [18321] [6] DEBUG: HTTP: Queue contains 2532 outstanding requests 2005-07-07 10:30:39 [18321] [6] DEBUG: HTTP: Queue contains 2531 outstanding requests 2005-07-07 10:31:09 [18321] [6] DEBUG: HTTP: Queue contains 2530 outstanding requests 2005-07-07 10:31:39 [18321] [6] DEBUG: HTTP: Queue contains 2529 outstanding requests But, please take care on the date: only a single HTTP request is being retried every 30 seconds!! That's due to the http-queue-delay param that was set to 30 sec. A quick calculation tells me that it will last about 20 hours to empty the queue. Digging into the smsbox.c code, I have found this schema: 1033    static void http_queue_thread (void) {     ... 1044        while ((id = gwlist_consume(smsbox_http_requests)) != NULL) { 1045         /* 1046          * Sleep for a while in order not to block other operting requests. 1047          * Defaults to 10 sec. if not given via http-queue-delay directive in 1048          * smsbox group. 1049          */ 1050         if (http_queue_delay > 0) 1051             gwthread_sleep(http_queue_delay); So it means that even if a retried message is successfully delivered in, let's say, 1 second, the next thread will effectively wait 30 seconds to process another message. Or, by default, 10 seconds. Questions: 1 - If I modify the http_queue_delay on the kannel.conf, will the outstanding HTTP queue be flushed, and so, will I loose the messages? 2 - Wouldn't it be better if we try to compose another schema to avoid this one, and sleep only between retry runs? Thanks a lot and keep with the good work !! Juan Fernández