Here’s what I see… someone with more pthread experience may want to chime in.
V2.6 code.
Each thread does this:
get_thr_arg(void)
{
thr_arg *res;
(void)pthread_mutex_lock(&arg_mut);
if(first == NULL)
(void)pthread_cond_wait(&arg_cond, &arg_mut);
if((res = first) != NULL)
if((first = first->next) == NULL)
last = NULL;
(void)pthread_mutex_unlock(&arg_mut);
if(first != NULL)
pthread_cond_signal(&arg_cond);
return res;
}
Basically:
Lock
If queue empty, block on condition
Pop from top
Unlock
Signal on condition if queue non-empty
Return data
Each time a new socket request comes in , pthread_cond_signal is called… Which,
intuitively, would mean 5 incoming requests = 5 woken up threads = 5 processed
requests. Plus once a thread’s awake, if it finishes its task, it might end up
grabbing another request anyway. (if that happens, one of the threads woken up
will throw a NULL warning) The additional cond_signal in get_thr_arg seems to
be just that, additional and extra.
The warning in the log indicates a thread woke up but had no data.
The problem I have is all the soft wording in the pthread docs:
“The pthread_cond_signal() call unblocks at least one of the threads”
“Spurious wakeups from the pthread_cond_wait() or pthread_cond_timedwait()
functions may occur”
There doesn’t seem to be any guarantee that 5 pthread_cond_signal() calls = 5
threads waking up. Nor is there any guarantee that pthread_cond_signal() won’t
wake up 5 threads instead of 1.
When I take out the extra call at the end of get_thr_arg, everything still runs
perfectly fine, and I don’t have get_thr_arg NULL warnings anymore. That
indicates to me that on Linux, on my environment, pthreads is doing 1 signal =
1 wakeup, and the extra call is unnecessary. But given all the soft language
in the manual I’m not comfortable removing the extra signal, as it might be
there for a reason, and/or to deal with other pthread implementations.
Code smell indicates to me that if the last signal is to stick around, the last
first != NULL should instead be res!=NULL && res->next!=NULL… as we shouldn’t
be accessing “first” outside of our mutex.
I would further guess that given the pthread manual soft language, the warning
should just be commented out or set to a less severe log message level, rather
than removing the extra signal.
Comments welcome.
Joe
From: Joe Gooch [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 1:09 PM
To: '[email protected]'
Subject: RE: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
The thr_arg is a queue of requests. When running with a threadpool, each thread
reads from the thr_arg queue to determine the request it should process. So
with 10 threads, 10 of them are stuck in a while loop, pulling thread arguments.
The get_thr_arg function looks at the queue. If it’s empty, it blocks waiting
for a signal. (no error)
When a request comes in, it’s pushed onto the queue, and a signal is generated.
One thread will wake up, grab the first item off the queue. If there’s more
than one item left in the queue, another signal is generated. (to have another
thread pick up the next request) Then what was popped off the top of the queue
is processed by the thread.
This error is generated when a thread wakes up, tries to process a request but
none is available. In that case, it throws this warning, and goes back into
waiting status. It’s either an indication that somewhere the conditional for
signaling a wakeup is possibly signaling too many wakeups, or that cond_wait is
prematurely waking up multiple threads.
If pound is functioning properly, these errors shouldn’t be a problem… as it
just goes back and starts waiting again. (and you could comment the warning
out in http.c) If it’s not, I’ll need more information.
How often are you seeing these messages?
What OS are you running?
What version of pound are you using?
Were you getting these before increasing the default number of threads?
Joe
From: Brad Allison [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 12:47 PM
To: [email protected]
Subject: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
Any ideas?
On Tue, Aug 27, 2013 at 3:45 PM, Brad Allison
<[email protected]<mailto:[email protected]>> wrote:
My syslog is getting spammed by :
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:37 pod01-lb-01 pound: NULL get_thr_arg
We just increased the pound threads from the default 128 to 500.
What does this error mean?
-brad
