if(first == NULL)
(void)pthread_cond_wait(&arg_cond, &arg_mut);
->
while(first == NULL)
(void)pthread_cond_wait(&arg_cond, &arg_mut);
This says "while there is no item in the list, you wait Mr. Thread"
Then when one "first" is not null, a thread gets the mutex and pulls the first
item. The other threads would wait, AND
the spurious thread that awakens by mistake would wait for the mutex, get it,
see first=null, and then go to sleep.
Maybe I am misunderstanding the semantics of pthread_cond_wait .
-- Jake
From: Joe Gooch [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 2:50 PM
To: '[email protected]'
Subject: RE: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
Then every thread would fully empty the queue. Each thread needs to grab one
request only. (and leave the rest for the
next thread)
Joe
From: jacob anderson [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 2:44 PM
To: [email protected]
Subject: RE: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
I like to refer to IBM's docs on pthreads when looking up this stuff:
http://publib.boulder.ibm.com/infocenter/iseries/v5r4/index.jsp?topic=%2Fapis%2Fusers_78.htm
I think you should change the first " if(first == NULL) " to " while(first ==
NULL) " .
My very humble and lacking-any-code-context-opinion.
-- jake
From: Joe Gooch [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 10:54 AM
To: '[email protected]'
Subject: RE: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
Here's what I see. someone with more pthread experience may want to chime in.
V2.6 code.
Each thread does this:
get_thr_arg(void)
{
thr_arg *res;
(void)pthread_mutex_lock(&arg_mut);
if(first == NULL)
(void)pthread_cond_wait(&arg_cond, &arg_mut);
if((res = first) != NULL)
if((first = first->next) == NULL)
last = NULL;
(void)pthread_mutex_unlock(&arg_mut);
if(first != NULL)
pthread_cond_signal(&arg_cond);
return res;
}
Basically:
Lock
If queue empty, block on condition
Pop from top
Unlock
Signal on condition if queue non-empty
Return data
Each time a new socket request comes in , pthread_cond_signal is called. Which,
intuitively, would mean 5 incoming
requests = 5 woken up threads = 5 processed requests. Plus once a thread's
awake, if it finishes its task, it might end
up grabbing another request anyway. (if that happens, one of the threads woken
up will throw a NULL warning) The
additional cond_signal in get_thr_arg seems to be just that, additional and
extra.
The warning in the log indicates a thread woke up but had no data.
The problem I have is all the soft wording in the pthread docs:
"The pthread_cond_signal() call unblocks at least one of the threads"
"Spurious wakeups from the pthread_cond_wait() or pthread_cond_timedwait()
functions may occur"
There doesn't seem to be any guarantee that 5 pthread_cond_signal() calls = 5
threads waking up. Nor is there any
guarantee that pthread_cond_signal() won't wake up 5 threads instead of 1.
When I take out the extra call at the end of get_thr_arg, everything still runs
perfectly fine, and I don't have
get_thr_arg NULL warnings anymore. That indicates to me that on Linux, on my
environment, pthreads is doing 1 signal =
1 wakeup, and the extra call is unnecessary. But given all the soft language
in the manual I'm not comfortable removing
the extra signal, as it might be there for a reason, and/or to deal with other
pthread implementations.
Code smell indicates to me that if the last signal is to stick around, the last
first != NULL should instead be
res!=NULL && res->next!=NULL. as we shouldn't be accessing "first" outside of
our mutex.
I would further guess that given the pthread manual soft language, the warning
should just be commented out or set to a
less severe log message level, rather than removing the extra signal.
Comments welcome.
Joe
From: Joe Gooch [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 1:09 PM
To: '[email protected]'
Subject: RE: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
The thr_arg is a queue of requests. When running with a threadpool, each thread
reads from the thr_arg queue to
determine the request it should process. So with 10 threads, 10 of them are
stuck in a while loop, pulling thread
arguments.
The get_thr_arg function looks at the queue. If it's empty, it blocks waiting
for a signal. (no error)
When a request comes in, it's pushed onto the queue, and a signal is generated.
One thread will wake up, grab the
first item off the queue. If there's more than one item left in the queue,
another signal is generated. (to have
another thread pick up the next request) Then what was popped off the top of
the queue is processed by the thread.
This error is generated when a thread wakes up, tries to process a request but
none is available. In that case, it
throws this warning, and goes back into waiting status. It's either an
indication that somewhere the conditional for
signaling a wakeup is possibly signaling too many wakeups, or that cond_wait is
prematurely waking up multiple threads.
If pound is functioning properly, these errors shouldn't be a problem. as it
just goes back and starts waiting again.
(and you could comment the warning out in http.c) If it's not, I'll need more
information.
How often are you seeing these messages?
What OS are you running?
What version of pound are you using?
Were you getting these before increasing the default number of threads?
Joe
From: Brad Allison [mailto:[email protected]]
Sent: Wednesday, August 28, 2013 12:47 PM
To: [email protected]
Subject: [Pound Mailing List] Re: NULL: get_thr_arg spamming my syslog
Any ideas?
On Tue, Aug 27, 2013 at 3:45 PM, Brad Allison <[email protected]> wrote:
My syslog is getting spammed by :
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:36 pod01-lb-01 pound: NULL get_thr_arg
Aug 27 19:44:37 pod01-lb-01 pound: NULL get_thr_arg
We just increased the pound threads from the default 128 to 500.
What does this error mean?
-brad
