lesha added the comment:
On a moment's reflection, a lot of my prior comment is wrong. Sorry about that.
- glog does not, that I know of, sanitize locks on fork. You just shouldn't log
after fork but before exec.
- Using `pthread_atfork` to clean up the `logging` lock might be enough to make
lesha added the comment:
I work on a multi-million-line C++ codebase that uses fork() from multithreaded
programs all over the place. We use `glog` ubiquitously.
This bug here that spans 6 years and has affected dozens of people
(conservatively) simply does not exist for us. That is because
lesha pybug.20.le...@xoxy.net added the comment:
I feel like I'm failing to get my thesis across. I'll write it out fully:
== Thesis start ==
Basic fact: It is an error to use threading locks in _any_ way after a
fork. I think we mostly agree on this. The programs we discussing
lesha pybug.20.le...@xoxy.net added the comment:
A slightly more ambitious solution than crashing / deadlocking always is to
have Python automatically spawn a fork server whenever you start using
threads.
Then, you would be able to have subprocess work cleanly, and not worry about
any
lesha pybug.20.le...@xoxy.net added the comment:
Actually, we might be able to automatically spawn a safe fork server _only_
when people start mixing threading and subprocess.
I'm not totally sure if this would allow us to salvage multiprocessing as
well...
The tricky bit is that we'd need
lesha pybug.20.le...@xoxy.net added the comment:
1) I'm totally in favor of making the standard library safe. For that purpose,
I think we should do a combination of:
a) Use file locks in logging, whenever possible.
b) Introduce LockUnsafelyReinitializedAtFork, using a generation counter
lesha pybug.20.le...@xoxy.net added the comment:
I am really alarmed by the reinit_locks patches.
I scanned the comment history, and looked at the patch. I may have missed
something, but it looks to me like the basic behavior is this:
After fork(), all locks are replaced by brand-new lock
lesha pybug.20.le...@xoxy.net added the comment:
I think forkall() on Solaris acts like that, but the normal fork()
function does not. Only the thread which performs fork() will survive
in the child process.
Sorry, brain fail. A slightly more contrived failure case
lesha pybug.20.le...@xoxy.net added the comment:
Deadlocks are dandy, but corruption is cruel.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6721
lesha pybug.20.le...@xoxy.net added the comment:
Re threading locks cannot be used to protect things outside of a single
process:
The Python standard library already violates this, in that the logging module
uses such a lock to protect the file/socket/whatever, to which it is writing.
If I
lesha pybug.20.le...@xoxy.net added the comment:
So what are you suggesting? That a lock of the default type should
raise an error if you try to acquire it when it has been acquired in a
previous process?
I was suggesting a way to make 'logging' fork-safe. No more, no less.
Does what my
lesha pybug.20.le...@xoxy.net added the comment:
This is a reply to: http://bugs.python.org/issue6721#msg151266
Charles-François raises two problems:
1) LinuxThreads have different PIDs per thread. He then points out that
LinuxThreads have long been deprecated.
2) you cannot release locks
New submission from lesha pybug.20.le...@xoxy.net:
Here is a great description of the issue:
http://docs.oracle.com/cd/E19683-01/806-6867/gen-1/index.html
This enhancement proposes a way to make Python more resistant to this kind of
deadlock.
Consider this program:
import threading
lesha pybug.20.le...@xoxy.net added the comment:
Actually, I think it does not matter which thread owns the lock, it is still
invalid to try to acquire a lock that was grabbed by the fork() parent. Why?
Because the fork() parent cannot free the child's copy of the lock anyway, and
it's
lesha pybug.20.le...@xoxy.net added the comment:
Just wanted to say that I spent something like 8 hours debugging a subprocess +
threading + logging deadlock on a real production system.
I suspected one of my locks at first, but I couldn't find any. The post-fork
code was very simple, and I
15 matches
Mail list logo
