Hi Maciek, Of course it's a difference between pypy and cpython. if you close all fds, cpython's random module still works, but PyPy's doesn't.
Cheers, Carl Friedrich On December 23, 2015 2:54:23 PM GMT+01:00, Maciej Fijalkowski <fij...@gmail.com> wrote: >well, ok, but that does not sound like a pypy bug then - "close all >existing fds and complain that some of them are closed" is a bit not >good - >maybe it's a bug in python-daemon and the PEP? > >On Wed, Dec 23, 2015 at 3:36 PM, hubo <h...@jiedaibao.com> wrote: > >> No, the python-daemon module is critical in this problem, because it >is >> the python-daemon module who closed the fd to /dev/urandom. When >process >> swith to daemon, it forks itself, and then close all open fds >(including >> stdin, stdout and stderr), so it also closes the fd for /dev/urandom >which >> is used by PyPy library. It is the standard behavior defined by >> https://www.python.org/dev/peps/pep-3143/#daemoncontext-objects and >also >> the standard behavior for unix daemons. And unfortunately there is >not a >> way to prevent the fd to be closed without knowing exactly what >number it >> is on. >> >> Without python-daemon (or similar libraries), it is only possible to >> reproduce the problem by closing the fd (usually 4) forcely, but it >does >> not make much sense. >> >> >> 2015-12-23 >> ------------------------------ >> hubo >> ------------------------------ >> >> *发件人：*Maciej Fijalkowski <fij...@gmail.com> >> *发送时间：*2015-12-23 21:22 >> *主题：*Re: Re: Re: [pypy-dev] Dead loop occurs when using python-daemon >and >> multiprocessing together in PyPy 4.0.1 >> *收件人：*"hubo"<h...@jiedaibao.com> >> *抄送：*"pypy-dev"<pypy-dev@python.org> >> >> can you reproduce the OSError problem without having the daemon >module >> involved either? >> >> On Wed, Dec 23, 2015 at 3:14 PM, hubo <h...@jiedaibao.com> wrote: >> >>> I can only reproduce the *OSError* problem. Maybe the CPU 100% is >not >>> really a dead lock, but rather some kind of automatic crash report? >>> Although it is quite easy to crash the program with os.urandom, it >>> only stops responding when the crash happens in system libraries >like >>> multiprocessing or email. >>> >>> The posix.urandom problem is quite easy to reproduce: >>> >>> #!/usr/bin/pypy >>> import os >>> os.urandom(16) >>> def test(): >>> print repr(os.urandom(16)) >>> import daemon >>> import sys >>> if __name__ == '__main__': >>> with daemon.DaemonContext(initgroups=False, >>> stderr=sys.stderr,stdout=sys.stdout): >>> test() >>> >>> (stderr and stdout is kept open to show console messages in the >daemon. >>> initgroups=False is a workaround on python-daemon not working in >Python2.6) >>> >>> Or, with module random: >>> >>> #!/usr/bin/pypy >>> import random >>> def test(): >>> random.Random() >>> import daemon >>> import sys >>> if __name__ == '__main__': >>> with daemon.DaemonContext(initgroups=False, >>> stderr=sys.stderr,stdout=sys.stdout): >>> test() >>> And when run scripts with pypy: >>> >>> pypy test3.py >>> >>> it crashes with OSError: >>> Traceback (most recent call last): >>> File "test2.py", line 13, in <module> >>> test() >>> File "test2.py", line 6, in test >>> random.Random() >>> File >"/opt/pypy-4.0.1-linux_x86_64-portable/lib-python/2.7/random.py", >>> line 95, in __init__ >>> self.seed(x) >>> File >"/opt/pypy-4.0.1-linux_x86_64-portable/lib-python/2.7/random.py", >>> line 111, in seed >>> a = long(_hexlify(_urandom(2500)), 16) >>> OSError: [Errno 9] Bad file descriptor >>> >>> It is still not clear why it causes dead loop (or long-time no >>> responding) in multiprocessing (should have thrown an ImportError) >and the >>> exact condition for the file descriptor of /dev/urandom appears >(just call >>> os.urandom and import random does not reproduce the result), but I >believe >>> it is definitely linked to the problem. >>> >>> 2015-12-23 >>> ------------------------------ >>> hubo >>> ------------------------------ >>> >>> *发件人：*Maciej Fijalkowski <fij...@gmail.com> >>> *发送时间：*2015-12-23 20:07 >>> *主题：*Re: Re: [pypy-dev] Dead loop occurs when using python-daemon >and >>> multiprocessing together in PyPy 4.0.1 >>> *收件人：*"hubo"<h...@jiedaibao.com> >>> *抄送：*"pypy-dev"<pypy-dev@python.org> >>> >>> That's very interesting, can you produce a standalone example that >does >>> not use multiprocessing? That would make it much easier to fix the >bug >>> (e.g. os.fork followed by os.urandom failing) >>> >>> On Wed, Dec 23, 2015 at 1:54 PM, hubo <h...@jiedaibao.com> wrote: >>> >>>> Thanks for the response. Should I put it directly in the bug >tracker? >>>> >>>> FYI, I've located the reason to be the incompatibility with >>>> python-daemon (or rather the standard unix-daemon behavior) and >PyPy >>>> *posix.urandom* implementation. >>>> >>>> It seems that in PyPy 4.0.1, when module *random* loaded, a file >>>> descriptor is created on /dev/urandom. I think PyPy implementation >use the >>>> shared descriptor to read from /dev/urandom. Sadly when >python-daemon fork >>>> the process and turns it into an unix daemon, it closes all the >currently >>>> open file descriptors. After that all os.urandom calls failed with >OSError. >>>> I think maybe the other functions of Random class is also using the >file >>>> descriptor in C code and just never detects if the return value is >0, and >>>> causes the dead loop. >>>> >>>> I think the problem will be solved if the implementation re-open >the >>>> handle when it is closed somehow. >>>> >>>> multiprocessing is using random internally. Also there are lots of >other >>>> modules using random, like email etc. The dead loop occurs when you >use any >>>> of the libraries in a daemon. >>>> >>>> >>>> >>>> 2015-12-23 >>>> ------------------------------ >>>> hubo >>>> ------------------------------ >>>> >>>> *发件人：*Maciej Fijalkowski <fij...@gmail.com> >>>> *发送时间：*2015-12-23 19:35 >>>> *主题：*Re: [pypy-dev] Dead loop occurs when using python-daemon and >>>> multiprocessing together in PyPy 4.0.1 >>>> *收件人：*"hubo"<h...@jiedaibao.com> >>>> *抄送：*"pypy-dev"<pypy-dev@python.org> >>>> >>>> Hi hubo >>>> >>>> Can you put it as a bug report? Those things get easily lost on the >>>> mailing list (and sadly I won't look at it right now, >multiprocessing >>>> scares me) >>>> >>>> On Wed, Dec 23, 2015 at 12:03 PM, hubo <h...@jiedaibao.com> wrote: >>>> >>>>> Hello devs, >>>>> >>>>> A (possible) dead loop is found when I use python-daemon and >>>>> multiprocessing together in PyPy 4.0.1, which does not appear in >Python(2.6 >>>>> or 2.7). Also it does not appear in earlier PyPy versions (2.0.2) >>>>> >>>>> *Reproduce*: >>>>> >>>>> First install python-daemon: >>>>> pypy_pip install python-daemon >>>>> >>>>> Use the following test script (also available in attachment): >>>>> >>>>> #!/usr/bin/pypy >>>>> import daemon >>>>> import multiprocessing >>>>> def test(): >>>>> q = multiprocessing.Queue(64) >>>>> if __name__ == '__main__': >>>>> with daemon.DaemonContext(): >>>>> test() >>>>> >>>>> When executing the script with pypy: >>>>> pypy test.py >>>>> >>>>> The background service does not exit, and is consuming 100% CPU: >>>>> ps aux | grep pypy >>>>> root 7769 99.1 0.5 235332 46812 ? R 17:52 2:09 >pypy >>>>> test.py >>>>> root 7775 0.0 0.0 103252 804 pts/1 S+ 17:54 0:00 >grep >>>>> pypy >>>>> >>>>> >>>>> >>>>> Executing the script with python: >>>>> python2.7 test.py >>>>> And the background service normally exits. >>>>> >>>>> *Environment:* >>>>> I'm using CentOS 6.5, with portable PyPy distribution for linux ( >>>>> >https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-4.0.1-linux_x86_64-portable.tar.bz2 >>>>> ) >>>>> I run the script on system built-in python (python 2.6.6), a >compiled >>>>> CPython (2.7.11), and pypy from epel-release(pypy 2.0.2, python >2.7.2), and >>>>> the problem does not appear. Though the compiled CPython is 2.7.11 >and PyPy >>>>> 4.0.4 is python 2.7.10, I think that does not matter much. >>>>> >>>>> Please contact if you have any questions or ideas. >>>>> >>>>> >>>>> 2015-12-23 >>>>> ------------------------------ >>>>> hubo >>>>> >>>>> _______________________________________________ >>>>> pypy-dev mailing list >>>>> pypy-dev@python.org >>>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>>> >>>>> >>>> >>> >> > > >------------------------------------------------------------------------ > >_______________________________________________ >pypy-dev mailing list >pypy-dev@python.org >https://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
