[issue17681] Work with an extra field of gzip and zip files
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue17681> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can lose data from buffered files in reference cycles
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34624] -W option and PYTHONWARNINGS env variable does not accept module regexes
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue34624> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
Nikolaus Rath added the comment: Is there a good reason to not explicitly initialize them nevertheless? Valgrind is a common tool, and this false positive will affect everyone attempting to run it on a Python extension module. -- ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20177] Derby #8: Convert 28 sites to Argument Clinic across 2 files
Nikolaus Rath added the comment: Sorry, no. I have long lost context and interest in this. -- ___ Python tracker <https://bugs.python.org/issue20177> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
Nikolaus Rath added the comment: $ ./configure CFLAGS="-O0 -g" --with-valgrind && make -j8 still gives ==13281== Memcheck, a memory error detector ==13281== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==13281== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info ==13281== Command: /home/nikratio/clones/cpython/python /home/nikratio/in-progress/pyfuse3/test/../examples/hello.py /tmp/pytest-of-nikratio/pytest-15/test_hello_hello_py_0 ==13281== ==13281== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==13281==at 0x584906A: epoll_ctl (syscall-template.S:84) ==13281==by 0xB5C07FA: pyepoll_internal_ctl (selectmodule.c:1392) ==13281==by 0xB5C08CB: select_epoll_register_impl (selectmodule.c:1438) ==13281==by 0xB5C112A: select_epoll_register (selectmodule.c.h:599) ==13281==by 0x173031: _PyMethodDef_RawFastCallKeywords (call.c:658) ==13281==by 0x2FEFCD: _PyMethodDescr_FastCallKeywords (descrobject.c:290) ==13281==by 0x21ED25: call_function (ceval.c:4611) ==13281==by 0x21AB4E: _PyEval_EvalFrameDefault (ceval.c:3183) ==13281==by 0x21007C: PyEval_EvalFrameEx (ceval.c:533) ==13281==by 0x17245F: function_code_fastcall (call.c:285) ==13281==by 0x1728B5: _PyFunction_FastCallKeywords (call.c:410) ==13281==by 0x21EDF4: call_function (ceval.c:4634) ==13281== Address 0xffeff2d94 is on thread 1's stack ==13281== in frame #1, created by pyepoll_internal_ctl (selectmodule.c:1379) -- ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
Nikolaus Rath added the comment: Same error with 3.7.1 and 3.6.7 (though line numbers differ slightly). -- ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
New submission from Nikolaus Rath : With current git master, configured --with-valgrind --with-pydebug, I get: ==31074== Command: /home/nikratio/clones/cpython/python /home/nikratio/in-progress/pyfuse3/test/../examples/hello.py /tmp/pytest-of-nikratio/pytest-11/test_hello_hello_py_0 ==31074== ==31074== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==31074==at 0x584906A: epoll_ctl (syscall-template.S:84) ==31074==by 0xBDAA493: pyepoll_internal_ctl (selectmodule.c:1392) ==31074==by 0xBDAA59F: select_epoll_register_impl (selectmodule.c:1438) ==31074==by 0xBDAA5F8: select_epoll_register (selectmodule.c.h:599) ==31074==by 0x174E15: _PyMethodDef_RawFastCallKeywords (call.c:658) ==31074==by 0x300BCA: _PyMethodDescr_FastCallKeywords (descrobject.c:290) ==31074==by 0x21FC05: call_function (ceval.c:4611) ==31074==by 0x22B5E7: _PyEval_EvalFrameDefault (ceval.c:3183) ==31074==by 0x2206FF: PyEval_EvalFrameEx (ceval.c:533) ==31074==by 0x173B61: function_code_fastcall (call.c:285) ==31074==by 0x174737: _PyFunction_FastCallKeywords (call.c:410) ==31074==by 0x21FDF4: call_function (ceval.c:4634) ==31074== Address 0xffeffeb4c is on thread 1's stack ==31074== in frame #1, created by pyepoll_internal_ctl (selectmodule.c:1379) To reproduce: $ python-dev -m pip install --user pyfuse3 # for dependencies $ git clone https://github.com/libfuse/pyfuse3.git $ valgrind --trace-children=yes "--trace-children-skip=*mount*" python-dev -m pytest test/ pyfuse3 provides a C extension module, but I believe the problem is in the interpreter core as the stacktrace does not include anything from the extension. -- components: Interpreter Core messages: 332348 nosy: nikratio priority: normal severity: normal status: open title: Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s) type: compile error versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34624] -W option and PYTHONWARNINGS env variable does not accept module regexes
Change by Nikolaus Rath : -- nosy: +nikratio title: -W option does not accept module regexes -> -W option and PYTHONWARNINGS env variable does not accept module regexes ___ Python tracker <https://bugs.python.org/issue34624> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34920] PYTHONWARNINGS entries are escaped
Nikolaus Rath added the comment: yes, this is a duplicate. -- ___ Python tracker <https://bugs.python.org/issue34920> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34920] PYTHONWARNINGS entries are escaped
New submission from Nikolaus Rath : According to https://docs.python.org/3/library/warnings.html#describing-warning-filters: "The meaning of each of these fields [of PYTHONWARNINGS] is as described in ." The description of the "The Warnings filter" says "module is a string containing a regular expression that the module name must match." However, when parsing PYTHONWARNINGS, the warnings module explicitly escapes the module field. -- components: Library (Lib) messages: 327272 nosy: nikratio priority: normal severity: normal status: open title: PYTHONWARNINGS entries are escaped type: behavior versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue34920> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue1230540> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can lose data from buffered files at exit
Nikolaus Rath <nikol...@rath.org> added the comment: Given the apparent difficulties getting this right, how about not attempting to implicitly flush buffers in the finalizer at all? This means scripts relying on this will break, but in contrast to the current behavior they will break consistently so it's easy to find and fix the problem. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31122] SSLContext.wrap_socket() throws OSError with errno == 0
Nikolaus Rath added the comment: Regarding "atrocious connection": I wish I knew, but I have no control of the connection. All I can tell is that there are frequent disconnects, occasional latency spikes, my remote ip address seems to change frequently (while the apparent local one stays as-is, so presumably some NAT in between), temporary bandwidth drops, etc. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31122> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31122] SSLContext.wrap_socket() throws OSError with errno == 0
New submission from Nikolaus Rath: With a particularly atrocious network connection, I often get the following exception: File "/usr/lib/python3/dist-packages/dugong/__init__.py", line 503, in connect self._sock = self.ssl_context.wrap_socket(self._sock, server_hostname=server_hostname) File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket _context=self) File "/usr/lib/python3.5/ssl.py", line 760, in __init__ self.do_handshake() File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake self._sslobj.do_handshake() OSError: [Errno 0] Error I don't think an error with errno == 0 should ever be raised by Python. -- assignee: christian.heimes components: SSL messages: 299759 nosy: christian.heimes, nikratio priority: normal severity: normal status: open title: SSLContext.wrap_socket() throws OSError with errno == 0 type: behavior versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31122> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: Would you be willing to review a patch to incorporate the handling into the SSL module? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: Stefan, sorry for ignoring your earlier reply. I somehow missed the question at the end. I believe that users of the Python module are *not* expected to make use of the WANT_READ, WANT_WRITE flags. Firstly because the documentation (of Python's ssl module) doesn't say anything about that, and secondly because the code that's necessary to handle these flags is a prime example for complexity that is imposed by the C API that should be hidden to Python users. That said, could you give a more specific reference to the O'Relly book (and maybe even page or chapter)? At the moment it's a little hard for me to follow the rest of your message. Essentially, if I'm trying to write to a non-blocking, Python SSL socket, I would expect that this either succeeds or raises SSL_WANT_WRITE/READ. Not having read the book, it seems to me this is the only information that's useful to a Python caller. In what situation would you need the more exact state that your C example tracks? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: This just happened again to someone else, also using Python 3.4: https://bitbucket.org/nikratio/s3ql/issues/87 Is there anything the affected people can do to help debugging this? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18713] Clearly document the use of PYTHONIOENCODING to set surrogateescape
Nikolaus Rath added the comment: The first thing that would come to my mind when reading Nick's proposed document (without first reading this bug report) is So why the heck is this not the default?. It would probably save a lot of people a lot of anger if there was also a brief explanation addressing this obvious first response :-). -- nosy: +nikratio ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18713 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: I believe Martin's patch (v8) is ready for a core committer review. At least I can't find anything to criticize anymore :-). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23529 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: Except for the pointless 'myfileobj' stuff in gzip.py, rev 8 of the patch looks good to me. (Btw, I'm not actually in favor of e.g. the seekable() change. The previous patch was intended as a proof-of-concept to see what would be necessary to inherit more from DecompressReader and if it's actually worth it. But having thought about it for a while more, I don't think there's a significant gain. But I'm happy to see that you were able to cherry-pick some useful pieces out of it nevertheless). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23529 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: As discussed in Rietveld, here's an attempt to reuse more DecompressReader for GzipFile. There is still an unexplained test failure (test_read_truncated). -- Added file: http://bugs.python.org/file38674/LZMAFile-etc.v7.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23529 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: If you want to add support for buffer_size=0 in a separate patch/issue I think that's fine. But in that case I would not add a buffer_size parameter now at all. IMO, not having it is better as having it but not supporting zero (even if it's documented that's pretty surprising). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23529 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: On Mar 06 2015, Martin Panter rep...@bugs.python.org wrote: Still to do: Need to find a better home for the _DecompressReader and _BaseStream classes. Currently it lives in “lzma”, but apparently it is possible for any of the gzip, bz2, lzma modules to not be importable, so it would have to live elsewhere. Yes. Possible options are the io module, or a brand new internal module (e.g. Lib/_compression.py). Thoughts? I think a new internal module would be the right choice, but I don't know what needs to be done to properly add it to the build system. So for now I'd just put it in the io module and wait for a core committer to complain :-). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- title: Limit decompressed data when reading from LZMAFile, BZ2File, GzipFile - Limit decompressed data when reading from LZMAFile and BZ2File ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23529 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: On Feb 27 2015, Martin Panter rep...@bugs.python.org wrote: In the code review, Nikolaus raised the idea of allowing a custom “buffer_size” parameter for the BufferedReader. I think this would need a bit of consideration about how it should work: 1. Should it be a direct wrapper around BufferedReader(buffer_size=...)? I'm not 100% sure what you mean by that, but I think the answer is Yes. 2. Should it also support an unbuffered reader mode like open(buffering=0), socket.makefile(buffering=0), and subprocess.Popen(bufsize=0)? Yes, in that case the DecompressReader should simply be used without encapsulating it in BufferedReader. 3. Should there be any consideration for buffering in write mode (mirroring the equivalent open(), etc parameters)? I don't think so. There's not much to be gained by buffering data before passing it to the compressor, so the only point of buffering is to avoid short writes (or reads in case of decompressing). But the low-level compressor code already avoids short writes. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23529 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23528] Limit decompressed data when reading from GzipFile
Nikolaus Rath added the comment:
Especially now that this is only going to go into 3.5, I think it makes more
sense to handle GzipFile, LZMAFile and BZ2File all in one go. Looking at the
code, otherwise there's going to be a lot of duplication.
How about introducing a base class 'CompressedFile' that defines most of the
logic that's currently in LZMAFile (including the max_size patch from issue
23529), and having {LZMA,BZ2,Gzip}File all inherit from that base?
BZ2File and LZMAFile would probably only need to define their own constructor
to instantiate the proper compressor/decompressor object.
GzipFile would need to additionally overwrite read() and write() in order to
handle the CRC and gzip header. But I think both methods could still be written
to call super().read/write().
Did I miss something?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23528
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Martin, I'll try to review your GzipFile patch. But maybe it would make sense to open a separate issue for this? I think the LZMAFile patch has not yet been reviewed or committed, and we probably want a patch for BZ2File too. The review page is already pretty cluttered right now, so I think it would make sense to use this issue for the low-level compressor/decompressor API and handle the *File API separately. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Updated patch attached. -- Added file: http://bugs.python.org/file38206/issue15955_bz2_rev2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Attached is a patch for the bz2 module. -- Added file: http://bugs.python.org/file38194/issue15955_bz2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: I've started to work on the bzip module. I'll attach I work-in-progress patch if I get stuck or run out of time. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Yes, I still plan to do it, but I haven't started yet. That said, I certainly won't be offended if someone else implements the feature either. Please just let me know when you start working on this (I'll do the same), so there's no duplication of efforts. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: On a more practical note: I believe Nadeem at one point said that the bz2 module is not exactly an example for good stdlib coding, while the lzma module's implementation is quite clean. Therefore Therefore, one idea I had for the bz2 module was to port the lzma module to use the bzip2 library (instead of adding a max_size parameter to the bz2 module). Just something to consider if you find time to work on this before me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23300] httplib is using a method that doesn't exist
Nikolaus Rath added the comment: Serhiy, did you add me to Cc just for information, or is there anything I should be doing (having written the patch that introduced this bug)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23300 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Martin Panter rep...@bugs.python.org writes: We still need a patch for max_length in BZ2Decompressor, and to use it in BZ2File. I'm still quite interested to do this. The only reason I haven't done it yet is that I'm waiting for the LZMA patch to be reviewed and committed first (no need to make any mistakes twice). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: *ping*. It's been another 8 months. It would be nice if someone could review the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Nikolaus Rath added the comment: This will probably be too radial, but I think it should at least be mentioned as a possible option. We could just not attempt to implicitly flush buffers in the finalizer at all. This means scripts relying on this will break, but in contrast to the current behavior they will break consistently so it's easy to find and fix the problem. I think this is still preferable to any solution that does not guarantee flushing in 100% of the cases. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17852 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Changes by Nikolaus Rath nikol...@rath.org: -- nosy: +nikratio ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17852 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Nikolaus Rath added the comment: Serhiy, I believe this still happens in Python 3.4, but it is harder to reproduce. I couldn't get Armin's script to produce the problem either, but I'm pretty sure that this is what causes e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771452#60. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17852 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
New submission from Nikolaus Rath: I received a bugreport due to a crash when calling SSLObject.send(). The traceback ends with: [...] File /usr/local/lib/python3.4/dist-packages/dugong-3.2-py3.4.egg/dugong/__init__.py, line 584, in _co_send len_ = self._sock.send(buf) File /usr/lib/python3.4/ssl.py, line 679, in send v = self._sslobj.write(data) ssl.SSLError: [SSL: BAD_WRITE_RETRY] bad write retry (_ssl.c:1636) At first I thought that this is an exception that my application should catch and handle. However, when trying to figure out what exactly BAD_WRITE_RETRY means I get the impression that the fault is actually in Python's _ssl.c. The only places where this error is returned by OpenSSL are ssl/s2_pkt.c:480 and ssl/s3_pkt.c:1179, and in each case the problem seems to be with the caller supplying an invalid buffer after an initial write request failed to complete due to non-blocking IO. This does not seem to be something that could be caused by whatever Python code, so I think there is a problem in _ssl.c. -- components: Library (Lib) messages: 227582 nosy: alex, christian.heimes, dstufft, giampaolo.rodola, janssen, nikratio, pitrou priority: normal severity: normal status: open title: [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636 type: crash versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22499 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
ssl.SSLError: [SSL: BAD_WRITE_RETRY] bad write retry (_ssl.c:1636)
Hello, Can someone explain help me understand what this exception means? [...] File /usr/local/lib/python3.4/dist-packages/dugong-3.2-py3.4.egg/dugong/__init__.py, line 584, in _co_send len_ = self._sock.send(buf) File /usr/lib/python3.4/ssl.py, line 679, in send v = self._sslobj.write(data) ssl.SSLError: [SSL: BAD_WRITE_RETRY] bad write retry (_ssl.c:1636) Presumably this is generated by OpenSSL, but how do I figure out what it means? The best I found in the OpenSSL documentation is https://www.openssl.org/docs/crypto/err.html, and Google only found brought me to https://stackoverflow.com/questions/2997218. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: *ping* -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: R. David Murray rep...@bugs.python.org writes: R. David Murray added the comment: Nikolaus: while I agree that Raymond's comments were a bit strongly worded, it doesn't read to me as if the thread you link to is on point for this issue. The thread was focused on a *specific* question, that of calling close twice. The question of what the docs mean by a file like object is a different question. Specifically, you will note that duck typing never came up in that thread (as far as I remember). That's quite correct. But did not come up in the original issue report either - that's why I still don't understand why we're discussing it here at all. My line of thought was as follows: - Standard library assumes that close() is idempotent in many places - Currently this isn't documented clearly - The best place to make it more explicit seemed to be the description of IOBase - However, changing the description of only IOBase.close() could easily give the impression that close() is somehow special, and that fewer/no assumptions are made in the standard lbirary about the presence/behavior of the other methods (e.g. fileno or readable). - Therefore, to me the best course of action seemed to add a paragraph explicitly describing that the standard library may assume that *any* method/attribute of a stream object behaves as described for IOBase. I still don't see how this contradicts / interacts with the concept of duck typing. I think the documentation (with and without the patch) clearly implies that you can implement an object that does not have the full IOBase API and successfully hand it to some standard library function - it's just that in that case you can't complain if it breaks. Or is the point of contention just the title of this issue? Maybe it was poorly chosen, I guess Clarify standard library's expectations from stream objects would have been better. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: On 06/17/2014 01:28 PM, R. David Murray wrote: Well, but we think it's pretty clear. This wasn't the impression that I had from the thread on python-devel, but I'll accept your judgement on that. I'll be more restrained when being asked for suggestions in the future. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: On 06/15/2014 06:26 PM, Raymond Hettinger wrote: Before creating more tracker items, please take time to learn about how Python's history, [...] It'd be nice if you would have at least followed the link to http://article.gmane.org/gmane.comp.python.devel/148199 before commenting. In case you still can't spare the time for that: I explicitly asked on python-dev about this *before* I opened a tracker item. Apart from that, I think your remarks are neither appropriate nor do they belong in the tracker. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: As discussed on python-devel, I'm attaching a new patch that uses memoryview.cast to ensure that the pure-Python readinto() now works with any object implementing the buffer protocol. -- Added file: http://bugs.python.org/file35647/issue20578_r5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: On 06/15/2014 08:29 AM, R. David Murray wrote: I don't think that's true, though. file like pretty much means has the file attributes that I actually use. That is, it is context dependent (duck typing). Well, but when you pass your file-like object to some function from the standard library, you don't know what file attributes will be used. So to make sure that things work as expected, you have to make sure that your file-like object behaves as prescribed by the IOBase* classes. I'm also not sure I see the point in the change. It is inherent in the definition of what ABCs are. True. But not everyone reading the io documentation is familiar enough with ABCs to immediately make that mental translation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: (refreshed patch, no changes) -- Added file: http://bugs.python.org/file35648/issue20578_r6.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: Maybe I'm missing some important point here, but I think that the documentation ought to tell me how I have to design a file-like object such that it fulfills all expectations of the standard library. Yes, you can get away with less than that in many situations, but that doesn't mean that the documentation should not tell me about the full set of expectations. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Nadeem, did you get a chance to look at this? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19414] iter(ordered_dict) yields keys not in dict in some circumstances
Nikolaus Rath added the comment: Raymond, it would be nice if you could respond to my last comment... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19414 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
New submission from Nikolaus Rath: It is currently not perfectly clear what Python (and the standard library) assumes about file-like objects (see e.g. http://article.gmane.org/gmane.comp.python.devel/148199). The attached doc patch tries to improve the current situation by stating explicitly that the description of IOBase et al specifies a *mandatory* interface for anything that claims to be file-like. -- assignee: docs@python components: Documentation files: iobase.diff keywords: patch messages: 220588 nosy: benjamin.peterson, docs@python, eric.araujo, ezio.melotti, georg.brandl, hynek, ncoghlan, nikratio, pitrou, stutzbach priority: normal severity: normal status: open title: Clarify requirements for file-like objects type: enhancement versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file35636/iobase.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21763 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21764] Document that IOBase.__del__ calls self.close
New submission from Nikolaus Rath: CPython's io.IOBase.__del__ calls self.close(), but this isn't documented anywhere (and may be surprised for derived classes). The attached patch extends the documentation. -- assignee: docs@python components: Documentation files: iobase2.diff keywords: patch messages: 220591 nosy: benjamin.peterson, docs@python, eric.araujo, ezio.melotti, georg.brandl, hynek, nikratio, pitrou, stutzbach priority: normal severity: normal status: open title: Document that IOBase.__del__ calls self.close type: enhancement versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file35637/iobase2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21764 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21764] Document that IOBase.__del__ calls self.close
Changes by Nikolaus Rath nikol...@rath.org: Removed file: http://bugs.python.org/file35637/iobase2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21764 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21764] Document that IOBase.__del__ calls self.close
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file35638/iobase2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21764 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment:
Thanks for taking the time, and apologies about the test failure. I was
probably too eager and ran only the test_io suite instead of everything.
I looked at the failure, and the problem is that the default Python
BufferedIOBase.readinto implementation is semi-broken. It should work with any
object implementing the memoryview protocol (like the C implementation), but it
really only works with bytearray objects. The testIteration test only worked
(prior to the patch) because there is a special case for array objects with
format 'b':
try:
b[:n] = data
except TypeError as err:
import array
if not isinstance(b, array.array):
raise err
b[:n] = array.array('b', data)
In other words, trying to read into any other object has always failed. In
particular, even format code 'B' fails:
import _pyio
from array import array
buf = array('b', b'x' * 10)
_pyio.open('/dev/zero', 'rb').readinto(buf)
10
buf = array('B', b'x' * 10)
_pyio.open('/dev/zero', 'rb').readinto(buf)
Traceback (most recent call last):
File stdin, line 1, in module
File /home/nikratio/clones/cpython/Lib/_pyio.py, line 1096, in readinto
buf[:len_] = array.array('b', buf2)
TypeError: bad argument type for built-in operation
The readline implementation that my patch adds for BufferedReader does not
contain this special case, and therefore with the patch even the test with a
'b'-array fails.
For now, I've added the same special casing of 'b'-type arrays to the
_readline() implementation in BufferedReader. This fixes the immediate problem
(and this time I definitely ran the entire testsuite).
However, the fix is certainly not what I would consider a good solution.. but I
guess that would better be addressed by a separate patch that also fixes the
same issue in BufferedIOBase?
--
Added file: http://bugs.python.org/file35539/issue20578_r4.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20578
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment:
I used the wrong interpreter when cutting and pasting the example above, here's
the correct version to avoid confusion with the traceback:
import _pyio
from array import array
buf = array('b', b'x' * 10)
_pyio.open('/dev/zero', 'rb').readinto(buf)
10
buf = array('B', b'x' * 10)
_pyio.open('/dev/zero', 'rb').readinto(buf)
Traceback (most recent call last):
File /home/nikratio/clones/cpython/Lib/_pyio.py, line 662, in readinto
b[:n] = data
TypeError: can only assign array (not bytes) to array slice
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File stdin, line 1, in module
File /home/nikratio/clones/cpython/Lib/_pyio.py, line 667, in readinto
b[:n] = array.array('b', data)
TypeError: bad argument type for built-in operation
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20578
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Missing stack frames?
Chris Angelico ros...@gmail.com writes: On Fri, Jun 6, 2014 at 12:16 PM, Nikolaus Rath nikol...@rath.org wrote: - Is there some way to make the call stack for destructors less confusing? First off, either don't have refloops, or explicitly break them. The actual code isn't as simple as the example. I wasn't even aware there were any refloops. Guess I have to start hunting them down now. That'll at least make things a bit more predictable; in CPython, you'll generally see destructors called as soon as something goes out of scope. Secondly, avoid threads when you're debugging a problem! I said this right at the beginning. If you run into problems, the first thing to do is isolate the system down to a single thread. Debugging is way easier without threads getting in each other's ways. I don't see how this would have made a difference in this case. I still would have gotten lots of apparently non-sensical backtraces. Only this time they would all come from MainThread. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
Re: Missing stack frames?
Paul Rubin no.email@nospam.invalid writes: Nikolaus Rath nikol...@rath.org writes: Still no context before the ominous close() call. I'm very confused. close() could be getting called from a destructor as the top level function of a thread exits, or something like that. Shouldn't the destructor have its own stack frame then, i.e. shouldn't the first frame be in a __del__ function? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
Re: Missing stack frames?
Nikolaus Rath nikol...@rath.org writes:
Chris Angelico ros...@gmail.com writes:
On Wed, Jun 4, 2014 at 12:30 PM, Nikolaus Rath nikol...@rath.org wrote:
I've instrumented one of my unit tests with a conditional
'pdb.set_trace' in some circumstances (specifically, when a function is
called by a thread other than MainThread).
I think the likelihood of this being an issue with interactive
debugging and threads is sufficiently high that you should avoid
putting the two together, at least until you can verify that the same
problem occurs without that combination.
Here's stacktrace as obtained by traceback.print_stack():
tests/t1_backends.py:563: test_extra_data[mock_s3c-zlib] PASSED
tests/t1_backends.py:563: test_extra_data[mock_s3c-bzip2] PASSED
87 tests deselected by '-kextra'
=
=== 5 passed, 1 skipped, 87 deselected in 0.65 seconds
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line
853, in close
self.fh.close()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 691,
in close
traceback.print_stack(file=sys.stdout)
something is wrong
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line
853, in close
self.fh.close()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 691,
in close
traceback.print_stack(file=sys.stdout)
something is wrong
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line
1050, in close
self.fh.close()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 691,
in close
traceback.print_stack(file=sys.stdout)
Still no context before the ominous close() call. I'm very confused.
It get's even funnier if I just repeat this exercise (without changing
code). Here are some more tracebacks that I got:
File /usr/bin/py.test-3, line 5, in module
sys.exit(load_entry_point('pytest==2.5.2', 'console_scripts', 'py.test')())
[...]
File /usr/lib/python3/dist-packages/py/_code/code.py, line 524, in
repr_traceback_entry
source = self._getentrysource(entry)
File /usr/lib/python3/dist-packages/py/_code/code.py, line 450, in
_getentrysource
source = entry.getsource(self.astcache)
File /usr/lib/python3/dist-packages/py/_code/code.py, line 199, in getsource
astnode=astnode)
File /usr/lib/python3/dist-packages/py/_code/source.py, line 367, in
getstatementrange_ast
astnode = compile(content, source, exec, 1024) # 1024 for AST
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line
1050, in close
self.fh.close_real()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 702, in
close_real
traceback.print_stack(file=sys.stdout)
or also
File /usr/bin/py.test-3, line 5, in module
sys.exit(load_entry_point('pytest==2.5.2', 'console_scripts', 'py.test')())
[...]
File /usr/lib/python3.4/logging/__init__.py, line 1474, in callHandlers
hdlr.handle(record)
File /usr/lib/python3.4/logging/__init__.py, line 842, in handle
rv = self.filter(record)
File /usr/lib/python3.4/logging/__init__.py, line 699, in filter
for f in self.filters:
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 853,
in close
self.fh.close()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line
1050, in close
self.fh.close_real()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 702, in
close_real
traceback.print_stack(file=sys.stdout)
and this one looks actually the way I would expect:
[...]
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 335,
in fetch
return self.perform_read(do_read, key)
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 58,
in wrapped
return method(*a, **kw)
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 309,
in perform_read
return fn(fh)
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 859,
in __exit__
self.close()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 853,
in close
self.fh.close()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line
1050, in close
self.fh.close_real()
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 702, in
close_real
traceback.print_stack(file=sys.stdout)
I am not using any C extension modules, but I guess I nevertheless have
to assume that something seriously messed up either the stack or the
traceback printing routines?
Best,
Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
https://mail.python.org/mailman/listinfo/python-list
Re: Missing stack frames?
dieter die...@handshake.de writes:
[...]
Someone else already mentioned that the close call
can come from a destructor. Destructors can easily be called
at not obvious places (e.g. s = C(); ... x = [s for s in ...];
in this example the list comprehension calls the C destructor
which is not obvious when one looks only locally).
The destructor calls often have intervening C code (which
one does not see). However, in your case, I do not see
why the cgi module should cause a destructor call of one
of your server components.
Paul, dieter, you are my heroes. It was indeed an issue with a
destructor. It turns out that the io.RawIOBase destructor calls
self.close(). If the instance of a derived class is part of a reference
cycle, it gets called on the next routine run of the garbage
collector -- with the stack trace originating at whatever statement was
last executed before the gc run.
The following minimal example reproduces the problem:
#!/usr/bin/env python3
import io
import traceback
import threading
class Container:
pass
class InnocentVictim(io.RawIOBase):
def close(self):
print('close called in %s by:'
% threading.current_thread().name)
traceback.print_stack()
def busywork():
numbers = []
for i in range(500):
o = Container()
o.l = numbers
numbers.append(o)
if i % 87 == 0:
numbers = []
l = [ InnocentVictim() ]
l[0].cycle = l
del l
t = threading.Thread(target=busywork)
t.start()
t.join()
If you run this, you could things like:
close called in Thread-1 by:
File /usr/lib/python3.4/threading.py, line 888, in _bootstrap
self._bootstrap_inner()
File /usr/lib/python3.4/threading.py, line 920, in _bootstrap_inner
self.run()
File /usr/lib/python3.4/threading.py, line 868, in run
self._target(*self._args, **self._kwargs)
File ./test.py, line 18, in busywork
o = Container()
File ./test.py, line 13, in close
traceback.print_stack()
Ie, a method being called by a thread that doesn't have access to the
object, and without any reference to the call in the source.
I am left wondering:
- Is there really a point in the RawIOBase destructor calling close?
- Is there some way to make the call stack for destructors less confusing?
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
https://mail.python.org/mailman/listinfo/python-list
Re: Missing stack frames?
Chris Angelico ros...@gmail.com writes: On Wed, Jun 4, 2014 at 12:30 PM, Nikolaus Rath nikol...@rath.org wrote: I've instrumented one of my unit tests with a conditional 'pdb.set_trace' in some circumstances (specifically, when a function is called by a thread other than MainThread). I think the likelihood of this being an issue with interactive debugging and threads is sufficiently high that you should avoid putting the two together, at least until you can verify that the same problem occurs without that combination. Is there a way to produce a stacktrace without using the interactive debugger? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
Re: Corrputed stacktrace?
Chris Angelico ros...@gmail.com writes: On Wed, Jun 4, 2014 at 12:20 PM, Nikolaus Rath nikol...@rath.org wrote: File /usr/lib/python3.3/threading.py, line 878 in _bootstrap Can you replicate the problem in a non-threaded environment? Threads make interactive debugging very hairy. Hmm. I could try to run the server thread in a separate process. I'll try that and report back. Thanks for the suggestion, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
Re: Missing stack frames?
Chris Angelico ros...@gmail.com writes: On Wed, Jun 4, 2014 at 12:30 PM, Nikolaus Rath nikol...@rath.org wrote: I've instrumented one of my unit tests with a conditional 'pdb.set_trace' in some circumstances (specifically, when a function is called by a thread other than MainThread). I think the likelihood of this being an issue with interactive debugging and threads is sufficiently high that you should avoid putting the two together, at least until you can verify that the same problem occurs without that combination. Here's stacktrace as obtained by traceback.print_stack(): tests/t1_backends.py:563: test_extra_data[mock_s3c-zlib] PASSED tests/t1_backends.py:563: test_extra_data[mock_s3c-bzip2] PASSED 87 tests deselected by '-kextra' = === 5 passed, 1 skipped, 87 deselected in 0.65 seconds File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 853, in close self.fh.close() File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 691, in close traceback.print_stack(file=sys.stdout) something is wrong File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 853, in close self.fh.close() File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 691, in close traceback.print_stack(file=sys.stdout) something is wrong File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 1050, in close self.fh.close() File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 691, in close traceback.print_stack(file=sys.stdout) Still no context before the ominous close() call. I'm very confused. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
Corrputed stacktrace?
Hello,
I'm trying to debug a problem. As far as I can tell, one of my methods
is called at a point where it really should not be called. When setting
a breakpoint in the function, I'm getting this:
/home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py(693)close()
- if not self.md5_checked:
(Pdb) bt
/usr/lib/python3.3/threading.py(878)_bootstrap()
- self._bootstrap_inner()
/usr/lib/python3.3/threading.py(901)_bootstrap_inner()
- self.run()
/usr/lib/python3.3/threading.py(858)run()
- self._target(*self._args, **self._kwargs)
/usr/lib/python3.3/socketserver.py(610)process_request_thread()
- self.finish_request(request, client_address)
/usr/lib/python3.3/socketserver.py(345)finish_request()
- self.RequestHandlerClass(request, client_address, self)
/usr/lib/python3.3/socketserver.py(666)__init__()
- self.handle()
/home/nikratio/in-progress/s3ql/tests/mock_server.py(77)handle()
- return super().handle()
/usr/lib/python3.3/http/server.py(402)handle()
- self.handle_one_request()
/usr/lib/python3.3/http/server.py(388)handle_one_request()
- method()
/home/nikratio/in-progress/s3ql/tests/mock_server.py(169)do_GET()
- q = parse_url(self.path)
/home/nikratio/in-progress/s3ql/tests/mock_server.py(52)parse_url()
- p.params = urllib.parse.parse_qs(q.query)
/usr/lib/python3.3/urllib/parse.py(553)parse_qs()
- encoding=encoding, errors=errors)
/usr/lib/python3.3/urllib/parse.py(585)parse_qsl()
- pairs = [s2 for s1 in qs.split('') for s2 in s1.split(';')]
/usr/lib/python3.3/urllib/parse.py(585)listcomp()
- pairs = [s2 for s1 in qs.split('') for s2 in s1.split(';')]
/home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py(853)close()
- self.fh.close()
/home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py(693)close()
- if not self.md5_checked:
To me this does not make any sense.
Firstly, the thread that is (apparently) calling close should never ever
reach code in common.py. This thread is executing a socketserver handler
that is entirely contained in mock_server.py and only communicates with
the rest of the program via tcp.
Secondly, the backtrace does not make sense. How can evaluation of
pairs = [s2 for s1 in qs.split('') for s2 in s1.split(';')]
in urllib/parse.py() result in a method call in backends/common.py?
There is no trickery going on, qs is a regular string:
(Pdb) up
(Pdb) up
(Pdb) up
(Pdb) l
580 into Unicode characters, as accepted by the bytes.decode()
method.
581
582 Returns a list, as G-d intended.
583
584 qs, _coerce_result = _coerce_args(qs)
585 - pairs = [s2 for s1 in qs.split('') for s2 in s1.split(';')]
586 r = []
587 for name_value in pairs:
588 if not name_value and not strict_parsing:
589 continue
590 nv = name_value.split('=', 1)
(Pdb) whatis qs
class 'str'
(Pdb) p qs
''
(Pdb)
I have also tried to get a backtrace with the faulthandler module, but
it gives the same result:
Thread 0x7f7dafdb4700:
File /usr/lib/python3.3/cmd.py, line 126 in cmdloop
File /usr/lib/python3.3/pdb.py, line 318 in _cmdloop
File /usr/lib/python3.3/pdb.py, line 345 in interaction
File /usr/lib/python3.3/pdb.py, line 266 in user_line
File /usr/lib/python3.3/bdb.py, line 65 in dispatch_line
File /usr/lib/python3.3/bdb.py, line 47 in trace_dispatch
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py, line 693 in
clos
File /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py, line 853
in c
File /usr/lib/python3.3/urllib/parse.py, line 585 in listcomp
File /usr/lib/python3.3/urllib/parse.py, line 585 in parse_qsl
File /usr/lib/python3.3/urllib/parse.py, line 553 in parse_qs
File /home/nikratio/in-progress/s3ql/tests/mock_server.py, line 52 in
parse_url
File /home/nikratio/in-progress/s3ql/tests/mock_server.py, line 169 in
do_GET
File /usr/lib/python3.3/http/server.py, line 388 in handle_one_request
File /usr/lib/python3.3/http/server.py, line 402 in handle
File /home/nikratio/in-progress/s3ql/tests/mock_server.py, line 77 in handle
File /usr/lib/python3.3/socketserver.py, line 666 in __init__
File /usr/lib/python3.3/socketserver.py, line 345 in finish_request
File /usr/lib/python3.3/socketserver.py, line 610 in process_request_thread
File /usr/lib/python3.3/threading.py, line 858 in run
File /usr/lib/python3.3/threading.py, line 901 in _bootstrap_inner
File /usr/lib/python3.3/threading.py, line 878 in _bootstrap
Is it possible that the stack got somehow corrupted?
Does anyone have a suggestion how I could go about debugging this?
I am using Python 3.3.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
https://mail.python.org/mailman/listinfo/python-list
Missing stack frames?
Hello, (This may or may not be related to my mail about a corrupted stack trace). I've instrumented one of my unit tests with a conditional 'pdb.set_trace' in some circumstances (specifically, when a function is called by a thread other than MainThread). However, when trying to print a back trace to figure out how this function got called, I get this: $ py.test-3 tests/t1_backends.py -k extra === test session starts === platform linux -- Python 3.3.5 -- py-1.4.20 -- pytest-2.5.2 -- /usr/bin/python3 collected 33 items tests/t1_backends.py:563: test_extra_data[mock_s3c-plain] SKIPPED tests/t1_backends.py:563: test_extra_data[mock_s3c-zlib] PASSED 31 tests deselected by '-kextra' = === 1 passed, 1 skipped, 31 deselected in 0.23 seconds /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py(693)close() - if not self.md5_checked: (Pdb) bt /home/nikratio/in-progress/s3ql/src/s3ql/backends/common.py(853)close() - self.fh.close() /home/nikratio/in-progress/s3ql/src/s3ql/backends/s3c.py(693)close() - if not self.md5_checked: (Pdb) q What does this mean? Why is there no caller above the backends/common.py code? At the very least, I would have expected some frames from threading.py...? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- https://mail.python.org/mailman/listinfo/python-list
[issue17277] incorrect line numbers in backtrace after removing a trace function
Changes by Nikolaus Rath nikol...@rath.org: -- nosy: +nikratio ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17277 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20177] Derby #8: Convert 28 sites to Argument Clinic across 2 files
Nikolaus Rath added the comment: Tal, I was referring to this mail: https://mail.python.org/pipermail/python-dev/2014-January/132066.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20177 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21555] gcmodule.c could use pytime.h
Changes by Nikolaus Rath nikol...@rath.org: -- nosy: +nikratio ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21555 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: On 05/13/2014 12:41 PM, Serhiy Storchaka wrote: Note that this is not work with the punycode encoding (and may be some third-party encodings). I do not understand. Could you elaborate? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21057 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Nikolaus Rath added the comment: Cybjit rep...@bugs.python.org writes: Cybjit added the comment: On 2014-05-10 00:23, nikratio wrote: Is pip maybe doing its own certificate check, and relying on HTTPSConnection.host to contain the final hostname rather than the proxy? I think the culprit might be here https://github.com/pypa/pip/blob/1.5.4/pip/_vendor/requests/packages/urllib3/connection.py#L172 Yes, that's the problem. I guess that nicely demonstrates why using inheritance as an API is not a good idea. I guess we nevertheless need to repair/work around this in Python 3.4? Unfortunately pip explicitly relies on _set_tunnel() to set self.host = self._tunnel_host. So we would need to change _set_tunnel() to save the original attribute somewhere, change the other methods to use the saved attribute in favor of the real one, and have connect() restore it (so that we can reconnect). This still would not allow pip to reconnect (because it overwrites the connect method), but then reconnecting a tunneled connection with pip did not work before either. Still, rather ugly. Alternatively, maybe we could also do nothing, because if pip is depending on undocumented semantics of a private method (_set_tunnel), they have to live with the consequences? Thinking about this, I think we should just revert the entire patch for 3.4, but keep it in for 3.5. That gives the pip folks enough time to fix their code. Fixing the issue in 3.4 is probably not that crucial (after all, it existed since about 2.6.3). Best, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7776 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Nikolaus Rath added the comment: On 05/09/2014 02:02 PM, Cybjit wrote: C:\Python34\Scriptspip -v install simplejson Downloading/unpacking simplejson Could not fetch URL https://pypi.python.org/simple/simplejson/: connection err or: hostname 'openwrt.lan' doesn't match either of '*.c.ssl.fastly.net', 'c.ssl. This looks as if pip tries to match the hostname in the certificate from pypi.python.org against the hostname of the local proxy. Looking at the code, I don't see why it would do that though. HTTPSConnection.connect definitely tries to match against the final hostname. Is pip maybe doing its own certificate check, and relying on HTTPSConnection.host to contain the final hostname rather than the proxy? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7776 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19414] iter(ordered_dict) yields keys not in dict in some circumstances
Nikolaus Rath added the comment: Raymond, I think your patch does not really address the issue reported here. The dict documentation still guarantees that mutating a dict during iteration will raise a RuntimeError or may skip elements. The OrderedDict documentation still does not point out that OrderedDicts behave differently, yet they still raise a different exception than a regular dict in the same situation. I believe it should be possible to pass an OrderedDict to any function expecting a regular dict. This is still not possible. But even if you think this is not desirable (or not worth the cost), could we at least *document* that OrderedDicts behave differently? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19414 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: Maybe. I have 1.0.1g. Could you maybe post the output of the failed test? I'd like to understand how the patch broke the test (looking at your patch alone didn't tell me much). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20951 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file35123/issue_21377_r4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21377 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file35100/issue_21377_r3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21377 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: Antoine, are you sure this was a problem related to this patch? The test seems to work just fine for me: $ hg update -C -r b0f6983d63df $ make clean $ ./configure --with-pydebug make -j1 $ ./python -m test -u network,urlfetch -j 8 test_poplib [1/1] test_poplib 1 test OK. Am I doing something wrong? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20951 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file35082/issue20951_r3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20951 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file35083/issue_21057_r3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21057 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Nikolaus Rath added the comment: Tentative patch attached. The test suite still passes, but I'm not sure if it actually exerts the new code path. Is there a standard way to test the C api? -- keywords: +patch Added file: http://bugs.python.org/file35084/issue_21377.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21377 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file35085/issue_21377_r2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21377 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: As discussed on python-dev, here is a patch that changes the behavior of send() and sendall() to raise SSLWant* exceptions instead of returning zero. -- Added file: http://bugs.python.org/file35062/issue20951_r2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20951 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10983] Errors in http.client.HTTPConnection class (python3)
Nikolaus Rath added the comment: This issue can be closed. The testcases have been added in 39ee3286d187. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10983 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment:
Attached is an updated patch that
- removes the code duplication in _pyio.BufferedIOBase
- adds an internal _readinto helper method to _pyio.BufferedReader that makes
the implementation similar to io.BufferedReader.
- implements _pyio.BuffereadReader.{readinto,readinto1} in terms of the new
helper method and, as a side effect, also increases their performance.
Performance of the _pyio implementation on my system is:
pre-patch:
readinto: 5.130e+00 seconds
readinto1 not available
post-patch:
readinto: 2.039e+00 seconds
readinto1: 1.995e+00 seconds
--
Added file: http://bugs.python.org/file34863/issue20578_r3.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20578
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file34864/benchmark_r3.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1738] Add match parameter to filecmp.dircmp to ignore using patterns
Nikolaus Rath added the comment: Updated patch to acknowledge original authors in Misc/ACKS. -- Added file: http://bugs.python.org/file34801/issue1738_r3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1738 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Nikolaus Rath added the comment: Refreshed patch. -- Added file: http://bugs.python.org/file34802/issue7776_r7_Py3.4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7776 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file34803/issue7776_r7_Py3.5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7776 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19414] iter(ordered_dict) yields keys not in dict in some circumstances
Nikolaus Rath added the comment: The patch applies cleanly to 3.4 and 3.5, not sure why the review link does not show up. I'm attaching the file again, maybe that helps. -- Added file: http://bugs.python.org/file34804/issue19414_r2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19414 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Here's a little script to estimate the performance difference between using read1 and readinto1 to read large amounts of data. On my system, I get: C readinto1: 4.960e-01 seconds C read1: 4.055e-01 seconds Python readinto1: 1.066e+00 seconds Python read1: 2.565e+00 seconds In other words, _pyio.BufferedReader.readinto1 is more than a factor of 2 faster than _pyio.BufferedReader.read1 and io.readinto1 is faster than io.read1 by about 20%. On its own, I think this would justify keeping an implementation of readinto1 in _pyio.BufferedReader instead of falling back on the default (that is implemented using read1). However, I believe that people who need performance are probably not using _pyio but io, so *my* argument for keeping it implemented in _pyio is to keep the implementations similiar. I found studying _pyio very helpful to understand the C code in io. If we implement BufferedReader.readinto1 in io, but not in _pyio.BufferedReader, this advantage would be reduced. That said, I am primary interested in getting readinto1 into io. So I'm happy to either extend the patch to also provide a fast readinto implementation for _pyio (to align it with io), or to remove the readinto1 implementation in _pyio. -- Added file: http://bugs.python.org/file34811/benchmark.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: (Rietveld is giving me errors, so I'm replying here) On 2014/04/13 02:22:23, loewis wrote: Again, why a separate implementation here? For performance reasons. Relying on the default implementation would fall back to using read1(), which means a new bytes object is created first. Hmm. a) if performance was relevant, it should apply to readinto() as well. I didn't even notice the readinto implementation was missing. But I agree, if we keep readinto1(), we should also add readinto(). b) if the code is duplicated for performance only, a comment should state that explicitly. I'm very sorry, but I still don't see which code in readinto1() is duplicate. You don't mean duplicate between io and _pyio, do you? c) to justify a performance argument, you should really provide hard numbers that demonstrate a performance gain justifying the code duplication. I posted a small benchmark to the issue tracker. Personally, I think the more important argument is to keep the Python and C implementations similar. It's really nice if you can look at _pyio to find out at least roughly what happens in io. (Yes, I did put performance first in my last reply, but only because I thought you were asking about readinto1 in general, rather than the additional Python implementation in _pyio.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Can you please extend your benchmark to also measure read and readinto? Yes - but I don't quite understand why it matters (if you need read1/readinto1, you cannot just use read/readinto instead). C readinto1: 4.638e-01 seconds C read1: 4.026e-01 seconds C readinto: 4.655e-01 seconds C read: 4.028e-01 seconds Python readinto1: 1.056e+00 seconds Python read1: 2.429e+00 seconds Python readinto: 1.895e+00 seconds Python read: 1.218e+00 seconds That shows that the Python readinto is definetely not up-to-par and could use improvement as well. Is that what you're getting at? I'm puzzled why you are treating readinto1 differently from readinto. Maybe this is why we seem to be talking past each other :-). I did not look or work on readinto at all. All I noticed is that there is a read1, but no readinto1. So I implemented a readinto1 as well as I could. -- Added file: http://bugs.python.org/file34812/benchmark.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file34792/issue15955_r3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Seems as if no one has an opinion on this at all: https://mail.python.org/pipermail/python-dev/2014-April/133739.html What next? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: My usecase is that I have a binary stream class that internally uses memoryviews. I would like to read text data from this stream and thus encapsulate it in a TextIOWrapper. Currently, TextIOWrapper (correctly) expects read() to return bytes and fails if it receives any other bytes-like object. I can change my custom class to internally convert to bytes, but this means that the data is needlessly copied around and affects every other consumer of the class as well. Changing TextIOWrapper to work with any bytes-like object is (as far as I can see) rather simple. It does not introduce any new branches in the code, and it does not change the behavior for bytes objects at all. It does, however, eliminate unnecessary memcopies for classes that do not internally work with bytes objects. Therefore, I was hoping this patch could be considered for inclusion. The MyByteStream example that I gave in the first message is useless. I merely included it as the smallest possible code fragment that currently does not work, but would work after the patch in an attempt to illustrate what I meant - but apparently it had the opposite effect. Thanks for considering! -- type: behavior - enhancement versions: -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21057 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Thanks for the review! Attached is a new patch. I was actually pretty careful to avoid any code duplication.. are you refering to the readinto1() implementations for BytesIO and BufferedReader in Lib/_pyio.py? -- Added file: http://bugs.python.org/file34793/issue20578_r2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20578 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: I've attached the second iteration of the patch. I've factored out a new function decompress_buf, and added two new (C only) instance variables input_buffer and input_buffer_size. I've tested the patch by enabling Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c and compiling --with-pydebug --without-pymalloc. Output is: $ valgrind --tool=memcheck --suppressions=Misc/valgrind-python.supp ./python -m test -R : -v test_lzma ==18635== Memcheck, a memory error detector ==18635== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==18635== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==18635== Command: ./python -m test -R : -v test_lzma ==18635== == CPython 3.5.0a0 (default:a8f3ca72f703+, Apr 11 2014, 21:48:07) [GCC 4.8.2] [] Ran 103 tests in 43.655s OK beginning 9 repetitions 123456789 [...] OK . 1 test OK. ==18635== ==18635== HEAP SUMMARY: ==18635== in use at exit: 2,328,705 bytes in 13,625 blocks ==18635== total heap usage: 2,489,623 allocs, 2,475,998 frees, 91,305,463,593 bytes allocated ==18635== ==18635== LEAK SUMMARY: ==18635==definitely lost: 0 bytes in 0 blocks ==18635==indirectly lost: 0 bytes in 0 blocks ==18635== possibly lost: 963,533 bytes in 1,334 blocks ==18635==still reachable: 1,365,172 bytes in 12,291 blocks ==18635== suppressed: 0 bytes in 0 blocks ==18635== Rerun with --leak-check=full to see details of leaked memory ==18635== ==18635== For counts of detected and suppressed errors, rerun with: -v ==18635== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2) When running the tests only once (no -R option), the number of possibly lost bytes is 544,068 in 1,131 blocks. When running without the patch, the number is 545,740 bytes in 1,134 blocks (for one iteration). I guess this means that Python is using interior pointers, so these blocks are not truly lost? -- Added file: http://bugs.python.org/file34789/issue15955_r2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15955 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Nikolaus Rath added the comment: Thanks for the commit! My intention is to fix the behavior itself for 3.5 (see issue 9521), so I think adding testcases for the old behavior in the meantime isn't necessary. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20375 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Changes by Nikolaus Rath nikol...@rath.org: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20375 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Nikolaus Rath added the comment: Thanks for your feedback! I've attached an updated patch. -- Added file: http://bugs.python.org/file34708/issue20375.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20375 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Changes by Nikolaus Rath nikol...@rath.org: Added file: http://bugs.python.org/file34709/issue20375.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20375 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
