[issue11258] ctypes: Speed up find_library() on Linux by 500%
New submission from Jonas H. jo...@lophus.org: (This applies to all versions of Python I investigated, although the attached patch is for Python 2.7) I wondered why `import uuid` took so long, so I did some profiling. It turns out that `find_library` wastes at lot of time because of this crazy regular expression in `_findSoname_ldconfig`. A quick look at the ldconfig source (namely, the print_cache routine which is invoked when you call `ldconfig -p`, http://sourceware.org/git/?p=glibc.git;a=blob;f=elf/cache.c#l127) confirmed my suspicion that the ldconfig's output could easily be parsed without such a regex monster. I attached two patches that fix this problem. Choose one! ;-) The ctypes tests pass with my fixes, and here comes some benchmarking: $ cat benchmark_ctypes.py from ctypes.util import find_library for i in xrange(10): for lib in ['mm', 'c', 'bz2', 'uuid']: find_library(lib) # Current implementation $ time python benchmark_ctypes.py real0m11.813s ... $ time python -c 'import uuid' real0m0.625s ... # With my patch applied $ cp /tmp/ctypesutil.py ctypes/util.py $ time python benchmark_ctypes.py real0m1.785s ... $ time python -c 'import uuid' real0m0.182s ... -- assignee: theller components: ctypes files: faster-find-library1.diff keywords: patch messages: 128910 nosy: jonash, theller priority: normal severity: normal status: open title: ctypes: Speed up find_library() on Linux by 500% type: performance versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file20808/faster-find-library1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Changes by Jonas H. jo...@lophus.org: Added file: http://bugs.python.org/file20809/faster-find-library2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Jonas H. jo...@lophus.org added the comment: (might also be related to http://bugs.python.org/issue11063) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Changes by Jonas H. jo...@lophus.org: Added file: http://bugs.python.org/file20874/faster-find-library1-py3k-with-escaped-name.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Jonas H. jo...@lophus.org added the comment: As far as I can tell, it doesn't matter. We're looking for the part after the = in any case - ignoring the ABI/architecture information - so the regex would chose the first of those entries. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Jonas H. jo...@lophus.org added the comment: Humm. Would be great to have the `ldconfig -p` output of such a machine... I can't get ldconfig to recognize 64-bit libraries on my 32-bit machines, so I have no output to test against... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11345] Fix a link in library/json docs
New submission from Jonas H. jo...@lophus.org: I guess this should be a link. -- assignee: docs@python components: Documentation files: fix-json-link.diff keywords: patch messages: 129629 nosy: docs@python, jonash priority: normal severity: normal status: open title: Fix a link in library/json docs versions: Python 3.3 Added file: http://bugs.python.org/file20932/fix-json-link.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11345 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Jonas H. jo...@lophus.org added the comment: the orig impl matches the abi_type at the beginning of the parentheses, yours simply ignores the abi_type (that should have caught my eye, but that regex looked so much like magic that I didn't try to make sense of it :-)) Same here. :) The version I attached seems to work for me. It's some kind of compromise -- basically it's the original regex but with the unneccessary, performance-decreasing cruft stripped away. btw, Jonas H. is perfectly fine - I don't care about being honored, I just want to `import uuid` without waiting forever. :-) -- Added file: http://bugs.python.org/file20935/faster-find-library1-py3k-with-escaped-name-try2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4600] __class__ assignment: new-style? heap? == confusing
Jonas H. jo...@lophus.org added the comment: Here comes a patch, changing the behaviour to: ./python -q class C: ... pass ... (1).__class__ = 1 Traceback (most recent call last): File stdin, line 1, in module TypeError: __class__ must be set to a class defined by a class statement, not 'int' object (1).__class__ = object Traceback (most recent call last): File stdin, line 1, in module TypeError: class__ must be set to a class defined by a class statement, not 'object' (1).__class__ = C Traceback (most recent call last): File stdin, line 1, in module TypeError: __class__ assignment: only for instances of classes defined by class statements -- keywords: +patch nosy: +jonash Added file: http://bugs.python.org/file20937/4600.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4600 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1731717] race condition in subprocess module
Changes by Jonas H. jo...@lophus.org: -- nosy: +jonash ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1731717 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11484] `with_traceback` in 2.7 docs but not implemented
New submission from Jonas H. jo...@lophus.org: Either a `BaseException.with_traceback` implementation is missing or the docs are wrong. http://docs.python.org/library/exceptions.html?highlight=with_traceback#exceptions.BaseException.with_traceback python3 -c 'print(with_traceback in dir(BaseException))' True python2 -c 'print(with_traceback in dir(BaseException))' False -- assignee: docs@python components: Documentation messages: 130760 nosy: docs@python, jonash priority: normal severity: normal status: open title: `with_traceback` in 2.7 docs but not implemented versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11484 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11258] ctypes: Speed up find_library() on Linux by 500%
Jonas H. jo...@lophus.org added the comment: *push* Any way to get this into the codebase? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11258 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix intersphinx-ing of built-in types (list, int, ...)
New submission from Jonas H. jo...@lophus.org: Intersphinx-ing of int, list, float, ... should work with :class:`int` (list, float, ...). Also, intersphinx-ing list methods, e.g. :meth:`list.insert`, should work. -- assignee: docs@python components: Documentation messages: 134923 nosy: docs@python, jonash priority: normal severity: normal status: open title: Fix intersphinx-ing of built-in types (list, int, ...) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11976] Provide proper documentation for list data type
New submission from Jonas H. jo...@lophus.org: Provide a proper `list` method reference (like the one for `dict`, http://docs.python.org/library/stdtypes.html#dict). Right now, documentation about lists is spread over multiple topics (.rst files) and methods are documented in footnotes. Also, intersphinx-ing and list methods is not possible -- :meth:`list.foo` does not create any links due to missing documentation. This is also related to #11975. -- assignee: docs@python components: Documentation messages: 134924 nosy: docs@python, jonash priority: normal severity: normal status: open title: Provide proper documentation for list data type ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11976 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11977] Document int.conjugate, .denominator, ...
New submission from Jonas H. jo...@lophus.org: Various `int` attributes and methods seem undocumented (at least it does not work to intersphinx them): * .conjugate * .denominator * .imag * .numerator * .real -- assignee: docs@python components: Documentation messages: 134926 nosy: docs@python, jonash priority: normal severity: normal status: open title: Document int.conjugate, .denominator, ... versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11977 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Actually I need to be able to intersphinx (because my documentation work is not the Python docs :-) but I guess it boils down to the same problem of incomplete Sphinx module/class indices. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Indeed they do; but documentation writers need to know that `int()` and `float()` are functions, which is counterintuitive. (and a CPython implementation detail) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Is this a problem in our markup or a bug in intersphinx? It's a markup problem -- those types are documented as functions, using the :func: role/`.. func::` directive. It's not only a markup mismatch but, strictly speaking, it's *wrong* documentation: str, int, ... aren't functions, they're *classes* and should be documented as such. It's a bit odd to search for information on the str *class* in the list of built-in *functions*. If this work within one documentation set (as I believe it does) It does not. For example, in http://docs.python.org/library/functions.html#max there's a reference to list.sort using :meth:`list.sort` but no link could be generated. How could it possibly work without decent documentation about the list data type? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11977] Document int.conjugate, .denominator, ...
Jonas H. jo...@lophus.org added the comment: It doesn't. Sphinx still can't make any links, which btw also means that it's impossible to reference those methods within the Python documentation. Also I want to point out that I find the information very hard to find as a human. The fact that integers are based on `numbers.Number` is -- at this point in time where 95% of all Python developers don't know about the `numbers` module or abstract base classes in general -- an implementation detail and as such should not affect the way `int` is documented. I propose to have decent class references for int, str, ... similar to the reference for dict -- that is, document all attributes and methods in one place and make them referencable. For those who want to deep-dive into CPython internals, a note about those functionality actually being implemented in ABCs could be added. (But I think that's out of scope for this ticket. I could open a new one if anyone agrees with me... :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11977 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Shouldn't have used decent here, sorry. What I was trying to say is that there's no reference-like documentation for the list datatype (as for dict). There's more than enough quality documentation about lists but I think the way it's arranged can be improved. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Does that look good to you? If it does, I'll go on using the script (http://paste.pocoo.org/show/396661/) on the 3.x docs. -- keywords: +patch Added file: http://bugs.python.org/file22164/p1.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Linking a class using a function directive is counter-intuitive. That's why we need to make use of class directives rather than function directives here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: I'm not. My patch doesn't address the problem of unlinkable methods but wrong type declarations (read, wrong usage of .. function:: directives) for builtins like int, float, bool, list etc. Because the directives change, the roles used to link to them (:func:`list`) have to be changed accordingly. That's what this patch does. I want to address `list` method documentation in the next step. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Could you make an effort to accept our word that using :class: instead of :func: would bring zero value to the indexing system nor to human readers? I'm already doing; but I don't see anyone having made a good point against my preference of using .. class:: to document classes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: What's wrong with the changes I propose with the patch, then? Sorry, I really don't get it, no matter how hard I try. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: when you mark up something with a mod, func, class or meth role, Sphinx will find the target without paying attention to its type. So changing :func: to :class: does not bring anything. From a quick test this seems to hold true for links within one project but not for Sphinx' intersphinx extension, which actually cares about types. So the question is whether we keep CPython implementation details (many builtins being both a class and a function) out of the documentation or we get the Sphinx developers to change intersphinx behaviour. I guess you'd suggest the latter, right? :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: So the intersphinx behavior is the correct one, but we can't change the other now because of compatibility. Could you be convinced to use that legacy behaviour for intersphinx, too? :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Jonas, I owe you an apology [...] Thanks Éric, I got a bit worried about getting on your nerves... Based on Ezio's idea: What happens if we have both a .. function:: foo and .. class:: foo -- where do :func:`foo` and :class:`foo` link to (internally and using intersphinx)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11975] Fix referencing of built-in types (list, int, ...)
Jonas H. jo...@lophus.org added the comment: Having one page with two objects of the same name, e.g. .. function:: foo .. class:: foo renders to two entries with the same anchor name (#foo). The first entry gets a link-to-this-paragraph marker, the second one doesn't. Internal references (from within the same document) always link to the first entry because they use #foo anchor. (So if you put the class directive first, all links go to the class anchor.) The first external reference (using intersphinx) always goes to the first target document element - no matter which type both have. The second reference isn't turned into a hyperlink. This behaviour seems consistent with how HTML anchors work. Having the two objects on two different pages however shows slightly odd results. Say we have this code on page 1: .. class:: foo :class:`foo` :func:`foo` and .. function:: foo on page 2, then both links in page 1 go to the page 1 'foo' (the class). However if you change order (putting the func role before the class role), those links go to the page 2 'foo' (the function). All intersphinx-ed links go to the object on page 1, no matter the role order on page 1 or the external page. I think we can conclude that using class and function directives at the same time doesn't help very much... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11975 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12284] argparse.ArgumentParser: usage example option
New submission from Jonas H. jo...@lophus.org: I'd like to see an `examples` option added to argparse.ArgumentParser as found in many man pages. This could also be done using the `epilog` option, but that misses the %(proc)s replacement which makes usage examples like this Example usage: ./script.py option1 option2 impossible. -- components: Library (Lib) messages: 137905 nosy: jonash priority: normal severity: normal status: open title: argparse.ArgumentParser: usage example option type: feature request versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12284 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12284] argparse.ArgumentParser: usage example option
Jonas H. jo...@lophus.org added the comment: Nope. I want an examples section, for example from `man git log`: EXAMPLES git log --no-merges Show the whole commit history, but skip any merges git log v2.6.12.. include/scsi drivers/scsi Show all commits since version v2.6.12 that changed any file in the include/scsi or drivers/scsi subdirectories ... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12284 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12877] Popen(...).stdout.seek(...) throws Illegal seek
New submission from Jonas H. jo...@lophus.org: from subprocess import Popen, PIPE p = Popen(['ls'], stdout=PIPE) p.wait() p.stdout.seek(0) Traceback (most recent call last): File t.py, line 5, in module p.stdout.seek(0) IOError: [Errno 29] Illegal seek Python 2.7.2, Arch Linux x86-64 (Kernel 3.0) -- messages: 143323 nosy: jonash priority: normal severity: normal status: open title: Popen(...).stdout.seek(...) throws Illegal seek versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12877] Popen(...).stdout.seek(...) throws Illegal seek
Jonas H. jo...@lophus.org added the comment: Why does it have a 'seek' method then? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18179] SMTP.local_hostname is undocumented
New submission from Jonas H.: The Sphinx docs don't contain any explanation for `local_hostname`. -- assignee: docs@python components: Documentation messages: 190898 nosy: docs@python, jonash priority: normal severity: normal status: open title: SMTP.local_hostname is undocumented versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18179 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19767] pathlib: iterfiles() and iterdirs()
New submission from Jonas H.: From my personal experience, listing all real files and all subdirectories in a directory is a very common use case. Here's a patch that adds the `iterfiles()` and `iterdirs()` methods as a shortcut for `[f for f in p.iterdir() if f.is_dir/file()]` . -- components: Library (Lib) files: iterdirs_iterfiles.diff keywords: patch messages: 204320 nosy: jonash priority: normal severity: normal status: open title: pathlib: iterfiles() and iterdirs() type: enhancement versions: Python 3.4 Added file: http://bugs.python.org/file32832/iterdirs_iterfiles.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19767 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24358] Should compression file-like objects provide .fileno(), misleading subprocess?
Jonas H. added the comment: I just hit this too. I'd say remove the fileno() method from wrapper objects like GzipFile. I'm happy to submit a patch. -- nosy: +jonash ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24358> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: https://github.com/python/cpython/pull/4589 - Add 3.7 What's New entry - Fix regression (thanks Tim for the report) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Change by Jonas H. <jo...@lophus.org>: -- pull_requests: +4513 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: Ah, the problem isn't that it's running getattr() on test methods, but that it runs getattr() on all methods. Former code: attrname.startswith(prefix) and \ callable(getattr(testCaseClass, attrname)) New code: testFunc = getattr(testCaseClass, attrname) isTestMethod = attrname.startswith(self.testMethodPrefix) and callable(testFunc) This is trivial to fix. @Core devs: Should I revert to original behaviour with the order of the prefix check and the getattr() call, and add a regression test that guarantees this behaviour? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: Sure! -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: > > 3) Is the approach of dynamically wrapping 'skip()' around to-be-skipped > > test cases OK? > I think this is the wrong approach. A test that isn't selected shouldn't be > skipped, it should not appear in the output at all. Another reason for > putting this in TestLoader :-) My first implementation actually was mostly the test loader. Two things made me change my mind and try to make the changes in the suite code: - The loader code really only deals with loading (i.e., finding + importing) tests. Yes it expects a file pattern like "test*.py" for identifying test case files. But apart from that it didn't "feel" right to put name based selection there. - In py.test you'll get a console output like "5 tests passed, 1 test failed, 12 tests deselected". We can't get anything similar without making bigger changes to the test loader, runner, etc. code. Using skip() we at least have some info on "skipped" tests, although they're technically not skipped. Are you still saying this should go to the test loader? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: Thanks Antoine. I will need some guidance as to what are the correct places to make these changes. I'm not quite sure about the abstractions here (runner, loader, suite, case, etc.) My PoC (see GitHub link in first post) uses a TestSuite subclass. (The subclass is only so that it's easier to assess the general implementation approach; I guess it should be put into the main class instead.) Things I'm unsure of: 1) Is suite the correct place for this kind of feature? 2) Is the hardcoded fnmatch-based pattern matcher ok, or do we need a new abstraction "NameMatcher"? 3) Is the approach of dynamically wrapping 'skip()' around to-be-skipped test cases OK? 4) The try...catch statement around 'test.id()' is needed because there are some unit tests (unit tests for the unittest module itself) that check for some error cases/error handling in the unittest framework, and crash if we try to call '.id()' on them. Please remove the try...catch to see these errors if you're interested in the details. Is the check OK like that, or is this a code smell? Thanks Jonas -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: Just to be clear, the current implementation is limited to substring matches. It doesn't support py.test like "and/or" combinators. (Actually, py.test uses 'eval' to support arbitrary patterns.) So say we have test case SomeClass test_foo test_bar Then - python -m unittest -k fo matches "test_foo" - python -m unittest -k Some matches "test_foo" and "test_bar" - python -m unittest -k some matches nothing The -k option may be used multiple times, combining the patterns with "or": - python -m unittest -k fo -k b matches "test_foo" and "test_bar" It's also possible to use glob-style patterns, like -k "spam_*_eggs". -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
New submission from Jonas H. <jo...@lophus.org>: I'd like to add test selection based on parts of the test class/method name to unittest. Similar to py.test's "-k" option: https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name Here's a proof of concept implementation: https://github.com/jonashaag/cpython/compare/master...unittest-select Is this something others find useful as well? If so, I'd like to work on getting this into Python stdlib proper. This is my first time contributing to the unittest framework; is the general approach taken in my PoC implementation correct in terms of abstractions? How can I improve the implementation? Jonas -- components: Library (Lib) messages: 306490 nosy: jonash priority: normal severity: normal status: open title: Add py.test-like "-k" test selection to unittest type: enhancement ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32071] Add py.test-like "-k" test selection to unittest
Jonas H. <jo...@lophus.org> added the comment: Interesting, Victor. I've had a look at the code you mentioned, but I'm afraid it doesn't really make sense to re-use any of the code. Here's a new patch, implemented in the loader as suggested by Antoine, and with tests. I'm happy to write documentation etc. once we're through with code review. https://github.com/python/cpython/pull/4496 -- keywords: +patch pull_requests: +4433 stage: needs patch -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31526] Allow setting timestamp in gzip-compressed tarfiles
Jonas H. <jo...@lophus.org> added the comment: This affects me too. -- nosy: +jonash ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32679] concurrent.futures should store full sys.exc_info()
Jonas H. <jo...@lophus.org> added the comment: See also https://stackoverflow.com/questions/19309514/getting-original-line-number-for-exception-in-concurrent-futures for other people having the same problem -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32679> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32679] concurrent.futures should store full sys.exc_info()
New submission from Jonas H. <jo...@lophus.org>: Use case: Try to get a future's result using concurrent.futures.Future.result(), and log the full exception if there was any. Currently, only "excinst" (sys.exc_info()[1]) is provided with the Future.exception() method. Proposal: Add new Future.exc_info() method that returns the full sys.exc_info() at the time of the exception. -- components: Library (Lib) messages: 310762 nosy: jonash priority: normal severity: normal status: open title: concurrent.futures should store full sys.exc_info() type: enhancement versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32679> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] django: segmentation fault on garbage collection in visit_decref()
Jonas H. added the comment: I can reproduce this on Ubuntu 18.04. INADA, I have a full gdb backtrace with Python 3.7 development build. I'd like to share it with you privately as I'm concerned it may contain sensible information. I know that's a bit unconventional; if you have other suggestions I'm happy to follow along. -- ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] django: segmentation fault on random places
Jonas H. added the comment: Here's a Docker image that reproduces the bug. FROM ubuntu:18.04 RUN apt update && apt install -y python3.7-dbg python3.7-venv python3-venv wget RUN python3.7 -m venv venv RUN venv/bin/pip install django RUN wget https://bugs.python.org/file47688/testproj.tar.gz -O - | tar xfz - CMD cd /testproj && /venv/bin/python manage.py runserver & sleep 5; wget -t1 localhost:8000 >/dev/null 2>/dev/null Of course this also works outside Docker. I have reproduced with macOS 10.13.5 (17F77) and Ubuntu 18.04 (Docker). On macOS it's Python 3.7.0 (default, Jun 29 2018, 20:13:13) [Clang 9.1.0 (clang-902.0.39.2)] on darwin, installed from Homebrew. On Ubuntu it's whatever is in the 18.04 repositories. -- ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] django: segmentation fault on random places
Jonas H. added the comment: Reduced it to something that seems unicode related? No extension modules involved. Vanilla Django project with a single url + template. See testproj/urls.py and tmpl/index.html -- Added file: https://bugs.python.org/file47688/testproj.tar.gz ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] django: segmentation fault on random places
Jonas H. added the comment: Sure. Unpack archive, create new 3.7 venv with Django (latest version is fine), ./manage.py runserver, curl localhost:8000. -- ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] int(s), float(s) and others may cause segmentation fault
Jonas H. added the comment: I don't think this can be tested with Python code, unless you can make sure the target buffer _PyUnicode_TransformDecimalAndSpaceToASCII operates on is initialised with garbage bytes. -- ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] int(s), float(s) and others may cause segmentation fault
Jonas H. added the comment: The assertion in the patched code, yes. The segfault in the unpatched code, no. -- ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] django: segmentation fault on garbage collection in visit_decref()
Jonas H. added the comment: I also have a segfault that goes away with malloc debugging. Not sure if it's the same issue. My extension modules are venv/lib/python3.7/site-packages//_yaml.cpython-37m-darwin.so venv/lib/python3.7/site-packages//netifaces.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_webp.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingft.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingcms.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imaging.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingmath.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingtk.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingmorph.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/builder.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/_elementpath.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/html/diff.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/html/clean.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/etree.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/objectify.cpython-37m-darwin.so venv/lib/python3.7/site-packages//coverage/tracer.cpython-37m-darwin.so Unfortunately I can't test the application without any of these, but maybe we can match my list with fenrrir's. -- nosy: +jonash ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34087] django: segmentation fault on garbage collection in visit_decref()
Jonas H. added the comment: Btw my segfault is from Django too, but that may just be a coincidence -- ___ Python tracker <https://bugs.python.org/issue34087> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45462] Speed up re.match with pre-compiled patterns
Change by Jonas H. : -- keywords: +patch pull_requests: +27224 stage: -> patch review pull_request: https://github.com/python/cpython/pull/28936 ___ Python tracker <https://bugs.python.org/issue45462> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45462] Speed up re.match with pre-compiled patterns
New submission from Jonas H. :
re.match(p, ...) with a pre-compiled pattern p = re.compile(...) can be much
slower than calling p.match(...). Probably mostly in cases with "easy" patterns
and/or short strings.
The culprit is that re.match -> re._compile can spend a lot of time looking up
p its internal _cache, where it will never find p:
def _compile(pattern, flags):
...
try:
return _cache[type(pattern), pattern, flags]
except KeyError:
pass
if isinstance(pattern, Pattern):
...
return pattern
...
_cache[type(pattern), pattern, flags] = p
...
_compile will always return before the _cache is set if given a Pattern object.
By simply reordering the isinstance(..., Pattern) check we can safe a lot of
time.
I've seen speedups in the range of 2x-5x on some of my data. As an example:
Raw speed of re.compile(p, ...).match():
time ./python.exe -c 'import re'\n'pat = re.compile(".").match'\n'for _ in
range(1_000_000): pat("asdf")'
Executed in 190.59 millis
Speed with this optimization:
time ./python.exe -c 'import re'\n'pat = re.compile(".")'\n'for _ in
range(1_000_000): re.match(pat, "asdf")'
Executed in 291.39 millis
Speed without this optimization:
time ./python.exe -c 'import re'\n'pat = re.compile(".")'\n'for _ in
range(1_000_000): re.match(pat, "asdf")'
Executed in 554.42 millis
--
components: Regular Expressions
messages: 403851
nosy: ezio.melotti, jonash, mrabarnett
priority: normal
severity: normal
status: open
title: Speed up re.match with pre-compiled patterns
type: performance
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue45462>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45462] Speed up re.match with pre-compiled patterns
Jonas H. added the comment:
I agree with your statement in principle. Here are numbers for the slowdown
that's introduced:
Without the change:
./python.exe -m timeit -s 'import re'\n'[re.compile(f"fill_cache{i}") for i
in range(512)]'\n'pat = re.compile(".")' 're.match(pat, "asdf")'
50 loops, best of 5: 462 nsec per loop
./python.exe -m timeit -s 'import re'\n'[re.compile(f"fill_cache{i}") for i
in range(512)]'\n'pat = re.compile(".")' 're.match(".", "asdf")'
100 loops, best of 5: 316 nsec per loop
With the change:
./python.exe -m timeit -s 'import re'\n'[re.compile(f"fill_cache{i}") for i
in range(512)]'\n'pat = re.compile(".")' 're.match(pat, "asdf")'
100 loops, best of 5: 207 nsec per loop
./python.exe -m timeit -s 'import re'\n'[re.compile(f"fill_cache{i}") for i
in range(512)]'\n'pat = re.compile(".")' 're.match(".", "asdf")'
100 loops, best of 5: 351 nsec per loop
So we have a 2x speedup in the uncommon case and a 10% slowdown in the common
case.
--
___
Python tracker
<https://bugs.python.org/issue45462>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45462] Speed up re.match with pre-compiled patterns
Jonas H. added the comment: pat.match() has 110 nsec. Feel free to close the issue and PR if you think this isn't worth changing. -- ___ Python tracker <https://bugs.python.org/issue45462> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
