[issue34232] Python3.7.0 exe installers (32 and 64 bit) failing on Windows7
Wolfgang Maier added the comment: Oh, sorry, I didn't realize there was another file and it seems I did not keep it so I just ran the installer again to reproduce. Attached is the new pair of log files. -- Added file: https://bugs.python.org/file47722/Python 3.7.0 (64-bit)_20180731180657_000_core_JustForMe.log ___ Python tracker <https://bugs.python.org/issue34232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34232] Python3.7.0 exe installers (32 and 64 bit) failing on Windows7
Change by Wolfgang Maier : Added file: https://bugs.python.org/file47723/Python 3.7.0 (64-bit)_20180731180657.log ___ Python tracker <https://bugs.python.org/issue34232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34232] Python3.7.0 exe installers (32 and 64 bit) failing on Windows7
Change by Wolfgang Maier : Added file: https://bugs.python.org/file47712/Python 3.7.0 (64-bit)_20180726120531.log ___ Python tracker <https://bugs.python.org/issue34232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34232] Python3.7.0 exe installers (32 and 64 bit) failing on Windows7
New submission from Wolfgang Maier : System: Windows7 Enterprise SP1 64-bit Downloaded the executable installer from python.org (tried both 32- and 64-bit -> same error) Selected the default user-install and got an almost immediate Error message: The TARGETDIR variable must be provided when invoking this installer. Clicking OK reveals 0x80070643 - Fatal error during installation and the attached log file. Rebooting didn't help and I also tried playing with custom installation settings, too, but no success so far. -- components: Installation messages: 322409 nosy: steve.dower, wolma priority: normal severity: normal status: open title: Python3.7.0 exe installers (32 and 64 bit) failing on Windows7 versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue34232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: Try to think of it this way: By choosing a default of True, every new project with subparsers that aims for Python <3.7 compatibility will have to take some measures (either overwrite the default or special-case 3.3-3.6). So no matter whether this is the "least surprising" choice, it is an inconvenient one that makes the default almost useless for years to come. In the long term, when support for Python<=3.6 is finally not important anymore, you would get a slightly more consistent API (though I never thought of a subparser as a regular positional argument before this issue), but the price for it seems too high to me. Since backwards compatibility is easy to restore by overwriting the default, I can certainly live with the choice of True, but I think it's not the best one could get out of this new and useful keyword. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24068] statistics module - incorrect results with boolean input
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: Fixed as part of resolving issue 25177. -- resolution: -> fixed stage: test needed -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24068> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25177] OverflowError in statistics.mean when summing large floats
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: Steven's commit here also fixed issue 24068. -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue25177> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33228] Use Random.choices in tempfile
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: Actually, in Python2.7 random.choice is implemented with the same susceptibility to the rounding bug as Python3's choices, still nobody ever reported a tempfile IndexError problem (I guess). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33228> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33228] Use Random.choices in tempfile
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: sorry, should have been issue 24567, of course. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33228> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33228] Use Random.choices in tempfile
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: @serhiy as I understand issue 33228, the double rounding problem potentially causing an IndexError can only affect choices() if the len of the sequence to choose from is greater than 2049, but the string in question here is only 37 characters long. Alternatively, choices may fail with certain weights (https://bugs.python.org/msg275594), but _RandomNameSequence is not using weights. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33228> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33228] Use Random.choices in tempfile
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- keywords: +patch pull_requests: +6093 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33228> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33228] Use Random.choices in tempfile
New submission from Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: A rather trivial change: tempfile._RandomNameSequence could make use of random.Random.choices introduced in 3.6. IMO, the suggested change would give clearer and also faster code. It also updates the docstring of the class, which currently talks about a six-character string, when an eight-character string gets returned. -- messages: 314969 nosy: wolma priority: normal severity: normal status: open title: Use Random.choices in tempfile type: performance versions: Python 3.8 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33228> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33193] Cannot create a venv on Windows when directory path contains dollar character
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment:
An exotic case, but it also affects Linux:
python3.7 -m venv 'at$test'
Error: Command '['/home/maier/at$test/bin/python3.7', '-Im', 'ensurepip',
'--upgrade', '--default-pip']' returned non-zero exit status 2.
[maier@nb19 ~]$ mkdir 'at$test'
mkdir: cannot create directory ‘at$test’: File exists
[maier@nb19 ~]$ cd 'at$test'
[maier@nb19 at$test]$ bin/python -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Exception:
Traceback (most recent call last):
File "/usr/lib64/python3.7/distutils/util.py", line 187, in subst_vars
return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
File "/usr/lib64/python3.7/re.py", line 198, in sub
return _compile(pattern, flags).sub(repl, string, count)
File "/usr/lib64/python3.7/distutils/util.py", line 184, in _subst
return os.environ[var_name]
File "/usr/lib64/python3.7/os.py", line 678, in __getitem__
raise KeyError(key) from None
KeyError: 'test'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py",
line 215, in main
status = self.run(options, args)
File
"/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line
342, in run
prefix=options.prefix_path,
File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py",
line 784, in install
**kwargs
File
"/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line
851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File
"/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line
1064, in move_wheel_files
isolated=self.isolated,
File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line
247, in move_wheel_files
prefix=prefix,
File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/locations.py", line
153, in distutils_scheme
i.finalize_options()
File "/usr/lib64/python3.7/distutils/command/install.py", line 307, in
finalize_options
self.expand_basedirs()
File "/usr/lib64/python3.7/distutils/command/install.py", line 486, in
expand_basedirs
self._expand_attrs(['install_base', 'install_platbase', 'root'])
File "/usr/lib64/python3.7/distutils/command/install.py", line 480, in
_expand_attrs
val = subst_vars(val, self.config_vars)
File "/usr/lib64/python3.7/distutils/util.py", line 189, in subst_vars
raise ValueError("invalid variable '$%s'" % var)
ValueError: invalid variable '$'test''
So the venv actually gets created, but it's ensurepip which chokes on the $.
--
nosy: +wolma
versions: +Python 3.7, Python 3.8
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33193>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33203] random.choice: raise IndexError on empty sequence even when not using getrandbits internally
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: @selik: it's true _randbelow doesn't work for negative numbers, but the difference is that both branches are affected, the docstring does not make any guarantees about it, and no public part of the random module is affected by this behavior. In addition, "fixing" _randbelow for negative input cannot be done without impacting performance of several public methods of random.Random so I don't think this should be done. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33203> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33203] random.choice: raise IndexError on empty sequence even when not using getrandbits internally
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: @rhettinger: the reason the ValueError gets raised correctly in the getrandbits-dependent branch is because getrandbits itself does a n<=0 check (in C for random.Random, in Python for random.SystemRandom). So I thought the real reason why _randbelow does not perform the check was that it would be redundant, which it isn't when only random is used. So instead of thinking about the suggested check as something that impacts performance (which certainly it does), I would rather see it as adding something into that second branch that was forgotten and gave that branch a performance benefit over the other one, which it never deserved. It is also worthwhile to remember that any performance impact of this will only hit subclasses of random.Random that define random, but not getrandbits. Your alternative idea of having random.choice catch (ValueError, ZeroDivisionError) would affect random.Random and all subclasses. If a subclass defines getrandbits, it needs to perform that check anyway and, thinking about this more, I suggest that the requirement for any user-provided getrandbits to raise ValueError on k <= 0 should actually be added to the getrandbits docs. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33203> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: ok, I've created issue 33203 to deal with raising ValueError in _randbelow consistently. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33203] random.choice: raise IndexError on empty sequence even when not using getrandbits internally
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- keywords: +patch pull_requests: +6050 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33203> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33203] random.choice: raise IndexError on empty sequence even when not using getrandbits internally
New submission from Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>:
from https://docs.python.org/3/library/random.html#random.choice:
Return a random element from the non-empty sequence seq. If seq is empty,
raises IndexError.
Indeed:
>>> import random
>>> random.choice([])
Traceback (most recent call last):
File "", line 1, in
File "/home/wolma/cpython/random.py", line 259, in choice
raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence
but when not using getrandbits internally:
>>> class MyRandom(random.Random):
... def random(self):
... return super().random()
...
>>> my_random=MyRandom()
>>> my_random.choice([])
Traceback (most recent call last):
File "", line 1, in
File "/home/wolma/cpython/random.py", line 257, in choice
i = self._randbelow(len(seq))
File "/home/wolma/cpython/random.py", line 245, in _randbelow
rem = maxsize % n
ZeroDivisionError: integer division or modulo by zero
This is because the ValueError that random.choice tries to catch gets raised
only in the getrandbits-dependent branch of Random._randbelow, but not in the
branch using only Random.random (even though Random._randbelow suggests uniform
behaviour.
--
components: Library (Lib)
messages: 314787
nosy: rhettinger, serhiy.storchaka, wolma
priority: normal
severity: normal
status: open
title: random.choice: raise IndexError on empty sequence even when not using
getrandbits internally
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33203>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: In addition, I took the opportunity to fix a bug in the original _randbelow in that it would only raise the advertised ValueError on n=0 in the getrandbits-dependent branch, but ZeroDivisionError in the pure random branch. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: So, the PR implements the behaviour suggested by Serhiy as his cases 1 and 2. Case 2 changes *existing* behaviour because before it was sufficient to have a user-defined getrandbits anywhere in the inheritance tree, while with the PR it has to be more recent (or be defined within the same class) as the random method. I'm not 100% sold on this particular aspect so if you think the old behaviour is better, then that's fine with me. In most real situations it would not make a difference anyway (or do people build complex inheritance hierarchies on top of random.Random?). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- pull_requests: +6015 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: Thanks, Raymond. I'll do that once I've addressed Serhiy's points. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: Serhiy: > I like the idea in general, but have comments about the implementation. > > __init_subclass__ should take **kwargs and pass it to > super().__init_subclass__(). type(cls.random) is not the same as > type(self.random). I would use the condition `cls.random is > _random.Random.random` instead, or check if the method is in cls.__dict__. > > This will break the case when random or getrandbits methods are patched after > class creation or per instance, but I think we have no need to support this. > My bad, sorry, and thanks for catching all these issues! You're absolutely right about the class type checks not being equivalent to the original ones at the instance level. Actually, this is due to the fact that I first moved the checks out of _randbelow and into __init__ just as Raymond would have done and tested this, but then I realized that __init_subclass__ looked just like the right place and moved them again - this time without testing on derived classes again. From a quick experiment it looks like types.MethodDescriptorType would be the correct type to check cls.random against and types.FunctionType would need to be checked against cls.getrandbits, but that starts to look rather esoteric to me - so you are probably right that something with a cls.__dict__ check or the alternative suggestion of `cls.random is _random.Random.random` are better solutions, indeed. > We could support also the following cases: > > 1. > class Rand1(Random): > def random(self): ... > # _randbelow should use random() > > class Rand2(Rand1): > def getrandbits(self): ... > # _randbelow should use getrandbits() > # this is broken in the current patch > Right, hadn't thought of this situation. > 2. > class Rand1(Random): > def getrandbits(self): ... > # _randbelow should use getrandbits() > > class Rand2(Rand1): > def random(self): ... > # _randbelow should use random() > # this is broken in the current code > May be worth fixing, too. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33144] random._randbelow optimization
New submission from Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: Given that the random module goes a long way to ensure optimal performance, I was wondering why the check for a match between the random and getrandbits methods is performed per call of Random._randbelow, when it could also be done at instantiation time (the attached patch uses __init_subclass__ for that purpose and, in my hands, gives 10-25% speedups for calls to methods relying on _randbelow). Is it really necessary to guard against someone monkey patching the methods rather than using inheritance? -- components: Library (Lib) files: randbelow.patch keywords: patch messages: 314455 nosy: rhettinger, wolma priority: normal severity: normal status: open title: random._randbelow optimization type: performance versions: Python 3.8 Added file: https://bugs.python.org/file47501/randbelow.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: _wants_ is a bit a strong word, but, at least, you can do a bit a nicer job than the default error, like printing a nicely formatted list of subcommands as you would get it with the main parsers help. In fact, in my own code I'm actually catching the missing subparser situation, then calling parse_args again with ['--help'] :) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: On 03/20/2018 04:38 PM, Anthony Sottile wrote: > > Anthony Sottile <asott...@umich.edu> added the comment: > > The intention of the change in issue 26510 was to pick the least surprising > behaviour for the default value of subparsers -- the compatiblity with the > behaviour before the regression was introduced in 3.3 was a nice side-effect. > As with the rest of positional arguments in argparse, the positional > subparsers were changed to required by default. > Since the 3.3 change happened a long time ago and has been kept through the next three releases, I never considered it a regression, but rather thought the original behavior was an oddity of early Python 3s (I've never written an argparse-based parser in Python 2), so it's interesting to see this in the larger historical context. Overall, I think "least surprising" is in the eye of the beholder here and I want to emphasize that I'm all for your change of adding the keyword argument, just not so convinced about your choice of the default. My main argument for a default of False and against True: having True as the default will only lead people used to the Python 2 and pre-3.3 way of things to think that they have code working for Python 3, when, in fact, that code will break under 3.3-3.6, and, at least, 3.6 will stay in widespread use for a long time (even Ubuntu 18.04 will still ship with it as the default python3, so Python3.6 will outlast the life cycle of Python 2 by a good measure). Conversely, most projects are dropping Python 3.2 support these days or have done so already, so nobody really cares about how things worked in this version (I think it's telling along these lines that your - corrected - SO link dates back from 2013). So I think, it is a rather unnecessary incompatibility you are introducing for project maintainers even if the original issue was a regression. > The main issue addressing the 3.3 regression I believe is > https://bugs.python.org/issue9253 and not the one linked. > > When I revived the patch, I surveyed a number of open source tools using > subparsers (~10-20) and they all fell into the following categories: > > - Used the workaround (part of this SO post: > https://stackoverflow.com/a/23354355/812183) (most fell into this category) > - crashed with some sort of TypeError (NoneType has no attribute startswith, > KeyeError: None, etc.) due to not handling "optional" subparsers > - Manually handled when the subparser was `None` to raise an argparse error As yet another option, and similar to the third one on your list, I'm using the set_defaults method of the subparser, and I'm checking whether the corresponding key is present in the Namespace. > > You can enable a 3.3-3.7 compatible "always optional subparsers" with a > similar pattern that was used to manually restore the pre-regression > behaviour: > > subparsers = parser.add_subparsers(...) > subparsers.required = False > Ah, right! That's a good option. Didn't realize it would work this way, too :) But a still think you should consider my above argument. > I believe the error message issue is already tracked: > https://bugs.python.org/issue29298 > I see, that looks as if it would fix this part. It would be great if it could get merged into Python 3.7 still. > -- > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue33109> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True
New submission from Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: I find the True default for 'required' quite cumbersome introduced as a result of issue 26510. With existing parsers it can unnecessarily break compatibility between Python3.x versions only to make porting a bit easier for Python2 users. I think, this late in the life cycle of Python2, within Python3 compatibility should be ranked higher than py2to3 portability. Command line parsing of a package of mine has long used optional subparsers (without me even thinking much about the fact). Now in 3.7, running python3.7 -m MyPackage without arguments (the parser is in __main__.py) I get the ill-formatted error message: __main__.py: error: the following arguments are required: while my code in 3.3 - 3.6 was catching the empty Namespace returned and printed a help message. Because the 'required' keyword argument did not exist in < 3.7 there was no simple way for me to write code that is compatible between all 3.x versions. What I ended up doing now is to check sys.argv before trying to parse things, then print the help message, when that only has a single item, just to keep my existing code working. OTOH, everything would be just fine with a default value of False. Also that truncated error message should be fixed before 3.7 gets released. -- components: Library (Lib) messages: 314145 nosy: Anthony Sottile, bethard, eric.araujo, memeplex, paul.j3, wolma priority: normal severity: normal status: open title: argparse: make new 'required' argument to add_subparsers default to False instead of True type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33109> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31978] make it simpler to round fractions
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: That, of course, wasn't my original suggestion, but since Mark started discussing other possible forms this could take, like round-to-nearest analogs of mod and divmod, I thought maybe it's worth pointing out this aspect and, yes, maybe the three-argument form would be nice. Essentially, this would be fractions.Fraction.__round__ then. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31978> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31978] make it simpler to round fractions
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: >>> for x in range(1,501): for y in range(1,501): if round(x/y, 1) != float(round(F(x,y), 1)): print(x,y) where F is fractions.Fraction Sorry! -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31978> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31978] make it simpler to round fractions
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: > (E.g., if both `a` and `b` are not-too-large integers, `round(a / b)` is > still "safe" in that it will give the same result as if a non-lossy integer > division is used.) Well, it does not take particularly large a and b to break round's tie-breaking through rounding-to-even though: >>> for x in range(1,501): for y in range(1,501): if round(x/y, 1) != float(round(F(x,y), 1)): print(x,y) 1 20 2 40 3 20 3 60 4 80 5 100 6 40 6 120 7 20 7 140 8 160 9 20 9 60 9 180 10 200 11 220 12 80 12 240 13 20 13 260 14 40 14 280 15 100 15 300 16 320 17 340 18 40 18 120 18 360 19 20 19 380 20 400 21 20 21 60 21 140 21 420 22 440 23 20 23 460 24 160 24 480 25 500 ... -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31978> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31978] make it simpler to round fractions
Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de> added the comment: ok, I agree with you that the returned type should not change with the value of an argument. I simply didn't think one vs two argument versions here, but in terms of three different code branches where one returns int already. Maybe 'slight' was the wrong wording - think of it as 'easy to revert' then :) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31978> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31978] make it simpler to round fractions
New submission from Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: Hi, because of floating point inaccuracies it is suboptimal to use round(int1/int2) for rounding of a fraction. fractions.Fraction, OTOH, offers exact rounding through its implementation of __round__, but using it requires users to create a fractions.Fraction instance from two ints first. The algorithm used by Fraction.__round__ is, essentially, the same as the one used in the pure-Python version of the datetime._divide_and_round module (which, in turn, is taken from a comment by Mark Dickinson in Objects/longobject.c). My suggestion is to promote this algorithm to an exposed function in the fractions module (actually, it may make sense to have it in the math module instead - compare the case of the gcd function, which started out in fractions, but is now in transition to math) so that it can be used by Fraction.__round__, but also any other code. Attached is a patch demonstrating the idea. In addition, to the above benefit, it turns out to actually speed up Fraction.__round__ substantially, when ndigits is not None because it then avoids the generation of temporary Fraction instances, and, in my hands at least, it even makes the general (ndigits=None) case slightly faster (apparently the copied datetime._divide_and_round code is more efficient than the original in Fraction.__round__). There is one slight additional tweak in the patch: in the case of ndigits < 0, it returns an int, not a Fraction (see test_fractions modification to make it pass). I think this is actually a mistake in the current Fraction.__round__, which already promotes the result to int in the general case. This change speeds up round to the next ndigits power of ten by ~ a factor of 5 in my hands because no new Fraction needs to be instantiated anymore. A full PR could include having pure-Python datetime import the function from fractions instead of rolling its own, but I'd first like to hear whether you think this should go into math instead. -- components: Library (Lib) files: fractions_divround.patch keywords: patch messages: 305817 nosy: mark.dickinson, wolma priority: normal severity: normal status: open title: make it simpler to round fractions type: enhancement versions: Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47254/fractions_divround.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31978> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31756] subprocess.run should alias universal_newlines to text
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: -wolma ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31756> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14465] xml.etree.ElementTree: add feature to prettify XML output
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue14465> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31756] subprocess.run should alias universal_newlines to text
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31756> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28286] gzip guessing of mode is ambiguous
Change by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue28286> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27268] Incorrect error message on float('')
Wolfgang Maier added the comment: Could somebody turn this into a PR to move things forward? I guess Nofar mistakenly set resolution to "works for me", but meant "patch works for me"? -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27268> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30413] Add fnmatch.filterfalse function
Wolfgang Maier added the comment: Yet another thing I just realized (sorry for being so annoying): With os.normcase calling os.fspath in 3.6+ it is not really a NOP anymore even on posix. As a consequence, you can now do some weird things with fnmatch: in all cases, and only in these, where the module *does* call normcase you can pass in Path-like objects as the names. This works with fnmatch.fnmatch on all platforms, never for fnmatchcase, and platform-dependently (on Windows, but not on posix platforms) for filter/filterfalse. It's mostly that last case that worries me because it makes it easy to accidentally write code that is not platform-independent. Also note that you can also pass a Path-like object as pat everywhere except in fnmatchcase. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30413] Add fnmatch.filterfalse function
Wolfgang Maier added the comment: > Good catch! It seems to me that they are redundant. Please open a new issue > for this. done: issue 30427 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30427] isinstance checks in os.path.normcase redundant with os.fspath
Wolfgang Maier added the comment: Just created a PR for this, which eliminates the redundancy. This also changes the error message (making it less specific), but not the type of a raised exception. If you think that the error message deserves to be preserved that could, of course, be done too. -- type: -> performance ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30427] isinstance checks in os.path.normcase redundant with os.fspath
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- pull_requests: +1803 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30427] isinstance checks in os.path.normcase redundant with os.fspath
New submission from Wolfgang Maier: os.path.normcase as defined in both posixpath and ntpath is now calling os.fspath on its argument first. With that I think the following isinstance(str, bytes) checks have become redundant since AFAIU os.fspath is guaranteed to return either str or bytes instances. -- components: Library (Lib) messages: 294130 nosy: wolma priority: normal severity: normal status: open title: isinstance checks in os.path.normcase redundant with os.fspath versions: Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30413] Add fnmatch.filter_false function
Wolfgang Maier added the comment: @serhiy: my bad! I just hadn't realized this behavior of the original. With this requirement I cannot see any simpler solution than Steven's. Some other questions though to everyone involved: 1) what do you think about "os.path is posixpath" vs just checking os.name == 'posix' as I suggested earlier? 2) speaking of missing functionality in filter: What do you think of a keyword argument like 'case' to both filter and filterfalse that, when True, would make these functions behave equivalently to [n for n in names if fnmatchcase(n, pattern)] The default would be False, of course. I know this would be inconsistent in terms of argument vs separate functions, but it is easy to explain and learn and without it Windows users of filter/filterfalse would really suffer from the much poorer performance due to the slow normcase call (even slower now with the new fspath protocol) even if they pass in normalized names already. 3) not directly related to this issue, but I came across it in this context: isn't normcase in both posixpath and ntpath doing isinstance(str, bytes) checks that are redundant with os.fspath? Is this oversight or am I missing something again? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30413] Add fnmatch.filter_false function
Wolfgang Maier added the comment: Does it? I thought it does so only if normalize_case is True. Did I miss something? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30413] Add fnmatch.filter_false function
Wolfgang Maier added the comment: Hi, seems I had the same thoughts as you, Steven. I had started working on a patch independently yesterday, but after making my changes to fnmatch itself, I found I had too many other things to do to address unittests and doc changes to turn this into a real patch - so thank you for spending time on all of this. I just downloaded your patch and merged it with mine because I think my version of fnmatch.py may be simpler and slightly faster (though like you I didn't run performance tests). Feel free to do whatever you like with this alternate version - it's not that there is much new in it to take credit for :) Another thing I noted: fnmatch takes a rather unusual approach to determine whether normcase is NOP or not. It imports posixpath only to see if it is the same as os.path. That looks pretty weird and wasteful to me (especially if you are running this on Windows and are effectively importing posixpath just for fun then). I think it would be better to just check os.name instead (like pathlib does for example). I moved the corresponding check to the top of the module to make this easier to address, should that be of interest. I'm also using the result of the check in fnmatch.fnmatch because I don't see any reason why you wouldn't want to benefit from it there as well. -- nosy: +wolma Added file: http://bugs.python.org/file46880/filterfalse.alternate_patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30307] https://docs.python.org/3/tutorial/introduction.html#strings Section 3.1.2 doc issue
Wolfgang Maier added the comment: The section is correct as it is. Just try it in the interactive interpreter to convince yourself. -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30152] Reduce the number of imports for argparse
Wolfgang Maier added the comment: @rhettinger: I do not quite understand this harsh reaction. Making argparse more responsive could, in fact, be quite a nice improvement. This is particularly true, I guess, if you want argument completion with a package like https://pypi.python.org/pypi/argcomplete. You have a point that the patch probably tries to optimize too many things in too many places at once, but that could be amended rather easily. The idea of delaying imports in argparse itself to when they are actually needed is a really good one and, for this module, it is also very reasonable that other places in the stdlib only import it when they are run as __main__. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30152> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30152] Reduce the number of imports for argparse
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30152> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30097] Command-line option to suppress "from None" for debugging
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25478] Consider adding a normalize() method to collections.Counter()
Wolfgang Maier added the comment: > >>> Counter(red=11, green=5, blue=4).normalize(100) # percentage > Counter(red=55, green=25, blue=20) I like this example, where the normalize method of a Counter returns a new Counter, but I think the new Counter should always only have integer counts. More specifically, it should be the closest approximation of the original Counter that is possible with integers adding up to the argument to the method or, statistically speaking, it should represent the expected number of observations of each outcome for a given sample size. -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25478> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29258] __init__.py required for pkgutil.walk_packages in python3
Wolfgang Maier added the comment: > all that's required here is to eliminate the check for __init__.py from > pkgutil._iter_file_finder_modules Ok, I was exaggerating here. To do it right would require a more complex change, but that's all that's needed to get an estimate of the effect the real thing would have. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29258] __init__.py required for pkgutil.walk_packages in python3
Wolfgang Maier added the comment: While it is rather trivial to implement the proposed functionality - all that's required here is to eliminate the check for __init__.py from pkgutil._iter_file_finder_modules - this would have undesired impacts on, e.g., pydoc.apropos: This function would then recursively report *any* directory/subdirectory on sys.path, which is quite surely not what people want. I think this is a fundamental problem with namespace packages: they are nice and flexible for specific imports, but they make it impossible to know whether a directory found on the filesystem is *intended* as a Python package or not. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19821] pydoc.ispackage() could be more accurate
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue19821> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17062] An os.walk inspired replacement for pkgutil.walk_packages
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue17062> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29258] __init__.py required for pkgutil.walk_packages in python3
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29769] pkgutil._iter_file_finder_modules should not be fooled by *.py folders
New submission from Wolfgang Maier:
The current implementation of _iter_file_finder_modules parses folders with a
valid Python module extension as modules (e.g. it would report a *folder* xy.py
as a module xy).
As a result, e.g., pydoc.apropos('') fails if such a folder is found anywhere
on sys.path.
I'm attaching a patch that fixes this and also brings a few minor improvements
(like using a set instead of a dict with 1 values and reusing the function in
ImpImporter).
However, I have a question about it (which is also the reason why I didn't turn
this into a PR right away): in addition to checking that an item detected as a
module is not a directory, I think it would be good to also check that an
__init__ module inside a possible package really is a file. If I uncomment the
respective check in the patch though, I'm getting a test_pydoc failure because
the test creates a package directory with no access to contained file
attributes. So even though there is an __init__.py file in the package dir the
isfile() check fails. I think that should, in fact, happen and the pydoc test
is wrong, but apparently whoever wrote the test had a different opinion.
Any thoughts?
--
components: Library (Lib)
files: pkgutil.patch
keywords: patch
messages: 289285
nosy: ncoghlan, wolma
priority: normal
severity: normal
status: open
title: pkgutil._iter_file_finder_modules should not be fooled by *.py folders
type: behavior
versions: Python 3.7
Added file: http://bugs.python.org/file46714/pkgutil.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29769>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15343] "pydoc -w " writes out page with empty "Package Contents" section
Wolfgang Maier added the comment: Sorry, for generating noise on this very old issue, but was there a specific reason to duplicate the code of ImpImporter.find_module in changeset 9101eab6178c instead of factoring it out? -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue15343> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29710] Incorrect representation caveat on bitwise operation docs
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29710> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29414] Change 'the for statement is such an iterator' in Tutorial
Wolfgang Maier added the comment: > [...] I prefere the chapter as it currently is, because IMHO it > introduces the concepts more gradually than your proposal. That's ok! It's your PR and I only wanted to show an alternative. I was hoping for a bit more people to provide feedback though. The main reason I'm interested in changes to the tutorial is because I'm teaching Python to undergraduates (not in CS, but in Biology) and I recommend them to work through the Tutorial alongside the course so I have a natural interest in its quality and really appreciate any improvement, which I think your PR is certainly offering :) Let me still comment on some of your points regarding my suggestion: > In addition the modification of the title section from "for > Statements" to "for Loops" IMHO makes the title not consistent with > the other section titles. I don't see this point. The other section titles are not about loops. The preceding chapter introduces while loops as a loop, not a statement, and this chapter talks about while and for loops further down the page. >> - restructured the for loop section to discuss Python for loops >> first and only compare them to Pascal/C afterwards > +0. I think for a reader who is coming from another language is better > to have the current introduction. But maybe it is not the same for a > reader who is learning Python as a first programming language. I don't agree with you on this. I'm coming from a C background myself, but if I want to learn a new language the first thing I'm interested in is not how it's *not* working, but how it does. Plus, it's 2017, Perl has foreach, Javascript has at least array.foreach and even C++11 has auto-iteration over collections so starting an explanation of for loops with 'hey, look Python is different from two > 40 years old languages' feels a bit outdated. Regarding my main point of removing most of the ranges section it is possible I went a bit too far with that and maybe one should still briefly mention the optional start argument and stress that stop is not included in the range, but my seealso is quite prominent and the linked section on range is pretty good. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29414] Change 'the for statement is such an iterator' in Tutorial
Wolfgang Maier added the comment: I studied the github PR and thought about it carefully, and that made me come to believe that the chapter deserves a larger rewrite not just of one section, but of several. I'm attaching my proposed change as a "classical" patch here for discussion. I hope that's still a valid way to do this as I did not want to mess with the original PR by Marco and, instead, wanted to see my proposal discussed first. Some of the changes I made: - I don't think range() really deserves its own section in a chapter about control flow, so I removed most of it and linked to the relevant stdtypes section instead - moved the definition of an iterable up into the for loop section - restructured the for loop section to discuss Python for loops first and only compare them to Pascal/C afterwards - moved the example for modifying a list while iterating over it to the datastructures chapter I find this new version much clearer, but lets see what people here think. -- nosy: +wolma Added file: http://bugs.python.org/file46681/controlflow_datastructures.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English
Wolfgang Maier added the comment: I think PEP 538 extended to the UTF-8 locale *would* help here. Specifically, it would coerce only LC_CTYPE to en_US.UTF-8 (unless OS X has C.UTF-8), which I guess is good enough for the purpose here. I do agree that it is not the kind of problem that PEP 538 tries to solve right now, but it could be extended to cover other types of problematic locales like this one. Just wanted to make you aware of this possibility. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18378> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English
Wolfgang Maier added the comment: To me this issue seems quite related to PEP 538. Maybe the LC_CTYPE coercion proposed in the PEP could be extended to cover the case of LC_CTYPE=UTF-8? -- nosy: +ncoghlan versions: +Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18378> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28956] return minimum of modes for a multimodal distribution instead of raising a StatisticsError
Wolfgang Maier added the comment: What's the justification for this proposed change? Isn't it better to report the fact that there isn't an unambiguous result instead of returning a rather arbitrary one? -- nosy: +steven.daprano, wolma versions: +Python 3.7 -Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28956> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28859] os.path.mount sometimes raises FileNotFoundError on Windows
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18971] Use argparse in the profile/cProfile modules
Wolfgang Maier added the comment: oops, typing in wrong window. Very sorry. -- nosy: +wolma title: calendar -> Use argparse in the profile/cProfile modules ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18971] calendar
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- title: Use argparse in the profile/cProfile modules -> calendar ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Wolfgang Maier added the comment: Just found issue15451, which reports a similar inconsistency between Windows and POSIX for 'PATH' provided through the Popen env parameter as for cwd. It seems that, on POSIX-platforms, the PATH environment variable passed through env affects the executable lookup if executable does *not* contain a dirname, but on Windows the new PATH never affects executable lookup. So, again, relative executable paths are causing platform-specific behavior. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue15533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14845] list() != []
Wolfgang Maier added the comment:
running with "-W always":
>>> def five(x):
... for _ in range(5):
... yield x
...
>>> F = five('x')
>>> [next(F) for _ in range(10)]
Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
StopIteration
>>> list(next(F) for _ in range(10))
__main__:1: DeprecationWarning: generator '' raised StopIteration
[]
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue14845>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14845] list() != []
Wolfgang Maier added the comment: Mark, PEP479 is not fully in effect in 3.6 yet. 3.7 will raise the RuntimeError, but 3.6 still only gives a DeprecationWarning. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue14845> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14845] list() != []
Wolfgang Maier added the comment: Isn't the difference between generator expressions and comprehensions what's dealt with by PEP479? So it seems this issue is outdated enough to deserve being closed? -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue14845> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9938] Documentation for argparse interactive use
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue9938> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28801] configparser: before_get() method of class Interpolation has positional 'parser' parameter that is not used.
Wolfgang Maier added the comment: > Ah. Something like self._interpolation.before_get(self, section, option, > value, d) could be better written as > self._interpolation.before_get(parser=self, ...) Yep, that's roughly what I was trying to explain. > I still don't grock the apparent 'get()' signature mis-match however. There is not much special here either. RawConfigParser inherits from MutableMapping, which in turn inherits from Mapping, which defines a get method, which RawConfigParser overwrites. The overwritten and the overwriting method *do* have different parameters, but I don't see why that matters. In general, this does not look like a topic for the Python bug tracker (you are not reporting a bug, but you try to understand how correctly working code does its job), but rather for news:comp.lang.python or a PyCharm mailing list. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28801> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28801] configparser: before_get() method of class Interpolation has positional 'parser' parameter that is not used.
Wolfgang Maier added the comment: Ah, that's kind of confusing at first! the 'self' in the method calls (like on line 796) refers to the ConfigParser instance and will be bound to the parser parameter. The first parameter, the 'self' in the interpolation method definition, is not provided as usual because the method is called on an _interpolation instance. => Nothing special here except the names of the arguments -- nosy: +wolma type: compile error -> ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28801> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22298] Lib/warnings.py _show_warning does not protect against being called with a file like object which is closed
Wolfgang Maier added the comment: Ah, I just found Issue607668, which discusses the changeset 2e7fe55c0e11 that introduced IOError suppression. The actual use case at the time was different and I have no idea whether it would still be a problem today. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22298] Lib/warnings.py _show_warning does not protect against being called with a file like object which is closed
Wolfgang Maier added the comment: Hi Julius, I guess it's a question of control and responsibilities. If you write a program that may issue warnings under some conditions, that program may, for example, be used with stderr redirected to a file. Now it is possible that file gets removed while your program is running, then, upon the next warning, the program would crash with an IOError. In effect, this would let external cirumstances that are not under control by the programmer escalate a non-critical warning message to a true exception. It's debatable whether that should be prevented or not, but that seems to be the decision that Mark Hammond made long ago. If, OTOH, your program chooses to close sys.stderr, that is a deliberate decision and it's the programmers responsibility then not to try to write to it anymore. Suppressing the resulting ValueError would only hide the fact that due to a programming error no warnings can be issued anymore. Not sure if this makes any sense to you or others, but that's how I came to think about this. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Wolfgang Maier added the comment: Before I forget again what I've gathered yesterday about this issue, here's a summary of the problem: When the the first element of args or the executable argument of subprocess.Popen does not specify an absolute path, the way the executable gets discovered is platform-dependent. On POSIX platforms, if the argument contains a directory part, the executable gets looked for relative to the current working directory. If the argument is just a basename, the PATH is searched. On Windows, the executable argument and the first element of args are treated in different ways: If the executable is specified through the executable argument, it is always looked for relative to the current working directory only, but if it's specified through args, it's searched for using Windows-specific rules as documented for the underlying CreateProcess function (see https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx). Now, on top of all this, if the cwd argument of Popen is used, then, in Python3 on POSIX-platforms, the current working directory gets changed to cwd *before* the above interpretation happens, i.e., if executable or args[0] contains a dirname, the executable is looked for relative to cwd. On Windows, however, cwd becomes the current working directory of the new process, but is *not* used during the executable lookup. I guess with this being rather complicated it would be nice to have a dedicated section explaining the interpretation of relative paths as arguments instead of trying to get only the cwd wording right. -- components: +Library (Lib) type: -> behavior ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue15533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15533] subprocess.Popen(cwd) documentation
Wolfgang Maier added the comment: Ping. This still isn't fixed several years later, i.e., the documentation still describes the POSIX, but not the Windows behavior. See also issue20927, which reports this as a bug. -- nosy: +wolma versions: +Python 3.5, Python 3.6, Python 3.7 -Python 2.7, Python 3.2, Python 3.3 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue15533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10379] locale.format() input regression
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10379> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22298] Lib/warnings.py _show_warning does not protect against being called with a file like object which is closed
Wolfgang Maier added the comment: Issue23016 fixed the "AttributeError: 'NoneType' object has no attribute 'write'" problem when sys.stderr is None. With that it's now possible and good practice to set sys.stderr = None when closing it, just as Antoine suggested. Should this issue be closed then? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27761] Private _nth_root function loses accuracy
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18943] argparse: default args in mutually exclusive groups
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18943> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22298] Lib/warnings.py _show_warning does not protect against being called with a file like object which is closed
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27583] configparser: modifying default_section at runtime
Wolfgang Maier added the comment: Well, you *can* change the value at runtime as you are demonstrating in your script, but you are misunderstanding the effect this will have. It *won't* cause a reevaluation of an already parsed config file. Instead it will affect the writing of the parsed settings to a new config file. This is explained a bit further up in the documentation of the module: https://docs.python.org/3/library/configparser.html#customizing-parser-behaviour where it says: "Its current value can be retrieved using the parser_instance.default_section attribute and may be modified at runtime (i.e. to convert files from one format to another)." So maybe this hint could be repeated in the actual parameter description of https://docs.python.org/3/library/configparser.html#configparser-objects to avoid confusion, but I don't think there is a bug here. -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27583> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28734] argparse: successive parsing wipes out nargs=? values
Wolfgang Maier added the comment:
try this:
parser.add_argument('file', nargs='?', default = argparse.SUPPRESS)
I don't think this is a bug. The default for the default parameter is None so
the behavior is consistent with the documentation.
--
nosy: +wolma
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28734>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28716] Fractions instantiation revisited
Wolfgang Maier added the comment: No, I don't think the numeric tower ABC should be replaced by duck-typing. One of the very reasons the fractions module exists is that it showcases how to use the numeric tower. If you want a class to be picked up as a Rational it should be registered as such. Pretty much the same goes for integer-like things. The direct type check for ints only exists to speed up the most obvious case, but anything integer-like should be registered as a numbers.Integral and it will just work with fractions. Beautiful stuff! This is not the case for as_integer_ratio because that method is not defined in the numeric tower and this is why I think duck-typing could be of interest here (I'm not sure though whether its worth the somewhat decreased performance). My reasoning is that the standard constructor is anyway slower for floats and ints than the specialized classmethods for the two so people who really care about speed here (probably not too many) can just use the classmethods. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28716> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28716] Fractions instantiation revisited
Wolfgang Maier added the comment:
sure, I just happened to have the profiling available since I used it to
optimize things. Here's similar microbenchmarks using perf:
STRINGS
===
$ python -m perf timeit -s "from fractions import Fraction"
"Fraction('1.23456e-7')"
.
Median +- std dev: 17.0 us +- 0.3 us
$ python -m perf timeit -s "from frc import Fraction" "Fraction('1.23456e-7')"
.
Median +- std dev: 8.95 us +- 0.16 us
$ python -m perf timeit -s "from fractions import Fraction"
"Fraction('234/567')"
.
Median +- std dev: 12.6 us +- 0.1 us
$ python -m perf timeit -s "from frc import Fraction" "Fraction('234/567')"
.
Median +- std dev: 5.45 us +- 0.16 us
$ python -m perf timeit -s "from fractions import Fraction" "Fraction('123456')"
.
Median +- std dev: 12.4 us +- 0.6 us
$ python -m perf timeit -s "from frc import Fraction" "Fraction('123456')"
.
Median +- std dev: 5.77 us +- 0.12 us
$ python -m perf timeit -s "from fractions import Fraction; f=Fraction(3/4)"
"Fraction(f)"
.
Median +- std dev: 4.36 us +- 0.06 us
$ python -m perf timeit -s "from frc import Fraction; f=Fraction(3/4)"
"Fraction(f)"
.
Median +- std dev: 4.59 us +- 0.07 us
$ python -m perf timeit -s "from fractions import Fraction" -s "class
myInt(int): pass" -s "i=myInt(123456)" "Fraction(i)"
.
Median +- std dev: 4.04 us +- 0.07 us
$ python -m perf timeit -s "from frc import Fraction" -s "class myInt(int):
pass" -s "i=myInt(123456)" "Fraction(i)"
.
Median +- std dev: 4.27 us +- 0.06 us
FLOATS
==
$ python -m perf timeit -s "from fractions import Fraction"
"Fraction(1.23456e-7)"
.
Median +- std dev: 6.30 us +- 0.28 us
$ python -m perf timeit -s "from frc import Fraction" "Fraction(1.23456e-7)"
.
Median +- std dev: 8.64 us +- 0.13 us
$ python -m perf timeit -s "from fractions import Fraction"
"Fraction.from_float(1.23456e-7)"
.
Median +- std dev: 8.68 us +- 0.14 us
$ python -m perf timeit -s "from frc import Fraction"
"Fraction.from_float(1.23456e-7)"
.
Median +- std dev: 3.37 us +- 0.17 us
DECIMALS (using C implementation this time)
===
$ python -m perf timeit -s "from fractions import Fraction; from decimal import
Decimal; d=Decimal('123456')" "Fraction(d)".
Median +- std dev: 6.95 us +- 0.21 us
$ python -m perf timeit -s "from frc import Fraction; from decimal import
Decimal; d=Decimal('123456')" "Fraction(d)"
.
Median +- std dev: 8.43 us +- 0.17 us
$ python -m perf timeit -s "from fractions import Fraction; from decimal import
Decimal; d=Decimal('123456')" "Fraction.from_decimal(d)"
.
Median +- std dev: 11.6 us +- 0.2 us
$ python -m perf timeit -s "from frc import Fraction; from decimal import
Decimal; d=Decimal('123456')" "Fraction.from_decimal(d)"
.
Median +- std dev: 4.14 us +- 0.28 us
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28716>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28716] Fractions instantiation revisited
New submission from Wolfgang Maier: I've spent a bit of time lately trying to optimize the instantiation of Fractions. This is related to Issue22464, but instead of focusing on constructing Fractions from ints, my attempts revolve around improving instantiation from strings, floats and decimals. I'm attaching a patch with all my changes for discussion, but here's an overview of what's in it: CHANGES TO INSTANTIATION FROM STRINGS: - isinstance checking for str is performed before the more expensive check for numbers.Rational (which has to go through abc) - instantiation from strings doesn't use a regex pattern anymore for parsing; this is faster in many cases (and never slower) than the current version - while reimplementing string parsing I added PEP 515 support (this is the only behavior change in the patch) combined this gives me the following performance changes for instantiation of Fractions from different arguments (profiled with 1,000,000 random inputs each): 'x/y'-type of strings: -- ncalls tottime percall cumtime percall filename:lineno(function) old: 100 12.1620.000 27.7780.000 fractions.py:84(__new__) new: 1009.6190.000 12.4280.000 frc.py:68(__new__) scientific notation (e.g., '1.234E-7'): --- ncalls tottime percall cumtime percall filename:lineno(function) old: 100 18.2090.000 37.3410.000 fractions.py:84(__new__) new: 100 15.5090.000 21.2520.000 frc.py:68(__new__) integer strings (e.g. '123456'): ncalls tottime percall cumtime percall filename:lineno(function) old: 100 11.2720.000 26.2010.000 fractions.py:84(__new__) new: 1009.3470.000 12.4250.000 frc.py:68(__new__) from other Fractions (to test effect on instantiation of numbers.Rational): ncalls tottime percall cumtime percall filename:lineno(function) old: 1004.7080.000 10.0930.000 fractions.py:84(__new__) new: 1004.8350.000 10.5720.000 frc.py:68(__new__) from int subclass (as another numbers.Rational): ncalls tottime percall cumtime percall filename:lineno(function) old: 1003.4460.0008.0440.000 fractions.py:84(__new__) new: 1003.7950.0008.8360.000 frc.py:68(__new__) SUMMARY of this part + very significant speedup for instatiation from strings of different forms with (near) negligible effects on instantiation from numbers.Rational. + correct parsing of PEP 515-like number strings - possibly slower error bubbling with invalid input (untested) CHANGES TO INSTANTIATION FROM FLOAT AND DECIMAL: - no explicit check for float and decimal in standard constructor; instead, simply try to call as_integer_ratio as last resort (even after checking for numbers.Rational) - speed up alternate from_float and from_decimal constructor class methods by rearranging type checks and making use of _normalize flag - import decimal only on demand (when using from_decimal) Resulting performance changes: standard constructor with float: ncalls tottime percall cumtime percall filename:lineno(function) old: 1004.3620.000 12.4520.000 fractions.py:84(__new__) new: 1006.6930.000 16.0200.000 frc.py:68(__new__) from_float: --- ncalls tottime percall cumtime percall filename:lineno(function) old: 1003.1460.000 17.7690.000 fractions.py:193(from_float) new: 1002.5440.0007.5910.000 frc.py:205(from_float) standard constructor with decimal: ncalls tottime percall cumtime percall filename:lineno(function) old: 1004.6720.000 20.7950.000 fractions.py:84(__new__) new: 1007.0970.000 24.5260.000 frc.py:68(__new__) from_decimal: - ncalls tottime percall cumtime percall filename:lineno(function) old: 1005.0540.000 34.4730.000 fractions.py:207(from_decimal) new: 1002.9420.000 16.0130.000 frc.py:220(from_decimal) SUMMARY of this part: - standard constructor becomes a bit slower for floats and Decimals specialized class methods become a lot faster (for Decimal the benchmarks are based on _pydecimal - with the C implementation the percent differences would be even larger) - eliminates decimal from regularly imported modules - allows Fraction instantiation from duck-typing classes that provide as_integer_ratio I hope at least some of this is interesting. -- components: Library (Lib) files: fractions.patch keywords: patch messages: 280977 nosy: wolma priority: normal severity: normal status: open title: Fractions instantiation revisited type: behavior versions: Python 3.7 Added file: http://bugs.python.org/file45507/fractions.patch
[issue28637] Python startup performance regression
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +vinay.sajip ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28637> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28637] Python startup performance regression
Wolfgang Maier added the comment: @serhiy.storchaka you've beaten me by a few minutes (still waiting for the test suite to finish). Your patch is "contaminated" by an additional change in collections/__init__.py though so I'm still uploading my version (which also tries to stick as close as possible to the existing behaviour though that probably won't matter much) -- Added file: http://bugs.python.org/file45390/no_use_re_in_site_py.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28637> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28637] Python startup performance regression
Wolfgang Maier added the comment:
STINNER Victor added the comment:
>BUT when Python is started from a virtual environment (created by the
>"venv" module), the re module is important by default.
>
>haypo@speed-python$ venv/bin/python3 -c 'import sys; print("re" in
>sys.modules)'
>True
Exciting, I just verified that this is true and running python3 from a venv
really seems to be the only situation, in which the re module gets imported
during startup (at least it's only this one branch in site.py that uses it).
If adding a single enum import to re causes such a big startup time difference
I wonder how much more could be gained for the venv case by not importing re at
all!
Turns out that the complete code block in site.py that is used by venvs and
that was partially shown by @haypo is:
CONFIG_LINE = r'^(?P(\w|[-_])+)\s*=\s*(?P.*)\s*$'
def venv(known_paths):
global PREFIXES, ENABLE_USER_SITE
env = os.environ
if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
executable = os.environ['__PYVENV_LAUNCHER__']
else:
executable = sys.executable
exe_dir, _ = os.path.split(os.path.abspath(executable))
site_prefix = os.path.dirname(exe_dir)
sys._home = None
conf_basename = 'pyvenv.cfg'
candidate_confs = [
conffile for conffile in (
os.path.join(exe_dir, conf_basename),
os.path.join(site_prefix, conf_basename)
)
if os.path.isfile(conffile)
]
if candidate_confs:
import re
config_line = re.compile(CONFIG_LINE)
virtual_conf = candidate_confs[0]
system_site = "true"
# Issue 25185: Use UTF-8, as that's what the venv module uses when
# writing the file.
with open(virtual_conf, encoding='utf-8') as f:
for line in f:
line = line.strip()
m = config_line.match(line)
if m:
d = m.groupdict()
key, value = d['key'].lower(), d['value']
if key == 'include-system-site-packages':
system_site = value.lower()
elif key == 'home':
sys._home = value
sys.prefix = sys.exec_prefix = site_prefix
# Doing this here ensures venv takes precedence over user-site
addsitepackages(known_paths, [sys.prefix])
# addsitepackages will process site_prefix again if its in PREFIXES,
# but that's ok; known_paths will prevent anything being added twice
if system_site == "true":
PREFIXES.insert(0, sys.prefix)
else:
PREFIXES = [sys.prefix]
ENABLE_USER_SITE = False
return known_paths
So all the re module is good for here is to parse simple config file records
with key/value pairs separated by '='. ´Shouldn't it be straightforward to
implement that logic right inside that block directly without requiring a giant
import?
This should easily be doable for 3.6 still, seems as if it would solve the
whole issue and probably speed up the performance tests much more than any
reverted changesets could.
What do you think?
--
nosy: +wolma
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28637>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28115] Use argparse for the zipfile module
Changes by Wolfgang Maier <wolfgang.ma...@biologie.uni-freiburg.de>: -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)
Wolfgang Maier added the comment: A warning like this sounds good to me though I'd prefer it to be slightly more verbose, like: "RuntimeWarning: '' found in sys.modules after import of package '', but prior to execution of '' as '__main__'; this may result in unpredictable behaviour" or it wouldn't really have helped me much to understand what was happening. P.S.: You understood correctly that my own code is fixed (patch sent to the affected user, next release will have the fix). Learnt something about the whole __spec__ and double import subject along the way so it wasn't completely wasted time. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)
Wolfgang Maier added the comment:
@Martin and regarding Python3.3: Right, you cannot do the replacement when
running the module as __main__. For my use case that's not required though so
in
the module I can just do:
if __name__ == '__main__':
print('running module as a script')
else:
# do not attempt this when __name__ == '__main__' in Python 3.3
sys.modules[__name__] = object()
Apart from this limitation, such code is compatible with Python 3.2 - 3.5.1,
but
breaks under 3.5.2. It's the fact that it breaks in a maintenance release that
I find disturbing. In fact, I learnt about the breakage only from an end-user
who bought a new laptop with OS X, downloaded the latest Python release and
suddenly could not get our package to run that had worked on his previous
machine. Turns out, none of our test machines was running 3.5.2 yet, but were
running 3.5.0 and 3.5.1.
OTOH, I agree with you that the circumstances under which your change causes
trouble are pretty exotic. As Nick points out, modules replacing themselves in
sys.modules are not that special. What *makes* the situation special is that
1) the package __init__.py has to import the self-replacing module and that
2) the self-replacing module is intended to be run via python3 -m package.module
That's why I wrote initially that I don't know if many (or, in fact, any other
package would be affected by this).
Nick's concern about the double import situation is more general, but since
this is something earlier releases have been doing, an improvement here would
be relevant only for a major release (3.6 or 3.7).
Regarding 3.5.3, I'm not sure what the best solution would be:
- the suggestion to copy over the __spec__ attribute to the replacement object
is putting the burden on package maintainers. Given that probably only few
packages will be affected that may well be acceptable. In fact, for my own
case I have now refactored things a bit so I can avoid the import from
__init__.py.
- from a strict versioning perspective, you may also argue like Nick that if it
was working in 3.5.0-1 it should work again in 3.5.3.
I'm personally pretty neutral here. I just thought it may be good to report the
issue so that it becomes easier for others to understand what is going on if
they encounter the same error. Maybe even the fact that this issue exists now
is serving this purpose as well as documenting the changed behaviour?
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue27487>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27487] -m switch regression in Python 3.5.2 (under rare circumstances)
New submission from Wolfgang Maier: As a result of Issue14285 Python 3.5.2 now imports packages in runpy. _get_module_details before calling importlib.util.find_spec. Although I'm not sure how important this is, I wanted to report that this new behaviour can have a side-effect under pretty exotic circumstances. When __init__.py imports the same module that is supposed to be invoked via the -m switch and that module replaces its own entry in sys.modules with another object, this causes importlib.util.find_spec to fail with a *very* cryptic: Error while finding spec for 'package.module' (ValueError: package.module.__spec__ is not set) without an exception traceback. I have no idea whether any other package would be affected by this, but it took me quite some time today to trace the cause of this and it is not what you'd expect from a maintenance release. I think the changed behaviour in runpy should at least be documented (beyond just mentioning the issue in the changelog). -- components: Library (Lib) messages: 270202 nosy: martin.panter, ncoghlan, wolma priority: normal severity: normal status: open title: -m switch regression in Python 3.5.2 (under rare circumstances) type: behavior versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26904] Difflib quick_ratio() could use Counter()
Wolfgang Maier added the comment: Given your comment about sum((fullacount & fullbcount).values()), why not use its in-place version: fullacount &= fullbcount matches = sum(fullacount.values()) ? -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26904> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18378] locale.getdefaultlocale() fails on Mac OS X with default language set to English
Wolfgang Maier added the comment: ping? Just ran into this issue on OS X El Capitan with Region set to Germany and Language to English. Just as Ned pointed out 2 years ago, this results in LC_CTYPE set to 'UTF-8' in the terminal and docutils still can't cope with it. -- nosy: +wolma ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18378> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26506] hex() documentation: mention "%x" % int
Wolfgang Maier added the comment:
> Hum, python3 looks faster on this dummy microbenchmark yeah. Who said that
> Python 3 is slower? :-)
If you're alluding to that seemingly endless thread over on python-list, let me
say that it is not my motivation to start anything like that here. Sorry also
if I sort of hijacked your documentation issue with my performance question.
I really only wondered whether there would be any argument for or against any
of the two versions (%-interpolation, format-based) other than stylistic ones.
That's why I ran the micro-benchmark and, in fact, I was expecting
%-interpolation to be faster exactly because it is less flexible.
What I am surprised by is not the fact that %-interpolation got faster in
Python3, but the fact that format didn't.
I was wondering whether %-interpolation maybe takes some fast path in Python3
that simply wasn't implemented for format. If that was the case it could have
been rewarding to just optimize format the same way.
As I know Victor is working on performance stuff I thought I'd just ask here,
but from your answer I gather that things are rather not so simple and that's
ok.
> I wrote a first article to explain my work on optimization:
https://haypo.github.io/pybyteswriter.html
Thanks for the link.
> str.format(args) was also optimized, but it's still faster than str%args.
You mean slower I assume ?
> Hum, I don't recall why you started to talk about performance :-D
See above.
> Why not documenting "%x" % value *and* format(value, 'x')?
> I prefer "%x" % value. I never use format(value, ...) but sometimes I use
> "{0:x}".format(value).
I prefer the last version, use the first sometimes, but documenting several
ways seems reasonable.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26506>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26506] hex() documentation: mention "%x" % int
Wolfgang Maier added the comment: Ah, but it's not that format() is slower in 3.5, but that %-formatting got faster. It looks as if it got optimized and I was wondering whether the same optimization could be applied to format(). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26506] hex() documentation: mention "%x" % int
Wolfgang Maier added the comment:
Your two suggestions prompted me to do a speed comparison between them and the
result surprised me.
I tried:
import random
nums = [random.randint(0, 255) for n in range(1000)]
then timed the simple:
for n in nums:
hx = '%X' % n # or hx = format(n, 'X')
I also tested a number of more complex formats like:
hx = '%{:02X}'.format(n) vs hx = '%%%02X' % n
In all cases, the old vs new formatting styles are rather similar in speed in
my system Python 2.7.6 (with maybe a slight advantage for the format-based
formatting).
In Python 3.5.0, however, old-style %-formatting is much speedier than under
Python 2, while new-style formatting doesn't appear to have changed much, with
the result that %-formatting is now between 30-50% faster than format-based
formatting.
So I guess my questions are:
- are my timings wrong?
and if not:
- how got %-formatting improved (generally? or for %X specifically?)
- can this speed up be transferred to format-based formatting somehow?
--
nosy: +wolma
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26506>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26501] bytes splitlines() method returns strings without decoding
Wolfgang Maier added the comment: oh and documentation is here: https://docs.python.org/3.5/library/stdtypes.html?highlight=bytes.splitlines#bytes.splitlines -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26501> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
