Re: Ref-strings in logging messages (was: Performance issue with CPython 3.10 + Cython)
On Fri, 7 Oct 2022 18:28:06 +0100
Barry wrote:
> > On 7 Oct 2022, at 18:16, MRAB wrote:
> >
> > On 2022-10-07 16:45, Skip Montanaro wrote:
> >>> On Fri, Oct 7, 2022 at 9:42 AM Andreas Ames
> >>>
> >>> wrote:
> >>> 1. The culprit was me. As lazy as I am, I have used f-strings all over the
> >>> place in calls to `logging.logger.debug()` and friends, evaluating all
> >>> arguments regardless of whether the logger was enabled or not.
> >>>
> >> I thought there was some discussion about whether and how to efficiently
> >> admit f-strings to the logging package. I'm guessing that's not gone
> >> anywhere (yet).
> > Letting you pass in a callable to call might help because that you could
> > use lambda.
>
> Yep, that’s the obvious way to avoid expensive log data generation.
> Would need logging module to support that use case.
I have some logging code that uses eval() to evaluate expressions using
locals and globals in a parent stack frame, together with a parser to
find `{...}` items in a string.
I guess this constitutes a (basic) runtime implementation of f-strings.
As such it can avoid expensive evaluation/parsing when disabled, though
it's probably slow when enabled compared to native f-strings. It seems
to work quite well in practise, and also allows one to add some extra
formatting features.
For details see:
https://git.ghostscript.com/?p=mupdf.git;a=blob;f=scripts/jlib.py;h=e85e9f2c4;hb=HEAD#l41
- Jules
--
http://op59.net
--
https://mail.python.org/mailman/listinfo/python-list
Please can i have your attention
Dear , Please can I have your attention and possibly help me for humanity's sake please. I am writing this message with a heavy heart filled with sorrows and sadness. Please if you can respond, i have an issue that i will be most grateful if you could help me deal with it please. Julian -- https://mail.python.org/mailman/listinfo/python-list
[issue46905] winsound.PlaySound should accept pathlib.Path instances
New submission from Julian : The library function winsound.PlaySound takes a sound parameter. One of the valid arguments is a path to a WAV filename. Since Python 3.4, paths can be cleanly represented with PathLib.Path instances. However, if you pass a Path instance to PlaySound, it responds with: TypeError: 'sound' must be str or None, not 'WindowsPath' This can be quickly fixed in the caller by passing str(the_path_instance) instead, but it would be cleaner if PlaySound accepted Python's preferred method of representing paths. -- components: Library (Lib) messages: 414393 nosy: Julian-O priority: normal severity: normal status: open title: winsound.PlaySound should accept pathlib.Path instances type: enhancement ___ Python tracker <https://bugs.python.org/issue46905> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46253] C API documentation of Py_UNICODE_* character properties macros use Py_UNICODE instead of Py_UCS4
Change by Julian Gilbey : -- keywords: +patch pull_requests: +28595 stage: -> patch review pull_request: https://github.com/python/cpython/pull/30387 ___ Python tracker <https://bugs.python.org/issue46253> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46253] C API documentation of Py_UNICODE_* character properties macros use Py_UNICODE instead of Py_UCS4
New submission from Julian Gilbey : The documentation at https://docs.python.org/3/c-api/unicode.html?highlight=isalpha#unicode-character-properties lists a series of macros such as Py_UNICODE_ISSPACE(Py_UNICODE ch). However, the input type for these macros was changed from Py_UNICODE to Py_UCS4 in commit 324ac65cebf4b0141b946452a2604715f1ca7010 on Wed Aug 18 20:44:58 2010 +, but it seems that the documentation has never been updated. The fix should be easy; I'll make a PR for it. -- assignee: docs@python components: Documentation messages: 409649 nosy: docs@python, juliangilbey priority: normal severity: normal status: open title: C API documentation of Py_UNICODE_* character properties macros use Py_UNICODE instead of Py_UCS4 versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46253> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45081] dataclasses that inherit from Protocol subclasses have wrong __init__
Julian Fortune added the comment: Ian, `MyProtocol` does not provide an `__init__()`, and thus ``` super().__init__() ``` is calling the `__init__()` from `Protocol`. This results in the `TypeError`. Simply remove `super().__init__()` to resolve your issue. This behavior was changed in https://github.com/python/cpython/pull/27545 (see `Lib/typing.py:1384`); I don't see what you are reporting as a regression, I see it as correct behavior that I would expect. Apologies if you feel differently. Cheers, Julian -- ___ Python tracker <https://bugs.python.org/issue45081> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45081] dataclasses that inherit from Protocol subclasses have wrong __init__
Change by Julian Fortune : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45081> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45081] dataclasses that inherit from Protocol subclasses have wrong __init__
New submission from Julian Fortune :
I believe [`bpo-44806: Fix __init__ in subclasses of
protocols`](https://github.com/python/cpython/pull/27545) has caused a
regression when using a Dataclass.
In Python `3.9.7`, a `dataclass` that inherits from a subclass of
`typing.Protocol` (i.e., a user-defined protocol), does not have the correct
`__init__`.
### Demonstration
```python
from dataclasses import dataclass
from typing import Protocol
class P(Protocol):
pass
@dataclass
class B(P):
value: str
print(B("test"))
```
In `3.9.7`:
```shell
Traceback (most recent call last):
File "test.py", line 11, in
print(B("test"))
TypeError: B() takes no arguments
```
In `3.9.6`:
```shell
B(value='test')
```
### Affected Projects
- [dbt](https://github.com/dbt-labs/dbt/issues/3843)
--
components: Library (Lib)
messages: 400868
nosy: julianfortune
priority: normal
severity: normal
status: open
title: dataclasses that inherit from Protocol subclasses have wrong __init__
type: behavior
versions: Python 3.9
___
Python tracker
<https://bugs.python.org/issue45081>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44986] Date formats in help messages of argparse
Julian Berman added the comment: This is documented already I believe: https://docs.python.org/3/library/argparse.html#help > As the help string supports %-formatting, if you want a literal % to appear > in the help string, you must escape it as %%. -- nosy: +Julian ___ Python tracker <https://bugs.python.org/issue44986> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44995] "Hide the prompts and output" works abnormal
Change by Julian Berman : -- keywords: +patch nosy: +Julian nosy_count: 2.0 -> 3.0 pull_requests: +26384 stage: -> patch review pull_request: https://github.com/python/cpython/pull/27939 ___ Python tracker <https://bugs.python.org/issue44995> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44994] datetime's C implementation verifies fromisoformat is ASCII, but the pure python implementation does not
New submission from Julian Berman :
This line (which contains a non-ASCII digit):
python3.9 -c "import datetime; datetime.date.fromisoformat('1963-06-1৪')"
raises:
Traceback (most recent call last):
File "", line 1, in
ValueError: Invalid isoformat string: '1963-06-1৪'
under the C implementation of the datetime module, but when the pure Python
implementation is the one imported, succeeds (and produces `datetime.date(1963,
6, 14)`)
The pure Python implementation should instead explicitly check and raise when
encountering a non-ASCII string.
(On PyPy, which always uses the pure-Python implementation, this contributes to
a behavioral difference)
--
components: Library (Lib)
messages: 400235
nosy: Julian, p-ganssle
priority: normal
severity: normal
status: open
title: datetime's C implementation verifies fromisoformat is ASCII, but the
pure python implementation does not
type: behavior
versions: Python 3.10, Python 3.11, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue44994>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44383] argparse.BooleanOptionalAction interacts poorly with ArgumentDefaultsHelpFormatter
Change by Julian Gilbey : -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue44383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44383] argparse.BooleanOptionalAction interacts poorly with ArgumentDefaultsHelpFormatter
Julian Gilbey added the comment: Yes, this is a duplicate. Marking this report as a duplicate. It's a shame that the other one has a conflict and has been sitting waiting for an update for over a year :-( I'd be happy to help if it is of use. -- resolution: -> duplicate ___ Python tracker <https://bugs.python.org/issue44383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44383] argparse.BooleanOptionalAction interacts poorly with ArgumentDefaultsHelpFormatter
New submission from Julian Gilbey : With code like the following: import argparse parser = argparse.ArgumentParser( description="Test program", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) parser.add_argument( "--foo", help="Use the foo component.", action=argparse.BooleanOptionalAction, default=True, ) args = parser.parse_args() a call "python prog.py --help" then gives: usage: prog.py [-h] [--foo | --no-foo] Test program optional arguments: -h, --help show this help message and exit --foo, --no-foo Use the foo component. (default: True) (default: True) Note the repeated "(default: True)", one produced by the BooleanOptionalAction class and the other by the ArgumentDefaultsHelpFormatter. It would be good if they didn't both add this helpful information. My suggestion would be that BooleanOptionalAction should not include this information; it is unique in doing so. -- components: Library (Lib) messages: 395559 nosy: juliangilbey priority: normal severity: normal status: open title: argparse.BooleanOptionalAction interacts poorly with ArgumentDefaultsHelpFormatter type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue44383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43742] tcp_echo_client in asyncio streams example does not work. Hangs for ever at reaser.read()
julian colomina added the comment: @jaswdr Thanks for your response. No I did not run the server that you mention. The language made me imply that the same process, in two separate coroutines, would be writing/reading from each end of the tcp connection. One writing to the tcp buffer, the second one draining it. If the intended usage is as you explain, please close the issue. My bad, sorry. JC -- ___ Python tracker <https://bugs.python.org/issue43742> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36384] ipaddress Should not reject IPv4 addresses with leading zeroes as ambiguously octal
Change by Julian Berman : -- nosy: +Julian ___ Python tracker <https://bugs.python.org/issue36384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22282] ipaddress module accepts octal formatted IPv4 addresses in IPv6 addresses
Change by Julian Berman : -- nosy: +Julian ___ Python tracker <https://bugs.python.org/issue22282> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43742] tcp_echo_client in asyncio streams example does not work. Hangs for ever at reaser.read()
New submission from julian colomina : taking the example verbatim into an ubuntu 20.04 with Python 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0] on linux will hand indefinitely at data = await reader.read(100) changing for data = await asyncio.wait_for(reader.read(100),5) will always leave on timeout. -- components: asyncio messages: 390295 nosy: asvetlov, jcolo, yselivanov priority: normal severity: normal status: open title: tcp_echo_client in asyncio streams example does not work. Hangs for ever at reaser.read() type: behavior versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue43742> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43148] Call sys.unraisablehook in the REPL when sys.excepthook is broken
Julian Berman added the comment: Thanks Victor. Yes likely happy to send a PR (have to clear a few things off the yak stack first) On Tue, Feb 9, 2021 at 5:38 AM STINNER Victor wrote: > > STINNER Victor added the comment: > > It makes sense to call _PyErr_WriteUnraisableMsg() when sys.excepthook > raise a new exception. Do you want to propose a PR for that? > > sys.excepthook documentation mentions sys.unraisablehook: > https://docs.python.org/dev/library/sys.html#sys.excepthook > > threading._make_invoke_excepthook logs threading.excepthook failure using > sys.excepthook. > > Note: Carl also suggested to add a way to invoke sys.unraisablehook > explicitly, not only for testing purpose. It would help in some cases. > > -- > > ___ > Python tracker > <https://bugs.python.org/issue43148> > ___ > -- nosy: +Julian2 ___ Python tracker <https://bugs.python.org/issue43148> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43154] code.InteractiveConsole can crash if sys.excepthook is broken
Change by Julian Berman : -- nosy: +Julian ___ Python tracker <https://bugs.python.org/issue43154> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43148] Call sys.unraisablehook in the REPL when sys.excepthook is broken
New submission from Julian Berman : At the REPL, when sys.excepthook is broken (below by setting it to a non-callable), one sees: ``` ⊙ python3.9 julian@Airm Python 3.9.1 (default, Feb 2 2021, 22:54:59) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys; sys.excepthook = object(); asdf Error in sys.excepthook: TypeError: 'object' object is not callable Original exception was: Traceback (most recent call last): File "", line 1, in NameError: name 'asdf' is not defined ``` The implementation (https://github.com/python/cpython/blob/5f18c223391eef8c7d01241b51a7b2429609dd84/Python/pythonrun.c#L805-L817) seems to do so by reimplementing something like 3.8+'s sys.unraisablehook (well, technically it predates sys.unraisablehook by quite a while). Seems like now that it exists, that code should now call sys.unraisablehook. -- messages: 386569 nosy: Julian, vstinner priority: normal severity: normal status: open title: Call sys.unraisablehook in the REPL when sys.excepthook is broken versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue43148> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41745] BoundArguments.arguments used in the recommended way to call a callable silently succeeds for nonexistent arguments
Julian Berman added the comment: Totally fair! Sorry, was just making sure the label change wasn't intended to say it *wasn't* enough to warrant a design change :) (which I agree should be discussed with folks who do use that functionality, of which I only recently had to). -- ___ Python tracker <https://bugs.python.org/issue41745> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41745] BoundArguments.arguments used in the recommended way to call a callable silently succeeds for nonexistent arguments
Julian Berman added the comment: Not sure I agree with it being just a doc issue -- happy to clarify if something was unclear, not sure from your message if it was or if you disagree, but e.g.: > However, your 'two' function takes no arguments, so valid values of args and > kwargs must be empty for them to be used in a call. In all cases, args() and > kwargs() must look at the signature to see which key-value pairs they should > extract from arguments. Right, and that's "dangerous" behavior to me -- BoundArguments.arguments will happily let you add arguments to it, but does no error checking at that point, so does not raise an exception if you typo an argument that isn't in the signature of the callable, and then when you try to call the callable (via the only way possible, namely using .args and .kwargs), that argument that was never added is just dropped. Does that make sense? Or were you disagreeing with that being undesirable behavior? -- ___ Python tracker <https://bugs.python.org/issue41745> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41745] BoundArguments.arguments used in the recommended way to call a callable silently succeeds for nonexistent arguments
Julian Berman added the comment: As a secondary behavior here, which is actually the one that matters more for my use case, the following seems surprising as well: import inspect s = inspect.signature(lambda **kwargs: kwargs).bind() s.arguments["foo"] = 12 will similarly silently drop "foo" when attempting to call the function (i.e. it will not appear in bound.kwargs). This behavior I suspect is the intention of the docs saying "Contains only explicitly bound arguments." but still seems easily misused when .arguments is being used as a frontend for adding additional arguments. To me just brainstorming again it seems it'd be useful to provide a harder-to-misuse frontend, e.g. `BoundArguments.with_arguments(*args, **kwargs)` that returns a bound arguments with some additional to-be-bound arguments which may end up in `kwargs`. -- ___ Python tracker <https://bugs.python.org/issue41745> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41745] BoundArguments.arguments used in the recommended way to call a callable silently succeeds for nonexistent arguments
Change by Julian Berman : -- nosy: +yselivanov versions: -Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue41745> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41745] BoundArguments.arguments used in the recommended way to call a callable silently succeeds for nonexistent arguments
New submission from Julian Berman : The following code succeeds "silently", which seems undesirable: from inspect import signature def two(): return 2 bound = signature(two).bind() bound.arguments["does_not_exist"] = 12 two(*bound.args, **bound.kwargs) where the mechanism for finally calling `two` is the recommended way shown in the docs for `inspect.BoundArguments`: https://docs.python.org/3/library/inspect.html#inspect.BoundArguments.apply_defaults What's happened there is that: print(b.args, b.kwargs) "silently" ignored the non-existent argument. Somewhere along the line here it seems like something should complain. I don't see previous discussion of this from quickly searching on the bug tracker, but obviously if I've missed something let me know. I'm also not really sure what the desirable solution is. To me, it's possibly that BoundArguments should have a fully-managed way to invoke a callable rather than asking a user to unpack *args and *kwargs, and that that mechanism, say arguments.be_passed_to(callable) should do the error checking). Having `.arguments` full-on reject unknown parameters seems like another possibility but there may be reasons that's not a good idea (e.g. if you were to for some reason use a bound arguments object to call some other function that *did* take that additional argument). -- components: Library (Lib) messages: 376578 nosy: Julian priority: normal severity: normal status: open title: BoundArguments.arguments used in the recommended way to call a callable silently succeeds for nonexistent arguments type: behavior versions: Python 3.10, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue41745> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40379] multiprocessing's default start method of fork()-without-exec() is broken
Change by Julian Berman : -- nosy: +Julian ___ Python tracker <https://bugs.python.org/issue40379> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21724] resetwarnings doesn't reset warnings registry
Julian Berman added the comment: Just ran into this myself -- not sure what the intended fix is (hopefully it's "add a function that restores the warnings configuration to its defaults?" Changing resetwarnings seems likely to be not doable I assume.) But in the meanwhile, is a doc patch acceptable? The current documentation for resetwarnings is really deceptive, and makes it look like it does what @pitrou (and I) thought it did: > Reset the warnings filter. This discards the effect of all previous calls to > filterwarnings(), including that of the -W command line options and calls to > simplefilter(). Compare to the docstring of the function (and to what it actually does): > """Clear the list of warning filters, so that no filters are active.""" But there are still too many implementation details of the warnings module leaking through here -- for end users it's just "restore the warnings configuration to its defaults" (what it looks like resetwarnings should do) or "unfilter all warnings, even beyond the filters configured by default" (what it actually does). Is at least making the docs reflect the latter a reasonable patch to submit, to start, while what to do about the former is thought about? -- assignee: -> docs@python components: +Documentation nosy: +Julian, docs@python versions: +Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue21724> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38895] performance degradation creating a mock object (by factor 7-8)
Change by Julian : -- title: performance degradation creating a mock object -> performance degradation creating a mock object (by factor 7-8) ___ Python tracker <https://bugs.python.org/issue38895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38895] performance degradation creating a mock object
New submission from Julian : There seems to be a performance issue when creating a Mock() object from unittest module. The performance difference between 3.7.x and 3.8.0 is about 7-8 times slower in 3.8 Heres the smalles sample i could generate: Using python 3.7.5 ``` python3 -m timeit -v --number=10 --setup="from unittest.mock import Mock" "Mock()" raw times: 2.99 sec, 2.96 sec, 3.33 sec, 2.98 sec, 2.92 sec 10 loops, best of 5: 29.2 usec per loop ``` Using python 3.8.0 ``` python3 -m timeit -v --number=10 --setup="from unittest.mock import Mock" "Mock()" raw times: 16.9 sec, 17 sec, 17.7 sec, 18.1 sec, 16.3 sec 10 loops, best of 5: 163 usec per loop ``` I did not find that issue, but a co-worker. -- messages: 357297 nosy: julianhille priority: normal severity: normal status: open title: performance degradation creating a mock object type: performance versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue38895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37941] python -m and runpy.run_module set different __name__ by default
Julian Berman added the comment: That all makes sense, I understand that in the general case you can't really promise someone that if you mutate global state in-process that the runpy module has any way of preventing that. Except IMO, the module gives exactly the impression you're saying is bad (that you want to use it if you need to do what -m does from Python, assuming you are OK with the constraints there). And then runpy.run_module is a footgun, because it does not in fact do what -m does, so it's hard to imagine the kind of person who *does* want to use runpy -- it's basically "try it on your specific -m, and possibly it works, and then possibly it will continue to work" (to be slightly but hopefully not overwhelmingly facetious). Quoting the docs for it themselves: See also The -m <https://docs.python.org/3/using/cmdline.html#cmdoption-m> option offering equivalent functionality from the command line. >From an external developer's perspective, if some other thing would be necessary tomorrow that would only be for -m, could one rely on it being added to any or all of the functions in the runpy module? That sounds unclear if there isn't a function that represents what -m does and promises to stay that way. It seems to me that either exposing a public API that promises to be -m as much as possible (which means at least if an end-user knows they're just as "clean" as the process that -m will run, you're probably good), or otherwise deprecating the functions in here entirely as public, and just making them all private, would be improvements to being able to understand if someone wants to use this module or not, but I do appreciate you explaining how things got here. -J On Wed, Sep 11, 2019 at 7:03 PM Nick Coghlan wrote: > > Nick Coghlan added the comment: > > There is one, it just isn't public: runpy._run_module_as_main. > > It's a private API that the -m switch implementation calls, and was > introduced after the original runpy.run_module was found not to offer > everything the switch needed. > > It isn't possible to fully emulate -m from Python code though, as the > lifecycle of the real main module is intertwined with the lifecycle of the > underlying interpreter. > > So if folks really want to try to do this, the source code is there to > look at, but we're not giving the false impression that it's easy to do > correctly with no unintended consequences. > > -- > > ___ > Python tracker > <https://bugs.python.org/issue37941> > ___ > -- ___ Python tracker <https://bugs.python.org/issue37941> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37941] python -m and runpy.run_module set different __name__ by default
Julian Berman added the comment: Is there no desire to have an API that works like -m (entirely. In this and any other way)? On Tue, Sep 3, 2019, 09:41 Nick Coghlan wrote: > > Change by Nick Coghlan : > > > -- > stage: -> resolved > status: open -> closed > > ___ > Python tracker > <https://bugs.python.org/issue37941> > ___ > -- ___ Python tracker <https://bugs.python.org/issue37941> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37941] python -m and runpy.run_module set different __name__ by default
New submission from Julian Berman :
This seems brutally simple, to the point where I'm concerned I'm missing
something (or have seen this issue filed elsewhere but can't find it), but
`python -m` and `runpy.run_module` don't set the same __name__ -- specifically
`runpy.run_module`, when given a non-package, defaults to setting __name__ to
`mod_name`.
So, given package/foo.py, with the "common"
`if __name__ == "__main__":` check at the bottom, `python -m package.foo`
successfully executes, but `runpy.run_module("package.foo")` exits silently,
unless explicitly passed `runpy.run_module("package.foo", run_name="__main__").
[n.b. pep517.{build,check} is a specific example of such a module that
advertises itself as wanting to be executed via `python -m`]
issue16737 seems related but not exactly the same from what I can tell.
--
messages: 350387
nosy: Julian
priority: normal
severity: normal
status: open
title: python -m and runpy.run_module set different __name__ by default
___
Python tracker
<https://bugs.python.org/issue37941>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37805] json.dump(..., skipkeys=True) has no unit tests
New submission from Julian Berman : Looks like there possibly are upstream tests that could be pulled in with modification: https://github.com/simplejson/simplejson/blob/00ed20da4c0e5f0396661f73482418651ff4d8c7/simplejson/tests/test_dump.py#L53-L66 (Found via https://bitbucket.org/pypy/pypy/issues/3052/json-skipkeys-true-results-in-invalid-json) -- components: Tests messages: 349317 nosy: Julian priority: normal severity: normal status: open title: json.dump(..., skipkeys=True) has no unit tests versions: Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue37805> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30754] textwrap.dedent mishandles empty lines
Julian Berman added the comment: I still disagree :) but docs are better than nothing. On Wed, Jun 12, 2019, 18:05 Guido van Rossum wrote: > > Change by Guido van Rossum : > > > -- > nosy: +gvanrossum > > ___ > Python tracker > <https://bugs.python.org/issue30754> > ___ > -- ___ Python tracker <https://bugs.python.org/issue30754> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12445] dict view values objects are missing tp_richcmp and tp_as_number
Julian Berman added the comment:
Yes I know *why* it worked in Py2 -- still seems like an oversight :)
To me, comparing (multi)set-like is the only reasonable behavior there
which is what IIRC the patch did, but even without that, for a given dict,
d.values() != d.values().
So, it's not like comparison is currently unimplemented. It returns
answers, they just mostly make no sense. (And of course I know that what's
happening is we're falling back to an identity check)
On Thu, Mar 28, 2019, 09:51 Inada Naoki wrote:
>
> Inada Naoki added the comment:
>
> > Well, surely there are reasonable semantics :), because dict.values ==
> > dict.values was comparable before we had view objects.
>
> Because it was list.
>
> Now values view is not sequence-like or set-like.
>
> >>> {"a": "foo", "b": "bar"}.values() == {"a": "bar", "b": "foo"}.value()
> True if set-like. False if sequence-like.
>
> If you want Python 2 behavior, you should convert it to list.
> Then you can use "sequence" semantics.
>
> --
>
> ___
> Python tracker
> <https://bugs.python.org/issue12445>
> ___
>
--
___
Python tracker
<https://bugs.python.org/issue12445>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12445] dict view values objects are missing tp_richcmp and tp_as_number
Julian Berman added the comment:
Well, surely there are reasonable semantics :), because dict.values ==
dict.values was comparable before we had view objects.
It's been awhile since I filed this, and still seems rather silly that:
>>>> {"a": "foo"}.values() != {"a": "foo"}.values()
True
On Thu, Mar 28, 2019 at 9:18 AM Inada Naoki wrote:
>
> Inada Naoki added the comment:
>
> There is no reasonable semantics for values view.
> Keep it unimplemented.
>
> --
> nosy: +inada.naoki
> resolution: -> rejected
> stage: patch review -> resolved
> status: open -> closed
>
> ___
> Python tracker
> <https://bugs.python.org/issue12445>
> ___
>
--
___
Python tracker
<https://bugs.python.org/issue12445>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34160] ElementTree not preserving attribute order
Julian Sivertsen added the comment: I don't understand why this library should go out of its way to support the old behavior when it seems like the only thing it breaks is tests that assume something that was never guaranteed and where you can get the old behavior in just two lines of Python: for node in root.iter(): node.attrib = dict(sorted(node.items())) Kind regards from confused Pythoner that just wonted the attribute order to make sense -- nosy: +sivert ___ Python tracker <https://bugs.python.org/issue34160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26467] Add async magic method support to unittest.mock.Mock
Change by Julian Mehnle : -- nosy: +jmehnle ___ Python tracker <https://bugs.python.org/issue26467> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17894] Edits to descriptor howto
Julian Berman added the comment: This seems very very slightly overly conversational (specifically the "That's all there is to it" sentence), but overall like a pretty decent improvement here. Personally I'd axe that sentence but then seems like this should be merged as-is and any further improvements could always pile on after given how long this has sat. -- nosy: +Julian ___ Python tracker <https://bugs.python.org/issue17894> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31861] aiter() and anext() built-in functions
Change by Julian Mehnle : -- nosy: +jmehnle ___ Python tracker <https://bugs.python.org/issue31861> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13041] argparse: terminal width is not detected properly
Julian Mehnle <jul...@mehnle.net> added the comment: What's holding up the merging of this patch? -- nosy: +jmehnle ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue13041> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33273] Allow multiple imports from one module while preserving its namespace
New submission from Julian DeMille <julian.demi...@demilletech.net>:
An example of this would be to have something like `import .{
, , ... }`
--
components: Interpreter Core
messages: 315247
nosy: jdemilledt
priority: normal
severity: normal
status: open
title: Allow multiple imports from one module while preserving its namespace
type: enhancement
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33273>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32804] urllib.retrieve documentation doesn't mention context parameter
New submission from Julian O <pythontrac...@somethinkodd.com>: In 2014, urlretrieve was changed to accept a context parameter to configure, for example, SSL settings. Ref: https://github.com/python/cpython/commit/b206473ef8a7abe9abf5ab8776ea3bcb90adc747 However, 2.7.14 documentation does not mention this parameter. > urllib.urlretrieve(url[, filename[, reporthook[, data]]]) Ref: https://docs.python.org/2/library/urllib.html -- assignee: docs@python components: Documentation messages: 311861 nosy: docs@python, julian_o priority: normal severity: normal status: open title: urllib.retrieve documentation doesn't mention context parameter versions: Python 2.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30754] textwrap.dedent mishandles empty lines
Julian Berman added the comment: @Terry IMHO it conflicts with the fundamental description of the function. > Remove any common leading whitespace from every line in text. If this behavior is intentional, textwrap.dedent does not do that, it does that, but also some other stuff. As for whether it *should* do that, I can't really see how it'd be a good idea to do this -- whether indented blank lines are like trailing whitespace or not is a good reason not to *add* them, but if you've got some already, textwrap.dedent isn't responsible for them, they were there already. It is also extremely hard to get the other behavior given a function that does this one, but trivial to do the reverse. Also, the #1 reason to use textwrap.dedent (and also the very next line of the docstring after the one I quoted) is to line up sections of source code. With this behavior, that usecase goes away any time you need trailing whitespace, which is exactly how I found this behavior, by unittesting a function that was returning: textwrap.dedent( """ foo ' bar baz quux ' """, ) -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30754> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30754] textwrap.dedent mishandles empty lines
New submission from Julian Berman:
⊙ python2 -c 'from textwrap import dedent; print repr(dedent(" " * 2 + "\n" +
" " * 4 + "\n"))'
'\n\n'
instead of the presumed '\n \n'
The same appears to be the case for py3.6.
(At first glance, this seems unrelated to http://bugs.python.org/issue19479
although that issue seems to have an unreviewed patch that changes the
implementation, so it might also "accidentally" fix the issue).
--
components: Library (Lib)
messages: 296825
nosy: Julian, georg.brandl, terry.reedy
priority: normal
severity: normal
status: open
title: textwrap.dedent mishandles empty lines
type: behavior
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30754>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30150] raw debug allocators to not return malloc alignment
Julian Taylor added the comment: The largest type is usually the long double. Its alignment ranges from 4 bytes (i386) to 16 bytes (sparc). So Py_MAX (sizeof (size_t), 8) should indeed do it. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30150] raw debug allocators to not return malloc alignment
Julian Taylor added the comment: no in numpy it is just a case of using the wrong allocator in a certain spot, an issue that can be fixed in numpy. But it is also minor bug/documentation issue in Python itself. Alignment isn't very important for SIMD any more but there are architectures where alignment is still mandatory so numpy is sprinkled with asserts checking alignment which triggered on x32. It is a very minor issue as to my knowledge none of the platforms with alignment requirement has the properties of x32 and x32 doesn't actually care about alignment either. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30150] raw debug allocators to not return malloc alignment
New submission from Julian Taylor: The debug raw allocator do not return the same alignment as malloc. See _PyMem_DebugRawAlloc: https://github.com/python/cpython/blob/master/Objects/obmalloc.c#L1873 The line return p + 2*SST adds 2 * sizeof(size_t) to the pointer returned by malloc. On for example x32 malloc returns 16 byte aligned memory but size_t is 4 bytes. This makes all memory returned by the debug allocators not aligned the what the system assumes on such platforms. -- components: Interpreter Core messages: 292187 nosy: jtaylor priority: normal severity: normal status: open title: raw debug allocators to not return malloc alignment versions: Python 2.7, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30150> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30073] binary compressed file reading corrupts newlines (lzma, gzip, bz2)
Julian Taylor added the comment: see also http://bugs.python.org/issue17083 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30073> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30073] binary compressed file reading corrupts newlines (lzma, gzip, bz2)
Julian Taylor added the comment: on second though not really worth an issue as it is a general problem of readline on binary streams. Sorry for the noise. -- stage: -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30073> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30073] binary compressed file reading corrupts newlines (lzma, gzip, bz2)
New submission from Julian Taylor:
Probably a case of 'don't do that' but reading lines in a compressed files in
binary mode produces bytes with invalid newlines in encodings that where '\n'
is encoded as something else:
with lzma.open("test.xz", "wt", encoding="UTF-32-LE") as f:
f.write('0 1 2\n3 4 5');
lzma.open("test.xz", "rb").readlines()[0].decode('UTF-32-LE')
Fails with:
UnicodeDecodeError: 'utf-32-le' codec can't decode byte 0x0a in position 20:
truncated data
as readlines() produces:
b'0\x00\x00\x00 \x00\x00\x001\x00\x00\x00 \x00\x00\x002\x00\x00\x00\n'
The last newline should be '\n'.encode('UTF-32-LE') == b'\n\x00\x00\x00'
--
components: Library (Lib)
messages: 291661
nosy: jtaylor
priority: normal
severity: normal
status: open
title: binary compressed file reading corrupts newlines (lzma, gzip, bz2)
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30073>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30054] Expose tracemalloc C API to track/untrack memory blocks
Julian Taylor added the comment: With this changeset it would: https://github.com/numpy/numpy/pull/8885 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30054> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30054] Expose tracemalloc C API to track/untrack memory blocks
Julian Taylor added the comment: I am not sure if _PyTraceMalloc_GetTraceback really needs to be a public function. Exposing the tracing information should probably just go over python interfaces. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30054> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26530] tracemalloc: add C API to manually track/untrack memory allocations
Julian Taylor added the comment: I don't see any reason why not to. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26530> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26530] tracemalloc: add C API to manually track/untrack memory allocations
Julian Taylor added the comment: The api looks good to me. Works fine in numpy. -- nosy: +jtaylor ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26530> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29182] Remove the warning in urllib docs that it doesn't do certificate validate by default.
Julian Berman added the comment: Cool! If I can nitpick one last time, in the versionchanged block, you have `HTTPS URIs` but in the warning you use `HTTPS Urls` -- probably best to use consistent casing and URL vs URI. Other than that, lgtm to merge. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29182] Remove the warning in urllib docs that it doesn't do certificate validate by default.
Julian Berman added the comment: The change note belongs outside the seealso. I think also that wasn't exactly what Martin had in mind, I think he meant a `.. versionchanged` directive -- and given that this was originally a warning, personally I'd leave the warning but reword it, it's still relevant for pre-2.7.9 users. E.g. something like: .. seealso:: .. versionchanged:: 2.7.9 HTTPS certificates now have hostname verification enabled by default. .. warning:: Versions prior to 2.7.9 do not safely verify hostnames. It is not recommended to use this module on these versions -- the aformentioned requests module can be used instead. Or the like. -- nosy: +Julian ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: UDP decode
Thanks for response! I don't know what can appear because the values changes 30 times per second. I only know the format. -- https://mail.python.org/mailman/listinfo/python-list
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Julian Taylor added the comment: which is exactly what malloc is already doing for, thus my point is by using malloc we would fullfill your request. But do you have an actual real work application where this would help? it is pretty easy to figure out, just run the application under perf and see if there is a relevant amount of time spent in page_fault/clear_pages. And as mentioned you can already change the allocator for arenas at runtime, so you could also try changing it to malloc and see if your application gets any faster. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Julian Taylor added the comment: I know one can change the allocator, but the default is mmap which I don't think is a very good choice for the current arena size. All the arguments about fragmentation and memory space also apply to pythons arena allocator itself and I am not convinced that fragmentation of the libc allocator is a real problem for python as pythons allocation pattern is very well behaved _due_ to its own arena allocator. I don't doubt it but I think it would be very valuable to document the actual real world use case that triggered this change, just to avoid people stumbling over this again and again. But then I also don't think that anything needs to be necessarily be changed either, I have not seen the mmaps being a problem in any profiles of applications I work with. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Julian Taylor added the comment: glibcs malloc is not obstack, its not a simple linear heap where one object on top means everything below is not freeable. It also uses MADV_DONTNEED give sbrk'd memory back to the system. This is the place where MADV_FREE can now be used now as the latter does not guarantee a page fault. But that said of course you can construct workloads which lead to increased memory usage also with malloc and maybe python triggers them more often than other applications. Is there an existing issues showing the problem? It would be a good form of documentation in the source. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Julian Taylor added the comment: it defaulted to 128kb ten years ago, its a dynamic threshold since ages. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Julian Taylor added the comment: ARENA_SIZE is 256kb, the threshold in glibc is up to 32 MB -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Julian Taylor added the comment: simplest way to fix this would be to not use malloc instead of mmap in the allocator, then you also get MADV_FREE for free when malloc uses it. The rational for using mmap is kind of weak, the source just says "heap fragmentation". The usual argument for using mmap is not that but the instant return of memory to the system, quite the opposite of what the python memory pool does. -- nosy: +jtaylor ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19959] argparse.FileType does not expand tilde "~"
Julian Berman added the comment: My support (or really, asking for a more generic callable) was to enable other use cases, not this one specifically -- e.g., allowing for constructing a pathlib Path or a twisted.python.filepath.FilePath without needing to write one's own argparse action. For ~ specifically -- it's true if you're going to go via a shell it'd be expanded first, but I'm not sure that'd count as "conflicting" with the shell's expansion, it just won't be needed. Consider that twisted.python.filepath.FilePath for example, expands ~ in its constructor -- people often construct paths in other places and occasionally expansion is convenient. Like in test cases. I certainly don't feel strongly about the patch though, especially as-is. -- status: pending -> open ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue19959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24961] shell stdout broken after exiting interactive python prompt
Changes by Julian Mehnle <jul...@mehnle.net>: -- nosy: +jmehnle ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24961> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23601] use small object allocator for dict key storage
Julian Taylor added the comment: ping, this has been sitting for 4 years and two python releases. Its about time this stupidly simple thing gets merged. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue23601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21592] Make statistics.median run in linear time
Julian Taylor added the comment: the median of median of 5 is quite significantly slower than a quickselect. numpy implements an introselect which uses quickselect but falls back to median of median of 5 if not enough progress is done. In the numpy implementation for 10 element median (multiselect with 2 selections, one median one min) quickselect is around 3 times faster than mom5 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue21592> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4963] mimetypes.guess_extension result changes after mimetypes.init()
Julian Sivertsen added the comment:
I bumped into a similar issue with mimetypes.guess_extension on Arch Linux
64-bit in February. The behavior is still present in python 3.4.3.
$ python test.py
.htm
$ python test.py
.html
$ cat test.py
from mimetypes import guess_extension
print(guess_extension('text/html'))
$ python
Python 3.4.3 (default, Mar 25 2015, 17:13:50)
[GCC 4.9.2 20150304 (prerelease)] on linux
Type help, copyright, credits or license for more information.
--
nosy: +sivert
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4963
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23601] use small object allocator for dict key storage
Julian Taylor added the comment: ok I ran it again, but note the machine was under use the full time so the results are likely have no meaning. python perf.py -r -b default /tmp/normal/bin/python3 /tmp/opt/bin/python3 Min: 0.399279 - 0.376527: 1.06x faster Avg: 0.410819 - 0.383315: 1.07x faster Significant (t=49.29) Stddev: 0.00450 - 0.00330: 1.3631x smaller ### etree_iterparse ### Min: 0.639638 - 0.630989: 1.01x faster Avg: 0.658744 - 0.641842: 1.03x faster Significant (t=14.82) Stddev: 0.00959 - 0.00617: 1.5557x smaller ### etree_parse ### Min: 0.433050 - 0.377830: 1.15x faster Avg: 0.444014 - 0.389695: 1.14x faster Significant (t=43.28) Stddev: 0.01010 - 0.00745: 1.3570x smaller ### tornado_http ### Min: 0.335834 - 0.326492: 1.03x faster Avg: 0.346100 - 0.334186: 1.04x faster Significant (t=13.66) Stddev: 0.01024 - 0.00689: 1.4864x smaller The following not significant results are hidden, use -v to show them: 2to3, django_v2, etree_process, fastpickle, fastunpickle, json_dump_v2, json_load, nbody, regex_v8. /tmp$ /tmp/normal/bin/python3 -c 'import timeit; print(timeit.repeat(dict(a=5, b=2)))' [0.5112445619997743, 0.514110946735, 0.5185121280010208] /tmp$ /tmp/opt/bin/python3 -c 'import timeit; print(timeit.repeat(dict(a=5, b=2)))' [0.4426167189994885, 0.4465744609988178, 0.4467797579982289] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23601 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23601] use small object allocator for dict key storage
Julian Taylor added the comment: Your benchmarks are not affected by this change see the other issue. They are also not representative of every workload out there. I can at least see the argument why you didn't want to put the other variant of this change in as it made the code a tiny bit more complicated, but I do not understand the reluctance for this variant. It doesn't change the complexity of the code one bit. If you doubt the performance of pythons own small object allocator, python should maybe stop using it alltogether? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23601 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23601] use small object allocator for dict key storage
Julian Taylor added the comment: Large objects are just if size 512: return malloc(size) there is no reason it should be slower. Also for large objects allocation speed does not matter as much. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23601 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21148] avoid needless pointers initialization in small tuple creation
Julian Taylor added the comment: right at best its probably too insignificant to really be worthwhile, closing. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21148 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Julian Taylor added the comment: any comments on the doc changes? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23601] use small object allocator for dict key storage
New submission from Julian Taylor: dictionary creation spends a not insignificant amount of time in malloc allocating keys objects. Python has a nice small object allocator that avoids a lot of this overhead and falls back to malloc for larger allocations. Is there a reason the dictionary does not use that allocator for its keys objects? doing so e.g. via attached incomplete patch improves small dict creation performance by 15%. import timeit print(timeit.repeat(dict(a=5, b=2))) with change: [0.4282559923725, 0.427258015296, 0.436232985377] without [0.516061002634, 0.518172000496, 0.51842199191] or is there something I am overlooking and the use of PyMem_Malloc instead of PyObject_Malloc is an intentional design decision? -- components: Interpreter Core files: 0001-use-small-object-allocator-for-keys-object.patch keywords: patch messages: 237439 nosy: jtaylor priority: normal severity: normal status: open title: use small object allocator for dict key storage versions: Python 3.5 Added file: http://bugs.python.org/file38371/0001-use-small-object-allocator-for-keys-object.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23601 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Julian Taylor added the comment: attached documentation update patch. -- keywords: +patch Added file: http://bugs.python.org/file38369/0001-Issue-23530-Update-documentation-clarify-relation-of.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23601] use small object allocator for dict key storage
Julian Taylor added the comment: PyObject_Malloc just calls malloc above the threshold so there is no problem for larger dicts. For larger dicts the performance of malloc is also irrelevant as the time will be spent elsewhere. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23601 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Julian Taylor added the comment: certainly for anything that needs good control over affinity psutils is the best choice, but I'm not arguing to implement full process control in python. I only want python to provide the number of cores one can work on to make best use of the available resources. If you code search python files for cpu_count you find on github 18000 uses, randomly sampling a few every single one was to determine the number of cpus to start worker jobs to get best performance. Every one of these will oversubscribe a host that restricts the cpus a process can use. This is an issue especially for the increasingly popular use of containers instead of full virtual machines. as a documentation update I would like to have a note saying that this number is the number of (online) cpus in the system may not be the number of of cpus the process can actually use. Maybe with a link to len(psutils.Process.get_affinity()) as a reference on how to obtain that number. there would be no dependence on coreutils, I just mentioned it as you can look up the OS api you need to use to get the number there (e.g. sched_getaffinity). It is trivial API use and should not be a licensing issue, one could also look at the code from psutil which most likely looks very similar. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Julian Taylor added the comment: oh thats great so python already has what I want. Then just an small documentation update would be good, I'll have a go at a patch later. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
New submission from Julian Taylor: multiprocessing.cpu_count and os.cpu_count which are often used to determine how many processes one can run in parallel do not respect the cpuset which may limit the process to only a subset of online cpus leading to heavy oversubscription in e.g. containerized environments: $ taskset -c 0 python3.4 -c 'import multiprocessing; print(multiprocessing.cpu_count())' 32 $ taskset -c 0 python3.4 -c 'import os; print(os.cpu_count())' 32 While the correct result here should be 1. This requires programs to have to use less portable methods like forking to gnu nproc or having to read the /proc filesystem. Having a keyword argument to switch between online and available cpus would be fine too. -- components: Library (Lib) messages: 236671 nosy: jtaylor priority: normal severity: normal status: open versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Julian Taylor added the comment: I do agree that its probably safer to not change the default return value. But adding a option (or new function) would still be good, the number of available cpus is more often the number you actually want in practice. To the very least the documentation should be improved to clearly state that this number does not guarantee that this amount of cpus are actually available to run on and you should use psutils instead. Code for getting this information for the major operating systems linux, bsd and windows is available in gnu coreutils. I can possibly work on a patch if it would get accepted but I can only actually test it linux. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20598] argparse docs: '7'.split() is confusing magic
Julian Berman added the comment: +1 to lists all over, this is just confusing. -- nosy: +Julian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20598 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5309] distutils doesn't parallelize extension module compilation
Julian Taylor added the comment: very nice, thanks for adding this. coincidentally numpy added the same to numpy.distutils independently just a week later, though numpy also accepts an environment variable to set the number of jobs. This is useful for e.g. pip installations where one does not control the command line. Also an environment variable allows parallel jobs in environments where it is not guaranteed that the feature is available. E.g. you could just put it into your .bashrc and when building with 3.5 it will just work and 2.7 will not fail. Is the naming --parallel/j already fixed? I'll change the numpy options to the same name then. Please also add it to the release notes so the feature can be discovered easier. -- nosy: +jtaylor ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5309 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23150] urllib parse incorrect handing of params
Julian Reschke added the comment: An example URI for this issue is: http://example.com/; The RFC 3986 path component for this URI is /;. After using urllib's parse function, how would you know? (I realize that changing behavior of the existing API may cause problems, but this is an information loss issue). One ugly, but workable way to fix this would be to also provide access to a RFC3986path component. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23150 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23150] urllib parse incorrect handing of params
New submission from Julian Reschke: urllib.parse tries to special-case params, which have been dropped from the general URI syntax back in RFC 2396 (16 years ago). In most cases this can be worked around by reconstructing the path from both path and params; however this fails for paths that *end* in a semicolon (because it's not possible to distinguish an empty param from an absent param). -- components: Library (Lib) messages: 233312 nosy: julian.resc...@gmx.de priority: normal severity: normal status: open title: urllib parse incorrect handing of params type: behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23150 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21963] 2.7.8 backport of Issue1856 (avoid daemon thread problems at shutdown) breaks ceph
Changes by Joe Julian jjul...@io.com: -- nosy: +Joe.Julian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21963 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21963] 2.7.8 backport of Issue1856 (avoid daemon thread problems at shutdown) breaks ceph
Joe Julian added the comment: I suspect the underlying problem is that the fix expects the daemon threads to hit a point where they try to acquire the GIL and that's not going to happen with these librados threads; they stay in librados. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21963 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11664] Add patch method to unittest.TestCase
Julian Berman added the comment: My opinion is already here re: patch vs patch.object, so I won't repeat it, but @Michael, if you really want .patch, are you open to adding .patch_object as well? (Regardless, thanks for working on this Julien.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11664 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: O'Reilly Python Certification
On Wednesday, September 3, 2014 5:53:12 PM UTC-4, jaron...@gmail.com wrote: Ethan, Steve, Tim, and others: I'm thinking of taking the program. How long, in hours, does it take to complete all four Python courses? I'm currently taking the first out of four modules. I have extensive PHP knowledge and I also have a certification by Zend so programming is not new to me. I've never done anything at all with Python. I've already completed 12 out of the 16 lessons and so far the reading has taken me from 15 to 30 minutes and the code assignments no more than 10. There was one exception that took me 30 minutes as I wasn't sure what the hell they were asking (it was a very obscure and useless algorithm which hasn't been the case on any of the other assignments). On top of that you have to answer 2 quizzes and each quiz usually has 2 questions (no more than 5). Those usually take 1 to 3 minutes max as they are mostly syntax or concepts but not coding. As I said, this is only true for the first module as I haven't taken the others yet. But if you already know programming and have a few years of coding experience you will breeze through the first module in probably 3 days (assuming you only do that). The syllabus for the other modules suggests (to me) that going so fast would not be that easy. -- https://mail.python.org/mailman/listinfo/python-list
[issue18983] Specify time unit for timeit CLI
Julian Gindi added the comment: Anything else need to be done on this patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18983 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21720] TypeError: Item in ``from list'' not a string message
Julian Gindi added the comment:
I'm trying to replicate this issue. I do not get an error when I run
```
__import__('datetime', fromlist=[u'datetime'])
```
any more information you could provide to help move this issue forward?
--
nosy: +Julian.Gindi
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21720
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21720] TypeError: Item in ``from list'' not a string message
Julian Gindi added the comment: Interesting...I'll try to dig in and see what's going on. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21720 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22189] collections.UserString missing some str methods
Julian Gindi added the comment: Good catch. I'm gonna look into this. Seems like you should be able to access these from UserString as well. -- nosy: +Julian.Gindi ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22189 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22088] base64 module still ignores non-alphabet characters
New submission from Julian Berman: The base64 module documentation claims that decode methods raise exceptions for non-base64 input, but they do not. There was a patch for Py3 done in issue1466065, but the documentation was not updated for Py2. I have not read that ticket carefully enough to be able to tell what the expected resolution was for Py2 (fixing the bug or just updating the docs). -- assignee: docs@python components: Documentation, Library (Lib) messages: 224089 nosy: Julian, docs@python, r.david.murray priority: normal severity: normal status: open title: base64 module still ignores non-alphabet characters type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22088 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1466065] base64 module ignores non-alphabet characters
Julian Berman added the comment: Created issue22088 to address not having fixed Py2 here. -- nosy: +Julian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1466065 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21592] Make statistics.median run in linear time
Julian Taylor added the comment: for median alone a multiselect is probably overkill (thats why I mentioned the minimum trick) but a selection algorithm is useful on its own for all of python and then a multiselect should be considered. Of course that means it would need to be implemented in C like sorted() so you actually have a significant performance gain that makes adding a new python function worthwhile. Also just to save numpys honor, you are benchmarking python list - numpy array conversion and not the actual selection in your script with the numpy comparison. The conversion is significantly slower than the selection itself. Also select2b is inplace while np.partition is out of place. Repeated inplace selection typically gets faster as the number of required swaps goes down and can even be constant in time if the requested value does not change. With that fixed numpy outperforms pypy by about a factor 2 (but pypys performance is indeed quite impressive as it is far more generic) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21592 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21690] re documentation: re.compile links to re.search / re.match instead of regex.search / regex.match
New submission from Julian Gilbey: In re.rst, the re.compile documentation says: Compile a regular expression pattern into a regular expression object, which can be used for matching using its :func:`match` and :func:`search` methods, described below. This results in linking to the re.match and re.search module functions instead of the regex.match and regex.search object methods. I'm not sure what the correct replacement syntax is, presumably something like :meth:`~regex.match` and :meth:`~regex.search`. -- assignee: docs@python components: Documentation messages: 219997 nosy: docs@python, jdg priority: normal severity: normal status: open title: re documentation: re.compile links to re.search / re.match instead of regex.search / regex.match versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21690 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21592] Make statistics.median run in linear time
Julian Taylor added the comment: in the case of the median you can archive similar performance to a multiselect by simply calling min([len(data) // 2 + 1]) for the second order statistic which you need for the averaging of even number of elements. maybe an interesting datapoint would be to compare with numpys selection algorithm which is a intromultiselect (implemented in C for native datattypes). It uses a standard median of 3 quickselect with a cutoff in recursion depth to median of median of group of 5. the multiselect is implemented using a sorted list of kth order statistics and reducing the search space for each kth by maintaining a stack of all visited pivots. E.g. if you search for 30 and 100, when during the search for 30 one has visited pivot 70 and 110, the search for 100 only needs to select in l[70:110]. The not particularly readable implementation is in: ./numpy/core/src/npysort/selection.c.src unfortunately for object types it currently falls back to quicksort so you can't directly compare performance with the pure python variants. -- nosy: +jtaylor ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21592 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21233] Add *Calloc functions to CPython memory allocation API
Julian Taylor added the comment: won't replacing _PyObject_GC_Malloc with a calloc cause Var objects (PyObject_NewVar) to be completely zeroed which I think they didn't before? Some numeric programs stuff a lot of data into var objects and could care about python suddenly setting them to zero when they don't need it. An example would be tinyarray. -- nosy: +jtaylor ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21233 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
