[issue29336] merge tuples in module

[STINNER Victor]

STINNER Victor added the comment:

I vote +0 on merge-constants.patch: LGTM, may be useful, but not impressed :-)

Hopefully, .py => .pyc compilation "should" occur only once. Except of the main 
script, but more and more applications use a tiny entry point of 3 lines. I 
don't expect any significant slowdown. But even if there is a slowdown, IMHO 
it's worth it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> merge-constants.patch looks simple enought, but I'm not really impressed by 
> such result. Is 2% worth it?

I'm not impressed too. And merging constants can take time.

> Since code objects loaded by import are likely for stay for the whole 
> lifetime of a process, I would be interested to experiment interning all 
> constant objects (the tuple of objects, but also each object of these tuples, 
> so support "nested interning") of all code objets (consts, names, varnames, 
> freevars, cellvars) in the marshal module.

This could slow down marhalling. Some patches in issue20416 implements this, 
but current code is faster and much simpler and the benefit of deduplicating 
constants is too small.

> More generally, would it be possible to share co_consts (None,) tuples 
> between code objects of two different modules? Or is it already the case with 
> merge-constants.patch?

All patches in this issue shares constants only in one compilation unit 
(module). Otherwise the global intern dict would increase the memory usage too 
much.

See also the statistics in issue16475. The most memory is spent by strings, and 
the majority of strings already are interned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[INADA Naoki]

INADA Naoki added the comment:

> More generally, would it be possible to share co_consts (None,) tuples 
> between code objects of two different modules? Or is it already the case with 
> merge-constants.patch?

No. I didn't do it because I don't know it's worth enough.

> merge-constants.patch looks simple enought, but I'm not really impressed by 
> such result. Is 2% worth it?

Maybe, our application is somewhat special.
I'll check memory usage again with only loading major OSS framework/libraries 
for this issue in next time.
(BTW, it's 3% if -OO is used)

But issue28813 or other optimizations may reduce tuple size and make it 
possible to share more tuples.
So I think we should evaluate this patch after them.

But I don't familiar with frontend (parser, AST).
I usually look bytecode and runtime.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29121] sqlite3 Controlling Transactions documentation not updated

[Ma Lin]

Changes by Ma Lin :


--
nosy: +Ma Lin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29365] parallel make test or make install can fail building pgen when configure --enable-optimizations is used

[Gregory P. Smith]

Gregory P. Smith added the comment:

great, the issue you filed covers my second paragraph comment which really 
should be its own issue. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14074] argparse allows nargs>1 for positional arguments but doesn't allow metavar to be a tuple

[paul j3]

paul j3 added the comment:

An alternative fix is to disallow tuple metavars for positionals.  

A tuple metavar may look nice in the usage.  But a well selected dest seems 
better in the help line, and error messages.  Using dest in all 3 roles, as we 
do now, is the simplest compromise.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[INADA Naoki]

INADA Naoki added the comment:

That's why I want to enable only --with-lto

--
nosy: +inada.naoki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29310] Document typing.NamedTuple default argument syntax

[Jelle Zijlstra]

Changes by Jelle Zijlstra :


--
assignee: Jelle Zijlstra -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29310] Document typing.NamedTuple default argument syntax

[Jelle Zijlstra]

Changes by Jelle Zijlstra :


--
keywords: +patch
Added file: http://bugs.python.org/file46410/issue29310.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29366] os.listdir has inconsistent behavior when run on a non-directory

[raylu]

New submission from raylu:

According to
https://github.com/python/cpython/blob/4ed71dae9d23a412f9f73d3a0b1f4be37543e49e/Lib/test/test_unicode_file_functions.py#L106
listdir can sometimes return FileNotFoundError on Windows.

Tangentially related: https://hg.python.org/cpython/rev/385c2ec78f16

--
components: Library (Lib), Windows
messages: 286225
nosy: paul.moore, raylu, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: os.listdir has inconsistent behavior when run on a non-directory

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29365] parallel make test or make install can fail building pgen when configure --enable-optimizations is used

[Xiang Zhang]

Xiang Zhang added the comment:

> Making the suggested "make, make test, make install" process effectively 
> useless or at least take 3x longer than it should.

About this, I once opened #29243.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29365] parallel make test or make install can fail building pgen when configure --enable-optimizations is used

[Gregory P. Smith]

New submission from Gregory P. Smith:

If you do a make -j12 test or make install before having run a separate 'make 
all' the build _can_ fail trying to build pgen.  (it seems to be parallel make 
related, it won't always fail)

Even if you have run a separate 'make all', the make test and make install 
steps are likely to rebuild a ton of stuff and rerun the profile generation 
stage.  Making the suggested "make, make test, make install" process 
effectively useless or at least take 3x longer than it should.

something along the lines of:

oss/cpython$ make -j12 install
gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  
Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o 
Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o 
Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o 
Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o 
Parser/printgrammar.o Parser/pgenmain.o -lpthread -ldl  -lutil -o Parser/pgen
Parser/grammar1.o: In function `PyGrammar_LabelRepr':
/home/gps/oss/cpython/build27/../2.7/Parser/grammar1.c:35: undefined reference 
to `__gcov_indirect_call_profiler'
Parser/grammar1.o: In function `PyGrammar_FindDFA':
/home/gps/oss/cpython/build27/../2.7/Parser/grammar1.c:13: undefined reference 
to `__gcov_indirect_call_profiler'
Parser/grammar1.o:(.data+0x80): undefined reference to `__gcov_merge_add'
Parser/grammar1.o: In function `_GLOBAL__sub_I_65535_0_PyGrammar_FindDFA':
/home/gps/oss/cpython/build27/../2.7/Parser/grammar1.c:57: undefined reference 
to `__gcov_init'
... a bunch more similar errors omitted ...
Parser/pgenmain.o: In function `_GLOBAL__sub_I_65535_0_Py_Exit':
/home/gps/oss/cpython/build27/../2.7/Parser/pgenmain.c:173: undefined reference 
to `__gcov_init'
collect2: error: ld returned 1 exit status
make: *** [Parser/pgen] Error 1
make: *** Waiting for unfinished jobs
Parser/pgen ../2.7/Grammar/Grammar Include/graminit.h Python/graminit.c
make[2]: Parser/pgen: Command not found
make[2]: *** [Include/graminit.h] Error 127
make[2]: *** Waiting for unfinished jobs
make[2]: Leaving directory `/home/gps/oss/cpython/build27'
make[1]: *** [build_all_generate_profile] Error 2
make[1]: Leaving directory `/home/gps/oss/cpython/build27'
make: *** [profile-opt] Error 2

--
components: Build
messages: 286223
nosy: gregory.p.smith
priority: normal
severity: normal
status: open
title: parallel make test or make install can fail building pgen when configure 
--enable-optimizations is used
type: compile error
versions: Python 2.7, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14074] argparse allows nargs>1 for positional arguments but doesn't allow metavar to be a tuple

[paul j3]

Changes by paul j3 :


--
priority: normal -> high

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29311] Argument Clinic: convert dict methods

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thank you for your review Martin.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29311] Argument Clinic: convert dict methods

[Roundup Robot]

Roundup Robot added the comment:

New changeset ffc0840762e4 by Serhiy Storchaka in branch 'default':
Issues #29311, #29289: Fixed and improved docstrings for dict and OrderedDict
https://hg.python.org/cpython/rev/ffc0840762e4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29289] Convert OrderedDict methods to Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:

New changeset ffc0840762e4 by Serhiy Storchaka in branch 'default':
Issues #29311, #29289: Fixed and improved docstrings for dict and OrderedDict
https://hg.python.org/cpython/rev/ffc0840762e4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17720] pickle.py's load_appends should call append() on objects other than lists

[Alexandre Vassalotti]

Alexandre Vassalotti added the comment:

I don't recall where I picked up the referred spec. Looking at PEP 307 and 
pickle docs [1], it does look like that objects which provides the listitems 
field of the reduce tuple should support extend() too.

It might best to try both in the implementation to avoid breaking existing code 
that relies on the previous behavior.

[1]: https://docs.python.org/3/library/pickle.html#object.__reduce__

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28845] Clean up known issues for AIX

[Eric N. Vander Weele]

Eric N. Vander Weele added the comment:

> I request that you review issue27435 - in particular msg284557 - as I feel 
> ctypes implementation for AIX is broken - at least as far as the supporting 
> routines are concerned.

I believe this request is outside the scope of this particular issue.

> [...] AIX to test it.

It appears there is an AIX system in Python's buildbot farm to validate the 
curses example.  Is the next step have a core maintainer conduct the validation 
on an AIX system before this can move forward?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29121] sqlite3 Controlling Transactions documentation not updated

[Aviv Palivoda]

Aviv Palivoda added the comment:

I would just like to note that I think that the correct solution is to do an 
implicit commit before:
1. VACUUM
2. ATTACH
3. DETACH
4. BEGIN

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29358] Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster object instanciation

[STINNER Victor]

STINNER Victor added the comment:

First benchmark result on the fast_init compared to the default branch. 
Warning: the patch is still a WIP, there are maybe performance issues (in this 
case, this benchmark should help to identify them).
---
haypo@speed-python$ python3 -m perf compare_to 
2017-01-23_14-02-default-cebc9c7ad195.json fast_init-0_ref_cebc9c7ad195.json -G 
--min-speed=2
Slower (6):
- telco: 13.0 ms +- 0.2 ms -> 13.8 ms +- 0.3 ms: 1.06x slower (+6%)
- scimark_sor: 391 ms +- 7 ms -> 413 ms +- 11 ms: 1.06x slower (+6%)
- fannkuch: 938 ms +- 7 ms -> 973 ms +- 13 ms: 1.04x slower (+4%)
- pickle_dict: 53.7 us +- 3.4 us -> 55.5 us +- 5.0 us: 1.03x slower (+3%)
- sqlite_synth: 8.00 us +- 0.12 us -> 8.25 us +- 0.14 us: 1.03x slower (+3%)
- scimark_monte_carlo: 202 ms +- 5 ms -> 207 ms +- 5 ms: 1.03x slower (+3%)

Faster (19):
- scimark_lu: 362 ms +- 13 ms -> 335 ms +- 17 ms: 1.08x faster (-8%)
- chameleon: 23.5 ms +- 0.3 ms -> 21.7 ms +- 0.3 ms: 1.08x faster (-7%)
- xml_etree_process: 179 ms +- 3 ms -> 168 ms +- 3 ms: 1.07x faster (-6%)
- pickle_pure_python: 1.03 ms +- 0.02 ms -> 982 us +- 11 us: 1.05x faster (-5%)
- regex_dna: 283 ms +- 1 ms -> 271 ms +- 1 ms: 1.05x faster (-4%)
- chaos: 239 ms +- 2 ms -> 230 ms +- 3 ms: 1.04x faster (-4%)
- django_template: 356 ms +- 5 ms -> 343 ms +- 5 ms: 1.04x faster (-4%)
- call_simple: 10.7 ms +- 0.4 ms -> 10.3 ms +- 0.3 ms: 1.04x faster (-3%)
- hexiom: 18.0 ms +- 0.1 ms -> 17.4 ms +- 0.1 ms: 1.04x faster (-3%)
- pathlib: 42.3 ms +- 0.5 ms -> 40.9 ms +- 0.4 ms: 1.03x faster (-3%)
- regex_compile: 379 ms +- 2 ms -> 367 ms +- 4 ms: 1.03x faster (-3%)
- go: 501 ms +- 7 ms -> 485 ms +- 4 ms: 1.03x faster (-3%)
- xml_etree_generate: 208 ms +- 4 ms -> 202 ms +- 3 ms: 1.03x faster (-3%)
- deltablue: 14.5 ms +- 0.3 ms -> 14.1 ms +- 0.2 ms: 1.03x faster (-3%)
- unpickle_pure_python: 652 us +- 8 us -> 634 us +- 9 us: 1.03x faster (-3%)
- richards: 147 ms +- 2 ms -> 143 ms +- 4 ms: 1.03x faster (-3%)
- nqueens: 213 ms +- 2 ms -> 208 ms +- 2 ms: 1.03x faster (-2%)
- raytrace: 1.08 sec +- 0.01 sec -> 1.05 sec +- 0.02 sec: 1.02x faster (-2%)
- unpickle: 31.0 us +- 0.4 us -> 30.3 us +- 1.0 us: 1.02x faster (-2%)

Benchmark hidden because not significant (34): (...)
Ignored benchmarks (5) of 2017-01-23_14-02-default-cebc9c7ad195.json: mako, 
sympy_expand, sympy_integrate, sympy_str, sympy_sum
---

Reminder: The fast_init branch is based on the issue #29259 (tp_fastcall), so 
the result combines tp_fastcall and tp_fastnew/fastinit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28810] Document bytecode changes in 3.6

[Brett Cannon]

Brett Cannon added the comment:

I'll review your patch sometime this week, Serhiy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28810] Document bytecode changes in 3.6

[Brett Cannon]

Changes by Brett Cannon :


--
assignee: docs@python -> brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29363] allow Python code to determine class help text

[Ethan Furman]

Ethan Furman added the comment:

Probably not because an enum class' __call__ comes from the type EnumMeta -- so 
having two different __call__ methods would mean two different metaclasses, and 
I'm pretty sure I don't want to go there.  ;)

There is a __doc__ defined for the __call__ method, though -- is there a reason 
inspect isn't using that?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Roundup Robot]

Roundup Robot added the comment:

New changeset 95e184bd2d89 by Ethan Furman in branch '3.6':
closes issue29167: fix race condition in (Int)Flag
https://hg.python.org/cpython/rev/95e184bd2d89

New changeset e6b98c270718 by Ethan Furman in branch 'default':
issue29167: fix race condition in (Int)Flag
https://hg.python.org/cpython/rev/e6b98c270718

--
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28810] Document bytecode changes in 3.6

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Several third-party bytecode manipulating projects still are not updated to 
3.6. Correct documentation is needed for them. Could anyone please make a 
review of the patch (or maybe totally rewrite it)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29363] allow Python code to determine class help text

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I'm wondering is it worth to set different __call__ methods for enum types with 
and without members?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29363] allow Python code to determine class help text

[Ethan Furman]

Ethan Furman added the comment:

There are actually two signatures:

EnumCls(value) --> return member with value `value`

EnumCls(name, members, module, qualname, type, start) --> create new Enum

An example of the first:

class A(Enum):
x = 1
A(1) --> 

an example of the second:

class A(Enum):
pass
B = A('B', {'y':2})
B(2) --> 

The reason for the error you see is that Enums with members cannot be further 
subclassed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29083] Readd PyArg_VaParse to the stable API

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5cdf383ef35c by Serhiy Storchaka in branch '3.5':
Issue #29083: Fixed the declaration of some public API functions.
https://hg.python.org/cpython/rev/5cdf383ef35c

New changeset 8eae65136000 by Serhiy Storchaka in branch '3.6':
Issue #29083: Fixed the declaration of some public API functions.
https://hg.python.org/cpython/rev/8eae65136000

New changeset b63d096403f4 by Serhiy Storchaka in branch 'default':
Issue #29083: Fixed the declaration of some public API functions.
https://hg.python.org/cpython/rev/b63d096403f4

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29337] BytesWarning at compile time

[STINNER Victor]

STINNER Victor added the comment:

I'm surprised that removing the type from the key works. Since tests pass,
it's ok for me :-)

I'm happy that I spent time to write these tests, I'm now more confident
when this code is modified. Serhiy asked me to write these tests if I
recall correctly ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29358] Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster object instanciation

[Josh Rosenberg]

Josh Rosenberg added the comment:

So just to be clear, the issue with non-string or non-unique keywords is purely 
about performance, right, not correctness/non-crashiness? Non-string keywords, 
while technically accepted by CPython, are at best barely legal by the language 
standard. The language standard for calls specifies that even using the 
**expression syntax "expression must evaluate to a mapping, the contents of 
which are treated as additional keyword arguments" and keywords_arguments only 
allows identifier=expression, where identifier is a legal variable name (and 
therefore, a str).

If misuse of ** unpacking is slower, but still works, and correct usage is 
significantly faster, I wouldn't consider an inability to fix the performance 
for the misuse case a blocking issue; nice to have, but making common, good 
code faster is worth making rare, bad code slower.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29352] provide the authorative source for s[i:j] negative slice indices (<-len(s)) behavior for standard sequences

[Josh Rosenberg]

Josh Rosenberg added the comment:

I think the wording could be improved, but there is another option I wanted to 
put here. Right now, we're being overly detailed about the implementation, 
specifying the bounds substitutions performed. If we're just trying to describe 
logical behavior, we could simplify footnote 4 to, dropping explicit 
descriptions for "out of bounds" cases, getting:

The slice of *s* from *i* to *j* is defined as the sequence of items with index 
*k* such that ``i <= k < j`` and ``0 <= k < len(s)``. If *i* is omitted or 
``None``, use ``0``. If *j* is omitted or ``None``, use ``len(s)``. If *i* is 
greater than or equal to *j*, the slice is empty.

That avoids needing to be explicit about substitutions in the < -len(s) case 
and > len(s) cases, since limiting the values of k to the intersection of 
range(i, j) and range(len(s)) covers both ends. I considered a single range, 
like ``max(0, i) <= k < min(j, len(s))``, but that wouldn't describe the upper 
bound on i or the lower bound on j properly, and ``min(max(0, i), len(s)) <= k 
< min(max(0, j), len(s))`` is ugly. Footnote 3 covers the adjustment for 
negative values already, which allows for the simpler description.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29337] BytesWarning at compile time

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29337] BytesWarning at compile time

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2bd2ef1696cf by Serhiy Storchaka in branch '3.5':
Issue #29337: Fixed possible BytesWarning when compare the code objects.
https://hg.python.org/cpython/rev/2bd2ef1696cf

New changeset f6c327f2daa6 by Serhiy Storchaka in branch '3.6':
Issue #29337: Fixed possible BytesWarning when compare the code objects.
https://hg.python.org/cpython/rev/f6c327f2daa6

New changeset 00fb30d4905d by Serhiy Storchaka in branch 'default':
Issue #29337: Fixed possible BytesWarning when compare the code objects.
https://hg.python.org/cpython/rev/00fb30d4905d

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29364] msilib Fetch raises MSIError rather than return None

[Jason Matthew]

New submission from Jason Matthew:

Fairly new to MSIs here.  Working towards extracting the same information which 
is available via Orca graphical interfaces.  

I believe I've hit a bug within _msi.View class, namely the inability to safely 
determine the number of records in my result.  

Examples I've found depicting how to obtain query results lead me to believe 
_msi.View.Fetch() will return None if no more rows exist.  My development using 
python 2.7.11, Windows 7 enterprise and 2.7.13, Windows Server 2016, has shown 
Fetch() will raise a msilib.MSIError when no more rows are available.  MSIError 
is commonly used within msilib which means explicit error handling (or leniency 
in this case) may incorrectly suppress other error scenarios.

Entering a bug with the hope either a patch addresses this issue or 
documentation can be updated to reflect proper usage.  Example provided below.


>>> import msilib
>>> db = msilib.OpenDatabase("msi-1.0.0.8.msi", msilib.MSIDBOPEN_READONLY)
>>> v = db.OpenView("select * from Property")
>>> v.Execute(None)
>>> while 1:
... r = v.Fetch()
... if not r:break
... print r.GetString(1)
...
UpgradeCode
WixSharp_UI_INSTALLDIR
WixSharp_InstallDialogs
WIXUI_INSTALLDIR
Manufacturer
ProductCode
ProductLanguage
ProductName
ProductVersion
SecureCustomProperties
Traceback (most recent call last):
  File "", line 2, in 
_msi.MSIError: unknown error 103

--
components: Library (Lib)
messages: 286204
nosy: Jason Matthew
priority: normal
severity: normal
status: open
title: msilib Fetch raises MSIError rather than return None
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29358] Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster object instanciation

[Yury Selivanov]

Yury Selivanov added the comment:

Great. Can you share the Richards benchmark results?  It specifically stresses 
class instantiation, so we should be able to see the improvement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29363] allow Python code to determine class help text

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This line is came from the signature of the __call__ method.

>>> import enum, inspect
>>> class A(enum.Enum):
...x = 1
... 
>>> inspect.signature(A)

>>> inspect.signature(A.__call__)

>>> inspect.signature(enum.EnumMeta.__call__)


But calling A with optional arguments doesn't work.

>>> A(1)

>>> A('B', {'y': 2})
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/enum.py", line 293, in __call__
return cls._create_(value, names, module=module, qualname=qualname, 
type=type, start=start)
  File "/home/serhiy/py/cpython/Lib/enum.py", line 378, in _create_
_, first_enum = cls._get_mixins_(bases)
  File "/home/serhiy/py/cpython/Lib/enum.py", line 436, in _get_mixins_
raise TypeError("Cannot extend enumerations")
TypeError: Cannot extend enumerations

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29282] Fused multiply-add: proposal to add math.fma()

[Eric Fahlgren]

Changes by Eric Fahlgren :


--
nosy: +eric.fahlgren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29282] Fused multiply-add: proposal to add math.fma()

[Mark Dickinson]

Mark Dickinson added the comment:

The patch needs tests for the case where a*b overflows and c is infinite 
(either of the same sign as a*b or not). This combination should never return 
NaN, but a poor emulation of fma might do so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29363] allow Python code to determine class help text

[Ethan Furman]

Ethan Furman added the comment:

That that is very unhelpful help text.  :(

Better would be:

Color(value)

So how do we allow Python code to determine the help text?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29363] allow Python code to determine class help text

[Ethan Furman]

New submission from Ethan Furman:

>From issue29338, msg286139:
--
It is easy to fix the [pydoc] test by adding missed lines. But I'm not sure 
that output the (correct) signature of enum classes makes the help better.

Color(value, names=None, *, module=None, qualname=None, type=None, start=1)

Ethan, what are your thoughts?

--
components: Argument Clinic
messages: 286199
nosy: ethan.furman, larry, ncoghlan, rhettinger, serhiy.storchaka, xiang.zhang, 
yselivanov
priority: normal
severity: normal
status: open
title: allow Python code to determine class help text
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Eric N. Vander Weele]

Eric N. Vander Weele added the comment:

> Hmm, the patch seems to be incomplete, as it just removes all the AIX support 
> scripts and mentions, without adding anything new to accommodate for the 
> removal.

The new changes to accommodate for the script removal are in 
https://github.com/ericvw/cpython/commit/e889f5fd04c1b73ef06e9f6e60108b2a7718b7d6#diff-e2d5a00791bce9a01f99bc6fd613a39dR9159
 and 
https://github.com/ericvw/cpython/commit/e889f5fd04c1b73ef06e9f6e60108b2a7718b7d6#diff-67e997bcfdac55191033d57a16d1408aR2403.

> Please note that building with both xlc_r and gcc needs to be supported.

I'll need to double check gcc, I have it working with xlc_r already.

> Please open a new issue for this to discuss this change there.

Absolutely, I'll clean up the patch and provide more details for the changes in 
another issue.  Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 24.01.2017 16:47, Eric N. Vander Weele wrote:
> 
> I have a (large) patch that completely eliminates the need for ld_so_aix and 
> makeexp_aix.  I've applied and been using this with for Python 2.7 and 3.5+, 
> but I still need to go back and validate the tests to ensure everything 
> passes as expected.
> 
> It's still a work in progress, but the patch can be found at 
> https://github.com/ericvw/cpython/commit/e889f5fd04c1b73ef06e9f6e60108b2a7718b7d6.

Hmm, the patch seems to be incomplete, as it just removes
all the AIX support scripts and mentions, without adding
anything new to accommodate for the removal.

Please note that building with both xlc_r and gcc needs to
be supported.

> If this is out of context for this particular issue, I'm happy to raise a new 
> issue to have a focus discussion about potentially landing this.  Ignore 
> changes for README.AIX since I am addressing that in issue28845.

Please open a new issue for this to discuss this change there.

> I believe if we can eliminate the wrapper scripts, it will simplify the build 
> and linking for AIX for the interpreter and C-extension modules.

Sure, if that's possible. The scripts are rather old and AIX
has moved on since those days.

FWIW: They still do work without any problems, provided the
Makefile and distutils can find them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29328] struct module should support variable-length strings

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
nosy: +twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Eric N. Vander Weele]

Eric N. Vander Weele added the comment:

I have a (large) patch that completely eliminates the need for ld_so_aix and 
makeexp_aix.  I've applied and been using this with for Python 2.7 and 3.5+, 
but I still need to go back and validate the tests to ensure everything passes 
as expected.

It's still a work in progress, but the patch can be found at 
https://github.com/ericvw/cpython/commit/e889f5fd04c1b73ef06e9f6e60108b2a7718b7d6.

If this is out of context for this particular issue, I'm happy to raise a new 
issue to have a focus discussion about potentially landing this.  Ignore 
changes for README.AIX since I am addressing that in issue28845.

I believe if we can eliminate the wrapper scripts, it will simplify the build 
and linking for AIX for the interpreter and C-extension modules.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29362] regrtest: don't fail immediately if a child does crash

[STINNER Victor]

New submission from STINNER Victor:

When regrtest is used with -jN, regrtest spawn child processes to run tests. If 
a child crashs, regrtest master process immedialtely fails. It would prefer to 
be able to continue to run following tests. It's not because a single test 
crashed that all tests would crash.

Example of output when a crash occurs:
-
(...)
0:02:45 [118/404/3] test_bytes crashed -- running: test_multiprocessing_spawn 
(67 sec), test_io (54 sec), test_subprocess (76 sec)
python: Objects/abstract.c:2186: _Py_CheckFunctionResult: Assertion `(callable 
!= NULL) ^ (where != NULL)' failed.
Fatal Python error: Aborted

Current thread 0x7f9210f11440 (most recent call first):
  File "/home/haypo/prog/python/git_cpython/Lib/unittest/case.py", line 177 in 
handle
  File "/home/haypo/prog/python/git_cpython/Lib/unittest/case.py", line 728 in 
assertRaises
  File "/home/haypo/prog/python/git_cpython/Lib/test/test_bytes.py", line 636 
in test_maketrans
  (...)
  File "/home/haypo/prog/python/git_cpython/Lib/runpy.py", line 85 in _run_code
  File "/home/haypo/prog/python/git_cpython/Lib/runpy.py", line 193 in 
_run_module_as_main

Traceback (most recent call last):
  File "/home/haypo/prog/python/git_cpython/Lib/runpy.py", line 193, in 
_run_module_as_main
"__main__", mod_spec)
  (...)
  File "/home/haypo/prog/python/git_cpython/Lib/test/libregrtest/main.py", line 
411, in run_tests
run_tests_multiprocess(self)
  File 
"/home/haypo/prog/python/git_cpython/Lib/test/libregrtest/runtest_mp.py", line 
221, in run_tests_multiprocess
raise Exception(msg)
Exception: Child error on test_bytes: Exit code -6
-

--
components: Tests
messages: 286194
nosy: haypo
priority: normal
severity: normal
status: open
title: regrtest: don't fail immediately if a child does crash
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29362] regrtest: don't fail immediately if a child does crash

[STINNER Victor]

STINNER Victor added the comment:

At least, I would like to see which tests were executed, which tests failed, 
total duration, etc. Get a summary!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Michael Felt]

Michael Felt added the comment:

OK - so details on the failing tests - for the record only.

However, I would appreciate some 'expert' feedback on what is expected to be 
happening in this test:

==
FAIL: test_sysconfig_compiler_vars 
(distutils.tests.test_sysconfig.SysconfigTestCase)
--
Traceback (most recent call last):
  File "/data/prj/python/Python-2.7.13/Lib/distutils/tests/test_sysconfig.py", 
line 102, in test_sysconfig_compiler_vars
self.assertEqual(global_sysconfig.get_config_var('LDSHARED'), 
sysconfig.get_config_var('LDSHARED'))
AssertionError: 'Modules/ld_so_aix xlc_r -bI:Modules/python.exp' != 
'/opt/lib/python2.7/config/ld_so_aix xlc_r 
-bI:/opt/lib/python2.7/config/python.exp'

--

BULK details...

==
ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
--
Traceback (most recent call last):
  File "/data/prj/python/Python-2.7.13/Lib/distutils/tests/test_install.py", 
line 216, in test_record_extensions
cmd.run()
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/install.py", line 
563, in run
self.run_command('build')
  File "/data/prj/python/Python-2.7.13/Lib/distutils/cmd.py", line 326, in 
run_command
self.distribution.run_command(command)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/dist.py", line 972, in 
run_command
cmd_obj.run()
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build.py", line 
127, in run
self.run_command(cmd_name)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/cmd.py", line 326, in 
run_command
self.distribution.run_command(command)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/dist.py", line 972, in 
run_command
cmd_obj.run()
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build_ext.py", 
line 340, in run
self.build_extensions()
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build_ext.py", 
line 449, in build_extensions
self.build_extension(ext)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build_ext.py", 
line 531, in build_extension
target_lang=language)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/ccompiler.py", line 691, 
in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/unixccompiler.py", line 
202, in link
raise LinkError, msg
LinkError: command '/opt/lib/python2.7/config/ld_so_aix' failed with exit 
status 1

==
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
--
Traceback (most recent call last):
  File "/data/prj/python/Python-2.7.13/Lib/distutils/tests/test_config_cmd.py", 
line 46, in test_search_cpp
match = cmd.search_cpp(pattern='xxx', body='/* xxx */')
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/config.py", line 
206, in search_cpp
src, out = self._preprocess(body, headers, include_dirs, lang)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/config.py", line 
129, in _preprocess
self.compiler.preprocess(src, out, include_dirs=include_dirs)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/unixccompiler.py", line 
113, in preprocess
raise CompileError, msg
CompileError: command 'xlc_r' failed with exit status 40

==
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
--
Traceback (most recent call last):
  File "/data/prj/python/Python-2.7.13/Lib/distutils/tests/test_build_ext.py", 
line 63, in test_build_ext
cmd.run()
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build_ext.py", 
line 340, in run
self.build_extensions()
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build_ext.py", 
line 449, in build_extensions
self.build_extension(ext)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/command/build_ext.py", 
line 531, in build_extension
target_lang=language)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/ccompiler.py", line 691, 
in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
  File "/data/prj/python/Python-2.7.13/Lib/distutils/unixccompiler.py", line 
202, in link
raise LinkError, msg
LinkError: command '/opt/lib/python2.7/config/ld_so_aix' failed with exit 
status 1

==
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)

[issue18235] _sysconfigdata.py wrong on AIX installations

[Michael Felt]

Michael Felt added the comment:

I am back at this: (note: short answer to msg277745 - No. does not work for me)

As ... expressed before, this needs to work in three contexts:

2) In-tree testing of build module feature (test_distutils).

Below are results of 'in tree' and 'out of tree' testing. in/out seems to have 
no (more) impact.

msg219888 also has a patch - that moves the code back to where it originally 
was: _init_posix

But - same issue - ld_so_aix is not found. (notice the "src" in string of 
'threading.py')
[112/401/2] test_distutils
xlc_r: 1501-218 (S) file _configtest.i contains an incorrect file suffix
unable to execute '/opt/lib/python2.7/config/ld_so_aix': No such file or 
directory
Warning -- os.environ was modified by test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for 
details
[113/401/3] test_dl
test_dl skipped -- No module named dl
[114/401/3] test_docxmlrpc
/data/prj/python/src/Python-2.7.13/Lib/threading.py:846: DeprecationWarning: 
sys.exc_clear() not supported in 3.x; use except clauses
  self.__exc_clear()
[115/401/3] test_dumbdbm
root@x064:[/data/prj/python/Python-2.7.13]


Now, back to 'in tree' with patch from msg219888 (no "src" in path)

root@x064:[/data/prj/python/Python-2.7.13]find . -name _sysconfigdata\*.py -ls 
-exec grep LDSHARED {} \;
48056876   16 -rw-r-  1 root  100015362 Jan 24 15:07 
./build/lib.aix-5.3-2.7/_sysconfigdata.py
 'BLDSHARED': 'Modules/ld_so_aix xlc_r -bI:Modules/python.exp',
 'LDSHARED': '/opt/lib/python2.7/config/ld_so_aix xlc_r 
-bI:/opt/lib/python2.7/config/python.exp',
[112/401/2] test_distutils
xlc_r: 1501-218 (S) file _configtest.i contains an incorrect file suffix
unable to execute '/opt/lib/python2.7/config/ld_so_aix': No such file or 
directory
Warning -- os.environ was modified by test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for 
details
[113/401/3] test_dl
test_dl skipped -- No module named dl
[114/401/3] test_docxmlrpc
/data/prj/python/Python-2.7.13/Lib/threading.py:846: DeprecationWarning: 
sys.exc_clear() not supported in 3.x; use except clauses
  self.__exc_clear()

Conclusion:
** As the code is - that is removed means the environment variable in 
sysconfigdata.py is correct,
   I would suggest the patch is to remove the 6 lines there.
   Further, the addition of the code below (per msg219888) has no effect, 
including it is not needed.

The .dist file is from the source tarball from last december for 2.7.13

michael@x071:[/data/prj/python/src]diff -u  Python-2.7.13/Lib/sysconfig.py.dist>
--- Python-2.7.13/Lib/sysconfig.py.dist 2016-12-17 20:05:06 +
+++ Python-2.7.13/Lib/sysconfig.py  2017-01-24 14:56:20 +
@@ -307,12 +307,6 @@
 msg = msg + " (%s)" % e.strerror
 raise IOError(msg)

-# On AIX, there are wrong paths to the linker scripts in the Makefile
-# -- these paths are relative to the Python source, but when installed
-# the scripts are in another directory.
-if _PYTHON_BUILD:
-vars['LDSHARED'] = vars['BLDSHARED']
-
 # There's a chicken-and-egg situation on OS X with regards to the
 # _sysconfigdata module after the changes introduced by #15298:
 # get_config_vars() is called by get_platform() as part of the
@@ -355,6 +349,11 @@
 # _sysconfigdata is generated at build time, see _generate_posix_vars()
 from _sysconfigdata import build_time_vars
 vars.update(build_time_vars)
+# On AIX, there are wrong paths to the linker scripts in the Makefile
+# -- these paths are relative to the Python source, but when installed
+# the scripts are in another directory.
+if _PYTHON_BUILD:
+vars['LDSHARED'] = vars['BLDSHARED']

 def _init_non_posix(vars):
 """Initialize the module as appropriate for NT"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[STINNER Victor]

Changes by STINNER Victor :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 24.01.2017 15:23, Michael Felt wrote:
> As far as issue10656 and this issue are concerned:
> 
> Python-3.4 is out of context (but 3.4.6 was just released) - and does not 
> work with 'out of tree' builds. 
> 
> The other versions: 2.7.13, 3.5.3 and 3.6.0 do build out-of-tree.
> Note 3.5.3 and 3.6.0 use a different name for sysconfigdata.py

I'm not sure what you mean with "out of tree". When building
Python, you are supposed to use the source code directory
as work dir.

> Notes:
> 
> FYI: regarding build and src in different directories
> 
> Not working...
> 
> unable to execute '../src/Python-3.4.6/Modules/ld_so_aix': No such file or 
> directory
> unable to execute '../src/Python-3.4.6/Modules/ld_so_aix': No such file or 
> directory
>  many many times
> 
> Concludes with:
> ImportError: No module named 'time'
> make: 1254-004 The error code from the last command is 1.
> make: 1254-005 Ignored error code 1 from last command.
> ../src/Python-3.4.6/install-sh: ../src/Python-3.4.6/Modules/ld_so_aix does 
> not exist
> make: 1254-004 The error code from the last command is 1.
> 
> root@x064:[/data/prj/python/Python-3.4.6]find . -name _sysconfigdata.py -ls 
> -exec grep LDSHARED {} \;
> 48049316   19 -rw-r-  1 root  felt 19209 Jan 24 13:55 
> ./build/lib.aix-5.3-3.4/_sysconfigdata.py
>  'BLDSHARED': '/opt/lib/python3.4/config/ld_so_aix xlc_r '
>  'LDSHARED': '/opt/lib/python3.4/config/ld_so_aix xlc_r '

This is strange: where does the "../src/Python-3.4.6/Modules/ld_so_aix"
originate if the paths in the sysconfig file are absolute ?

> +++ Working 
> 
> Working: 2.7.13 -- Note: the 'lines' in sysconfig.py are removed:
> root@x064:[/data/prj/python/Python-2.7.13.0]find . -name _sysconfigdata\*.py 
> -ls -exec grep LDSHARED {} \;
> 48038426   16 -rw-r-  1 root  felt 15807 Jan 24 13:36 
> ./build/lib.aix-5.3-2.7/_sysconfigdata.py
>  'BLDSHARED': 'Modules/ld_so_aix xlc_r -bI:Modules/python.exp',
>  'LDSHARED': '/opt/lib/python2.7/config/ld_so_aix xlc_r 
> -bI:/opt/lib/python2.7/config/python.exp',

BLDSHARED should be fixed to use an absolute path, I guess.

> Working: 3.5.3
> 48049196   21 -rw-r-  1 root  felt 20640 Jan 24 13:55 
> ./build/lib.aix-5.3-3.5/_sysconfigdata.py
>  'BLDSHARED': '/opt/lib/python3.5/config-3.5m/ld_so_aix xlc_r '
>  'LDSHARED': '/opt/lib/python3.5/config-3.5m/ld_so_aix xlc_r '

Here BLDSHARED is wrong, but shouldn't do any harm since
it's only used for building Python itself, not for
extension modules.

> Working: 3.6.0 -- Note new file name!!
> root@x064:[/data/prj/python/Python-3.6.0]find . -name _sysconfigdata\*.py -ls 
> -exec grep LDSHARED {} \;
> 47794146   20 -rw-r-  1 root  felt 20394 Jan 24 13:49 
> ./build/lib.aix-5.3-3.6/_sysconfigdata_m_aix5_.py
>  'BLDSHARED': 'Modules/ld_so_aix xlc_r -bI:Modules/python.exp',
>  'LDSHARED': '/opt/lib/python3.6/config-3.6m/ld_so_aix xlc_r '

This looks odd and like a regression. Again, the path
should be absolute and point to the build tree, not the
installation tree.

PS: The BLDSHARED setting in sysconfig is not really relevant.
It's only there because we have it in the Makefile.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19500] Add client-side SSL session resumption

[Rob Reilink]

Rob Reilink added the comment:

With this code in place, ftplib should / could also be updated to support 
session resumption. This would fix bugs with connections to FTP servers that 
require session resumption [1], [2]

In ftplib.FTP_TLS.ntransfercmd, just add a reference to the current session in 
the wrap_socket call (maybe make this an option to do session resumption or 
not; I don't know if it could break something)

Proposed patch is attached.

[1] http://stackoverflow.com/questions/14659154/ftpes-session-reuse-required
[2] https://forum.filezilla-project.org/viewtopic.php?t=36903

--
nosy: +robr
Added file: http://bugs.python.org/file46409/ftplib_session.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Michael Felt]

Michael Felt added the comment:

As far as issue10656 and this issue are concerned:

Python-3.4 is out of context (but 3.4.6 was just released) - and does not work 
with 'out of tree' builds. 

The other versions: 2.7.13, 3.5.3 and 3.6.0 do build out-of-tree.
Note 3.5.3 and 3.6.0 use a different name for sysconfigdata.py

Notes:

FYI: regarding build and src in different directories

Not working...

unable to execute '../src/Python-3.4.6/Modules/ld_so_aix': No such file or 
directory
unable to execute '../src/Python-3.4.6/Modules/ld_so_aix': No such file or 
directory
 many many times

Concludes with:
ImportError: No module named 'time'
make: 1254-004 The error code from the last command is 1.
make: 1254-005 Ignored error code 1 from last command.
../src/Python-3.4.6/install-sh: ../src/Python-3.4.6/Modules/ld_so_aix does not 
exist
make: 1254-004 The error code from the last command is 1.

root@x064:[/data/prj/python/Python-3.4.6]find . -name _sysconfigdata.py -ls 
-exec grep LDSHARED {} \;
48049316   19 -rw-r-  1 root  felt 19209 Jan 24 13:55 
./build/lib.aix-5.3-3.4/_sysconfigdata.py
 'BLDSHARED': '/opt/lib/python3.4/config/ld_so_aix xlc_r '
 'LDSHARED': '/opt/lib/python3.4/config/ld_so_aix xlc_r '

+++ Working 

Working: 2.7.13 -- Note: the 'lines' in sysconfig.py are removed:
root@x064:[/data/prj/python/Python-2.7.13.0]find . -name _sysconfigdata\*.py 
-ls -exec grep LDSHARED {} \;
48038426   16 -rw-r-  1 root  felt 15807 Jan 24 13:36 
./build/lib.aix-5.3-2.7/_sysconfigdata.py
 'BLDSHARED': 'Modules/ld_so_aix xlc_r -bI:Modules/python.exp',
 'LDSHARED': '/opt/lib/python2.7/config/ld_so_aix xlc_r 
-bI:/opt/lib/python2.7/config/python.exp',

Working: 3.5.3
48049196   21 -rw-r-  1 root  felt 20640 Jan 24 13:55 
./build/lib.aix-5.3-3.5/_sysconfigdata.py
 'BLDSHARED': '/opt/lib/python3.5/config-3.5m/ld_so_aix xlc_r '
 'LDSHARED': '/opt/lib/python3.5/config-3.5m/ld_so_aix xlc_r '

Working: 3.6.0 -- Note new file name!!
root@x064:[/data/prj/python/Python-3.6.0]find . -name _sysconfigdata\*.py -ls 
-exec grep LDSHARED {} \;
47794146   20 -rw-r-  1 root  felt 20394 Jan 24 13:49 
./build/lib.aix-5.3-3.6/_sysconfigdata_m_aix5_.py
 'BLDSHARED': 'Modules/ld_so_aix xlc_r -bI:Modules/python.exp',
 'LDSHARED': '/opt/lib/python3.6/config-3.6m/ld_so_aix xlc_r '

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29308] venv should match virtualenv VIRTUAL_ENV_DISABLE_PROMPT config

[Jack Bennett]

Jack Bennett added the comment:

That works fine here.

Thank you

On 24 January 2017 at 13:15, Vinay Sajip  wrote:

>
> Vinay Sajip added the comment:
>
> Other scripts (activate, activate.csh, activate.csh) support this
> environment variable already. This patch updates Activate.ps1 to respect
> the variable - can you confirm if this change works for you?
>
> --
> assignee:  -> vinay.sajip
> keywords: +patch
> stage:  -> patch review
> Added file: http://bugs.python.org/file46407/fix-29308-01.diff
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[STINNER Victor]

STINNER Victor added the comment:

Thanks Serhiy and Naoki. I really wanted more eyes on this weird issue :-)

It's also a reminder that we should be careful when converting keyword 
arguments, especially when duplicates are not allowed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2d2210b36b25 by Victor Stinner in branch 'default':
Issue #29360: _PyStack_AsDict() doesn't check kwnames
https://hg.python.org/cpython/rev/2d2210b36b25

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

pystack_asdict-2.patch LGTM.

--
assignee:  -> haypo
stage:  -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Michael Felt]

Michael Felt added the comment:

re: msg286115, 

1) Building modules in the tree during the build process.
** if "in the tree" means building outside of source - such as:
../src/python-X.Y.Z/configure ...
make

** the tarball for Python-2.7.13 is building from 
../src/Python-2.7.13/configure where as ../src/Python-2.7.12/configure and the 
earlier (ones I tried) were not.

** However, 'make distclean' is not working. 
   Further details in a later issue - this bit about make distclean id merely 
FYI. 

2) In-tree testing of build module feature (test_distutils).

** Is this as simple as (from the 'build directory':
make test 

** shall check this later, make test in general seems to be working - is 
test_distutils a specific test you want verified?
3) Building and installing modules with an installed version of Python.
** FYI: this will work with the 5 lines deleted - as I mentioned above.
   This is how I manually patched _sysconfigdata.py in Python-2.7.12 when I ran 
into this.

Now reading the other comments from last night...

re: 286172 and 286173

Short response:
it looks as if patches elsewhere in the 'auto' config files has taken care of 
this issue - at least for Python-2.7

I shall download and check the latest Python-3.{4|5|6} and report back.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[STINNER Victor]

STINNER Victor added the comment:

> I don't know. To that that is crashed when pass non-string keys.

Oh ok. I found a way to create a dictionary with equal keys and pass it to 
dict(**kw) in my fast_init branch (I already removed the PyUnicode_Check() 
asssertion in this branch):

haypo@selma$ ./python bug.py 
call with: {'a': 1, 'b': 2}
python: Objects/abstract.c:2471: _PyStack_AsDict: Assertion 
`PyDict_GetItem(kwdict, key) == NULL' failed.
Aborted


So yes, with special objects, the PyDict_GetItem()==NULL assertion fails. With 
pystack_asdict-2.patch, the duplicated (equal) keys are merged as expected:

$ ./python bug.py 
call with: {'a': 1, 'b': 2}
got: {'a': 2}

Again, there is also a bug in my fast_init branch, I should avoid double 
conversion dict=>kwnames=>dict to call a type.

--
Added file: http://bugs.python.org/file46408/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29308] venv should match virtualenv VIRTUAL_ENV_DISABLE_PROMPT config

[Vinay Sajip]

Vinay Sajip added the comment:

Other scripts (activate, activate.csh, activate.csh) support this environment 
variable already. This patch updates Activate.ps1 to respect the variable - can 
you confirm if this change works for you?

--
assignee:  -> vinay.sajip
keywords: +patch
stage:  -> patch review
Added file: http://bugs.python.org/file46407/fix-29308-01.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Martin Panter]

Martin Panter added the comment:

Sorry I meant Issue 10656. I recently committed a fix regarding out-of-tree 
builds and ld_so_aix, and it sounds like you were thinking of reverting that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29349] Update old Python 2 code in Docs/tools/extensions/patchlevel.py

[Jim Fasarakis-Hilliard]

Changes by Jim Fasarakis-Hilliard :


Added file: http://bugs.python.org/file46406/patchlevel_with.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I don't know. To that that is crashed when pass non-string keys.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Martin Panter]

Martin Panter added the comment:

I don’t run AIX, but my understanding is there are three distinct branches 
(2.7, 3.5, and 3.6+). Some of the following is guessed, so please correct me if 
I am wrong:

Python 2.7:
_sysconfigdata.py incorrectly created with LDSHARED = Modules/ld_so_aix.
Result: Installed distutils fails (unless you’re in the build tree).
Fix: Apply David’s issue18235.patch + Michael H’s msg219888 fix.

Python 3.5:
David’s issue18235.patch already applied.
_sysconfigdata.py incorrectly created with BLDSHARED = $(LIBPL)/ld_so_aix, i.e. 
the installed location.
Distutils does not refer to _sysconfigdata.py.
Result: Distutils unaffected, but running from the build tree, 
sysconfig.get_config_var("LDSHARED") would incorrectly use the installed 
location.
Fix: Apply Michael H’s msg219888.

Python 3.6+:
Distutils does use _sysconfigdata*.py, thanks to revision 3fa8aebed636.
Result: Distutils shouldn’t work unless Python is already installed. Since the 
3.6 and 3.x buildbots pass test_distutils, does that mean that both those 
versions of Python are already installed?
Fix: Should also be fixed by merging msg219888 from 3.5.

Marc-Andre, regarding configure.ac and absolute vs relative paths, I suggest 
you open a separate bug. That is an orthogonal issue, and this bug is already 
too complicated. But also keep in mind Issue 18235. You want the build tree, 
not the source tree.

Also, if you mean to drop the assignment in sysconfig, I think that will break 
in-tree usage of sysconfig (3.5+) and distutils (3.6+), unless Python happens 
to also be installed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29349] Update old Python 2 code in Docs/tools/extensions/patchlevel.py

[Jim Fasarakis-Hilliard]

Jim Fasarakis-Hilliard added the comment:

I'm breaking these to separate files to make it easier to apply. I also noticed 
that other files in `Doc/tools/extensions/` use old constructs so I'm not sure 
about the *with*.

I'm guessing that either it should be changed in other files too or, since it's 
working fine, leave it as is.

--
Added file: http://bugs.python.org/file46405/patchlevel_print.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "Try following keys: (...)"

To which function am I supposed to pass such keys? I was unable to find a 
simple way how to pass arbitrary keys to _PyStack_AsDict().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Try following keys:

class Key:
def __eq__(self, other):
return self is other or random.getrandbits(1)
def __hash__(self):
return random.getrandbits(1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[STINNER Victor]

STINNER Victor added the comment:

Attached pystack_asdict-2.patch removes the two assertions and rewrites 
_PyStack_AsDict() documentation to describe better how it handles non-string 
and duplicated keys (don't check keys type and merge duplicated keys).


> We should either ensure that _PyStack_AsDict() is called only by 
> CALL_FUNCTION_KW, and not by other way, or remove both assertions. I prefer 
> the second option. They were here only for self-testing.

CALL_FUNTION_EX doesn't use _PyStack_AsDict() (at least, currently, I should 
carefully review my tp_fast{new,init,call} patches).

When I wrote _PyStack_AsDict(), I added the GetItem==NULL assertion because I 
didn't know how to handle this corner case.

For best performances, I'm in favor of not calling GetItem in _PyStack_AsDict() 
and so merge duplicated keys. But before removing the assertion, I wanted to 
make sure that it doesn't break the Python semantics.

The problem is that I'm unable to find any concrete tcase in Python 3.6 nor 
Python 3.7 where it would be possible to call _PyStack_AsDict() with duplicate 
keys. See my analysis below.

I agree to remove both assertion. I just checked, _PyStack_AsDict() already 
explicitly explain that keys must be strings and must be unique, but are not 
checked. My patch now also mentions that duplicated keys are merged.

--

(1) When a Python is called by _PyFunction_FastCallDict(), a TypeError is 
raised if an argument is passed as a positional argument and a keyword 
argument. Example: "def func(x): pass; func(1, x=2)".

(2) If for some reasons, two keys of keyword arguments are equal (same hash 
value and are equals) but go into "**kwargs", the last value is kept: 
_PyEval_EvalCodeWithName() calls PyDict_SetItem() which replaces existing key 
to the new value. I failed to find an example here.

(3) Calling other functions using kwargs dictionary from a FASTCALL function 
converts stack+kwnames arguments to kwargs dictionary using _PyStack_AsDict(). 
_PyCFunction_FastCallKeywords() implements that for the 
METH_VARARGS|METH_KEYWORDS calling convention. For example, round(3.14, 
ndigits=1) calls _PyCFunction_FastCallKeywords() with kwnames=('ndigits',) and 
args=[3.14, 1], _PyStack_AsDict() creates {'ndigits': 1} dictionary.


The tricky question is how (3) handles duplicated keys in dictionary.


"round(3.14, ndigits=1)" uses CALL_FUNCTION_KW which creates 
kwnames=('ndigits,') constant tuple. The compiler raises a SyntaxError if a 
function is called with a duplicate key. Without modifying a code object, I 
don't see any obvious way to pass duplicate keys in kwnames to CALL_FUNCTION_KW.

CALL_FUNCTION_EX expects a dictionary on the stack. For 
METH_VARARGS|METH_KEYWORDS C functions, the dictionary is passed through, there 
is not conversion.

If CALL_FUNCTION_EX gets a mapping which is not exactly the dict type, 
PyDict_New() + PyDict_Update() is used to convert the mapping into a regular 
dict. This function merges duplicated keys.


Without modifying code objects nor writing buggy C code using 
_PyObject_FastCallKeywords(), I'm not sure that it's possible to pass 
duplicated keys to _PyStack_AsDict(). It seems that the issue is more 
theorical, no?

--
Added file: http://bugs.python.org/file46404/pystack_asdict-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] _PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

[STINNER Victor]

Changes by STINNER Victor :


--
title: Don't check if all keys are strings in _PyStack_AsDict() -> 
_PyStack_AsDict(): Don't check if all keys are strings nor if keys are unique

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy:  -pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

BTW: Does the ticket still apply to 3.5+ ? From reading the ticket, it seems 
that the problem is already fixed for those versions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Hmm, looking at the patch again:

diff -r a089a8b1f93d Lib/sysconfig.py
--- a/Lib/sysconfig.py  Fri Jun 21 18:37:02 2013 -0400
+++ b/Lib/sysconfig.py  Fri Jun 21 22:33:15 2013 -0700
@@ -368,7 +368,7 @@
 # -- these paths are relative to the Python source, but when installed
 # the scripts are in another directory.
 if _PYTHON_BUILD:
-vars['LDSHARED'] = vars['BLDSHARED']
+vars['BLDSHARED'] = vars['LDSHARED']
 
 # There's a chicken-and-egg situation on OS X with regards to the
 # _sysconfigdata module after the changes introduced by #15298:

I think that with the configure fix, the special case for AIX can be dropped 
altogether. It just polishes over the bug in configure, turning a relative path 
into an absolute one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18235] _sysconfigdata.py wrong on AIX installations

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Reading the ticket, it seems that there is some confusion about what LDSHARED 
and BLDSHARED are used for. BLDSHARED is used to override the LDSHARED value 
when building libpython and the shared modules (via setup.py).

LDSHARED is what is meant to be used for building shared modules after 
installation.

This distinction is not being followed by all targets in the Makefile, but is 
needed in cases where special build tools are necessary, as is the case on AIX.

For the latter, BLDSHARED has to point to the source tree version of those 
build tools, while LDSHARED has to point to the installed version of these.

On AIX, BLDSHARED should therefore point to the ./Modules/ld_so_aix (or better: 
the absolute dir of the source tree), while LDSHARED needs to point to the 
installation target for ld_so_aix, e.g. 
/usr/local/lib/python2.7/config/ld_so_aix.

So the fix which was already applied to 3.4 is correct for 2.7 as well. The 
missing part is the fix to the configure script, since in Python 2.7, this 
still uses relative paths:

BLDSHARED="Modules/ld_so_aix \$(CC) -bI:Modules/python.exp"
LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) 
-bI:\$(BINLIBDEST)/config/python.exp"

instead of the ones from 3.4:

BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) 
-bI:\$(srcdir)/Modules/python.exp"
LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) 
-bI:\$(BINLIBDEST)/config/python.exp"

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29361] bug in pyserial: serialposix.py

[Berker Peksag]

Berker Peksag added the comment:

Thanks for the report, but pyserial is not part of the Python stdlib. Please 
open your bug report on https://github.com/pyserial/pyserial.

--
nosy: +berker.peksag
resolution:  -> third party
stage:  -> resolved
status: open -> closed
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> func(**{'x': 1, 'x': 2})

In this example the dictionary contains only one item. You need more complex 
example for making the assertion failing. Keys with non-constant hashes or 
equality, so first they are different, and later they become equal.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[STINNER Victor]

STINNER Victor added the comment:

I don't know what is the most efficient technic to reduce the memory usage of 
module, but at least I can say imports are the top #1 memory consumer. Look at 
tracemalloc examples:

https://docs.python.org/dev/library/tracemalloc.html#display-the-top-10

"We can see that Python loaded 4855 KiB data (bytecode and constants) from 
modules"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29361] bug in pyserial: serialposix.py

[Reto Cavelti]

New submission from Reto Cavelti:

bug in pyserial: serialposix.py

line 50:
# set custom divisor
buf[6] = buf[7] / baudrate

TypeError: integer argument expected, got float

fix: do cast: buf[6] = int(buf[7] / baudrate)

--
components: Library (Lib)
messages: 286168
nosy: rc
priority: normal
severity: normal
status: open
title: bug in pyserial: serialposix.py
type: crash
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[STINNER Victor]

STINNER Victor added the comment:

> Thanks.  Your patch reduced memory consumption by 2%,

merge-constants.patch looks simple enought, but I'm not really impressed by 
such result. Is 2% worth it?

Since code objects loaded by import are likely for stay for the whole lifetime 
of a process, I would be interested to experiment interning all constant 
objects (the tuple of objects, but also each object of these tuples, so support 
"nested interning") of all code objets (consts, names, varnames, freevars, 
cellvars) in the marshal module.

To intern constants, we need to generate an unique key to not merge 1 int and 
1.0 float. The risk of such global intern dict is to increase the memory usage 
of the unique keys uses more memory than the decreased caused by the removal of 
duplicate objects.

More generally, would it be possible to share co_consts (None,) tuples between 
code objects of two different modules? Or is it already the case with 
merge-constants.patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

We should either ensure that _PyStack_AsDict() is called only by 
CALL_FUNCTION_KW, and not by other way, or remove both assertions. I prefer the 
second option. They were here only for self-testing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[STINNER Victor]

STINNER Victor added the comment:

> But this can be done better with the AST optimizer (issue1346238, issue11549).

There are also the PEP 511 (Code transformers) which was created to implement 
*external* AST optimizers, and the FAT Python which implements an AST optimizer.

Hum, after "-o no_annotation" discussed on the mailing list, maybe it's time to 
look again at this PEP? ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[STINNER Victor]

STINNER Victor added the comment:

Victor> PyPy (and other Python implementations?) don't support non-string in 
type dictionary.
INADA Naoki> Wow! I thought it's Python's language. If we can prohibit non 
string name in all namespace, there might be possible optimization.

That's a different topic :-) It would be a backward incompatible change. 
Well... I don't think that anyone relies on this strange feature :-) IMHO it's 
more a side effect of the implementation, not a deliberate feature.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "The assertion was valid while all keywords did came from a 
constant keywords tuple in CALL_FUNCTION_KW. But if the FASTCALL protocol is 
extended for var-keyword arguments, keywords can be arbitrary and the assertion 
can fail. The assertion on the next line can fail too."

The second _PyStack_AsDict() assertion is:

   assert(PyDict_GetItem(kwdict, key) == NULL);

This assertion fails if kwnames (tuple) argument of _PyStack_AsDict() contains 
duplicated keys, or at least two keys which have the same hash value and are 
equal.

If someone pass kwnames with duplicates keys on purpose (for whatever) reasons, 
the assertion fails. I'm not sure that the assertion makes sense since it's a 
feature of Python to be able to replace dictionary values:

>>> def func(**kw): print(kw)
... 
>>> func(**{'x': 1, 'x': 2})
{'x': 2}
>>> {'x': 1, 'x': 2}
{'x': 2}

Are you suggesting to remove "assert(PyDict_GetItem(kwdict, key) == NULL);" too?


Note: In Python 3.6, CALL_FUNCTION_EX already uses FASTCALL for C functions 
using METH_FASTCALL. In this case, _PyCFunction_FastCallDict() uses 
_PyStack_UnpackDict(). For example, open() already uses METH_FASTCALL in Python 
3.6. In there an issue with _PyStack_UnpackDict()? (I don't think so.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "The assertion was valid while all keywords did came from a 
constant keywords tuple in CALL_FUNCTION_KW. But if the FASTCALL protocol is 
extended for var-keyword arguments, keywords can be arbitrary and the assertion 
can fail. The assertion on the next line can fail too."

Oh right, I didn't notice that. Since it's an issue of the 
tp_fastnew/tp_fastinit patch, I replied there:
http://bugs.python.org/issue29358#msg286161

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29358] Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster object instanciation

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "Calling _PyStack_AsDict() with non-string or non-unique 
keywords means that the var-keyword argument was first unpacked to arrays of 
keyword names and values and then converted back to a dict. Seems something is 
done non-efficiently." (msg286159 of the issue #29360)

Python code from test_dict:

   dict(**invalid)

Bytecode:

LOAD_GLOBAL  1 (dict)
BUILD_TUPLE  0
LOAD_FAST1 (invalid)
CALL_FUNCTION_EX 1

Call stack:

* _PyEval_EvalFrameDefault()
* do_call_core()
* PyObject_Call()
* _Py_RawFastCallDict() -- conversion from dict to stack+kwnames
* type_call()
* call_init()
* _PyStack_AsDict() -- convertsion from stack+kwnames to dict

Oh right, there are two conversions using a temporary FASTCALL format for 
(keyword) arguments.

This code was not upstream yet, it comes from the pull request of this issue.

Maybe we need two flavors of type_call(): type_call(args: tuple, kwargs: tuple) 
if tp_fastinit isn't set, type_fastcall(stack, nargs, kwnames) (FASTCALL) if 
tp_fastinit is set.

But it seems that the logic should be implemented in PyObject_Call() and 
_PyObject_FastCallDict(), it cannot be implemented in type_call(), it's too 
late.

For best performances (avoid any kind of conversion and avoid any temporary 
object), we should implement something like that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

merge-constants.patch is rather a proof of concept. I think it may be more 
efficient after removing unneeded folded constants (issue28813). But this can 
be done better with the AST optimizer (issue1346238, issue11549). It would be 
worth also to merge nested constants. This is similar to interning string 
constants (see in Objects/codeobject.c).

The benefit is small and we should check that merging constants doesn't have 
too large cost (CPU time or temporally consumed memory) at compile time.

I think we should first implement the AST optimizer, and then try to combine 
merging constants with interning string constants.

--
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The assertion was valid while all keywords did came from a constant keywords 
tuple in CALL_FUNCTION_KW. But if the FASTCALL protocol is extended for 
var-keyword arguments, keywords can be arbitrary and the assertion can fail. 
The assertion on the next line can fail too.

Calling _PyStack_AsDict() with non-string or non-unique keywords means that the 
var-keyword argument was first unpacked to arrays of keyword names and values 
and then converted back to a dict. Seems something is done non-efficiently.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[INADA Naoki]

INADA Naoki added the comment:

patch LGTM.

Python 2 supported `dict(**{1:3})`.  But I don't know there are any functions
supporting non-string keyword argument.

> PyPy (and other Python implementations?) don't support non-string in type 
> dictionary.

Wow! I thought it's Python's language.
If we can prohibit non string name in all namespace, there might be possible
optimization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[STINNER Victor]

New submission from STINNER Victor:

While running the test suite on issues #29259 (tp_fastcall) and #29358 
(tp_fastnew and tp_fastinit), a few test failed on the following assertion of 
_PyStack_AsDict():

   assert(PyUnicode_CheckExact(key));

For example, test_dict calls dict(**{1: 2}) which must raise a TypeError. I'm 
not sure who is responsible to check if all keys are strings: 
_PyStack_AsDict(), PyArg_ParseXXX() or the final function?


The check is already implemented in dict_init(), dict_update_common() calls 
PyArg_ValidateKeywordArguments():

>>> dict(**{1:2})
Traceback (most recent call last):
  File "", line 1, in 
TypeError: keyword arguments must be strings


In Python 3.7, PyArg_ParseTupleAndKeywords(), 
_PyArg_ParseTupleAndKeywordsFast() and _PyArg_ParseStackAndKeywords() and 
PyArg_ValidateKeywordArguments() raise an error if a key of keyword argumens is 
not a string:

if (!PyUnicode_Check(key)) {
PyErr_SetString(PyExc_TypeError,
"keywords must be strings");
return cleanreturn(0, );
}


IMHO the CALL_FUNCTION_EX instruction and the FASTCALL calling convention must 
not check if all keys are string: the check must only be done in the function 
(which can be a type constructor like dict_init()), for performance. Almost all 
functions use one the PyArg_ParseXXX() function. Only very special cases like 
dict_init() use more specific code to handle keyword arguments.

By the way, _PyObject_FastCallKeywords() already contains the following comment:

/* kwnames must only contains str strings, no subclass, and all keys must
   be unique: these checks are implemented in Python/ceval.c and
   _PyArg_ParseStackAndKeywords(). */


Note: Technically, I know that it's possible to put non-string keys in a 
dictionary and in a type dictionary. PyPy (and other Python implementations?) 
don't support non-string in type dictionary. Do you know use cases where 
func(**kwargs) must accept non-string keys?


At the end, this long issue is a simple patch replacing an assertion with a 
comment in _PyStack_AsDict(): see attached patch ;-)

--
components: Interpreter Core
files: pystack_asdict.patch
keywords: patch
messages: 286157
nosy: haypo
priority: normal
severity: normal
status: open
title: Don't check if all keys are strings in _PyStack_AsDict()
versions: Python 3.7
Added file: http://bugs.python.org/file46403/pystack_asdict.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29360] Don't check if all keys are strings in _PyStack_AsDict()

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +inada.naoki, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29359] Deprecate string concatenation without plus

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also the topic "Implicit string literal concatenation considered harmful?" 
on Python-Ideas:
https://mail.python.org/pipermail/python-ideas/2013-May/020527.html

If you want to revive this issue, it is better to do this on the Python-Ideas 
mailing list.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29359] Deprecate string concatenation without plus

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See PEP 3126.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29336] merge tuples in module

[INADA Naoki]

INADA Naoki added the comment:

merge-constants.patch LGTM

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29357] New NamedTuple syntax silently ignores method definitions

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

This has been already reported in https://github.com/python/typing/issues/352 
and fixed in https://hg.python.org/cpython/rev/f100619e7137 and 
https://github.com/python/typing/pull/364

Now adding new methods works but overwriting existing special attributes raises 
AttributeError:

class A(NamedTuple):
x: int
def spam(self):  # this works
...
def _fields(self):  # this is an error (and also for __repr__ etc)

If you think that overwriting all special attributes should be allowed (or only 
some of them) then we could discuss this at python/typing tracker.

--
nosy: +levkivskyi
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29359] Deprecate string concatenation without plus

[Aivar Annamaa]

Changes by Aivar Annamaa :


--
components: +Interpreter Core

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29359] Deprecate string concatenation without plus

[Aivar Annamaa]

New submission from Aivar Annamaa:

How quickly will you notice the bug here:

for sequence in ["Command",
 "MagicCommand",
 "Open",
 "Save",
 "SaveAs",
 "NewFile",
 "EditorTextCreated"
 "ShellCommand",
 "ShellInput",
 "ShowView",
 "HideView",
 "TextInsert",
 "TextDelete",
 ]:
do_something(sequence)

? 

Given the title of the issue, its probably not so hard, but usually a list 
literal is not the first thing to suspect.

Similar problem of missing comma may occur easily with multiline calls.

If concatenation without plus was deprecated, these mistakes would be so easy 
to notice.

--
messages: 286152
nosy: Aivar.Annamaa
priority: normal
severity: normal
status: open
title: Deprecate string concatenation without plus
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29358] Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster object instanciation

[STINNER Victor]

New submission from STINNER Victor:

After #29259 "Add tp_fastcall to PyTypeObject: support FASTCALL calling 
convention for all callable objects", the two last slots which still use the 
(args: tuple, kwargs: dict) calling convention are tp_new and tp_init, two 
major slots to instanciate objects.

I implemented tp_fastnew/tp_fastinit on top of the issue #29259 pull request 
(tp_fastcall). The implementation is a WIP, it's just complete enough to start 
benchmarking the code.

Example of benchmarks on the two types currently optimized in my WIP fast_init 
branch, list and _asyncio.Future:
---
haypo@smithers$ ./python -m perf timeit -s 'List=list' 'List()' --duplicate=100 
--compare-to=../default-ref/python 
Median +- std dev: [ref] 81.9 ns +- 0.2 ns -> [fast_init] 69.3 ns +- 0.4 ns: 
1.18x faster (-15%)

haypo@smithers$ ./python -m perf timeit -s 'List=list' 'List((1,2,3))' 
--duplicate=100 --compare-to=../default-ref/python 
Median +- std dev: [ref] 137 ns +- 6 ns -> [fast_init] 107 ns +- 0 ns: 1.28x 
faster (-22%)

haypo@smithers$ ./python -m perf timeit -s 'import _asyncio, asyncio; 
Future=_asyncio.Future; loop=asyncio.get_event_loop()' 'Future(loop=loop)' 
--compare-to=../default-ref/python
Median +- std dev: [ref] 411 ns +- 20 ns -> [fast_init] 355 ns +- 18 ns: 1.16x 
faster (-14%)
---

The speedup of tp_fastnew + tp_fastinit is between 1.16x faster and 1.28x 
faster. The question is now if it is worth it.

Warning: The code is not fully optimized and is likely to have subtle bugs. The 
pull request is not ready for a review, but you may take a look if you would 
like to have an idea of the required changes. The most tricky changes are made 
in typeobject.c to support backward compatibility (call tp_new if tp_fastnew is 
not set) and stable API (support Python 3.6 PyTypeObject without the two slots).

Note: tp_fastnew and tp_fastinit slots are the expected end of my large 
FASTCALL optimization project.

--
components: Interpreter Core
messages: 286151
nosy: haypo, inada.naoki, rhettinger, serhiy.storchaka, yselivanov
priority: normal
pull_requests: 23
severity: normal
status: open
title: Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster object 
instanciation
type: performance
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29337] BytesWarning at compile time

[INADA Naoki]

INADA Naoki added the comment:

LGTM

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28556] typing.py upgrades

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

Thank you Martin!
Fixed this upstream https://github.com/python/typing/pull/365

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com