[issue33009] inspect.signature crashes on unbound partialmethods

[Antony Lee]

New submission from Antony Lee :

The following example crashes Python 3.6:

from functools import partialmethod
import inspect

class T:
g = partialmethod((lambda self, x: x), 1)

print(T().g())  # Correctly returns 1.
print(T.g(T()))  # Correctly returns 1.
print(inspect.signature(T.g))  # Crashes.

with

  File "/usr/lib/python3.6/inspect.py", line 3036, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
  File "/usr/lib/python3.6/inspect.py", line 2786, in from_callable
follow_wrapper_chains=follow_wrapped)
  File "/usr/lib/python3.6/inspect.py", line 2254, in 
_signature_from_callable
assert first_wrapped_param is not sig_params[0]
IndexError: tuple index out of range

--
components: Library (Lib)
messages: 313309
nosy: Antony.Lee
priority: normal
severity: normal
status: open
title: inspect.signature crashes on unbound partialmethods
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32999] issubclass(obj, abc.ABC) causes a segfault

[Alexey Izbyshev]

Change by Alexey Izbyshev :


--
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33008] urllib.request.parse_http_list incorrectly strips backslashes

[W. Trevor King]

New submission from W. Trevor King :

Python currently strips backslashes from inside quoted strings:

  $ echo 'a="b\"c",d=e' | python3 -c 'from sys import stdin; from 
urllib.request import parse_http_list; print(parse_http_list(stdin.read()))'
  ['a="b"c"', 'd=e']

It should be printing:

  ['a="b\"c"', 'd=e']

The bug is this continue [1], which should be removed.  This was not a problem 
with the original implementation [2].  It was introduced in [3] as a fix for 
#735248 with explicit tests asserting the broken behavior [3].  Stripping 
backslashes from the insides of quoted strings is not appropriate, because it 
breaks explicit unquoting with email.utils.unquote [4]:

  import email.utils
  import urllib.request
  list = r'"b\\"c"'
  entry = urllib.request.parse_http_list(list)[0]
  entry  # '"b\\"c"', should be '"b"c"'
  email.utils.unquote(entry)  # 'b"c', should be 'b\\"c'

I'm happy to file patches against the various branches if that would help, but 
as a one-line removal (plus adjusting the tests), it might be easier if a 
maintainer files the patches.

[1]: https://github.com/python/cpython/blob/v3.7.0b2/Lib/urllib/request.py#L1420
[2]: 
https://github.com/python/cpython/commit/6d7e47b8ea1b8cf82927dacc364689b8eeb8708b#diff-33f7983ed1a69d290366fe426880581cR777
[3]: 
https://github.com/python/cpython/commit/e1b13d20199f79ffd3407bbb14cc09b1b8fd70d2#diff-230a8abfedeaa9ae447490df08172b15R52
[4]: https://docs.python.org/3.5/library/email.util.html#email.utils.unquote

--
components: Library (Lib)
messages: 313308
nosy: labrat
priority: normal
severity: normal
status: open
title: urllib.request.parse_http_list incorrectly strips backslashes
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32997] Catastrophic backtracking in fpformat

[Benjamin Peterson]

Benjamin Peterson  added the comment:


New changeset 55d5bfba9482d39080f7b9ec3e6257ecd23f264f by Benjamin Peterson 
(Jamie Davis) in branch '2.7':
[2.7] closes bpo-32997: Fix REDOS in fpformat (GH-5984)
https://github.com/python/cpython/commit/55d5bfba9482d39080f7b9ec3e6257ecd23f264f


--
nosy: +benjamin.peterson
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33007] Objects referencing private-mangled names do not roundtrip properly under pickling.

[Antony Lee]

New submission from Antony Lee :

Consider the following example:

import pickle

class T:
def __init__(self):
self.attr = self.__foo

def __foo(self):
pass

print(pickle.loads(pickle.dumps(T(

This fails on 3.6 with `AttributeError: 'T' object has no attribute '__foo'` 
(i.e. there's a lookup on the unmangled name).  As a comparison, replacing 
`__foo` with `_foo` results in working code.

--
components: Library (Lib)
messages: 313306
nosy: Antony.Lee
priority: normal
severity: normal
status: open
title: Objects referencing private-mangled names do not roundtrip properly 
under pickling.
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23159] argparse: Provide equivalent of optparse.OptionParser.{option_groups, option_list, get_option}

[Eric McDonald]

Eric McDonald  added the comment:

Yes, this issue could be closed. I think the concept is still valid and perhaps 
worthy of future consideration as it is a means of unifying two different 
configuration processing mechanisms in the standard library without having 
developers reinvent that wheel every time.

I'll close it, though, if having a "stale" issue floating around is bothersome.

--
resolution:  -> later
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33005] 3.7.0b2 Interpreter crash in dev mode (or with PYTHONMALLOC=debug) with 'python -X dev -c 'import os; os.fork()'

[Xiang Zhang]

Change by Xiang Zhang :


--
nosy: +eric.snow, vstinner, xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33004] Shutil module functions could accept Path-like objects

[Marco Rougeth]

Marco Rougeth  added the comment:

You're right @josh.r! Thank you!

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25197] Allow documentation switcher to change url to /3/ and /dev/

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

Yeah I believe this issue is out of date. Thanks.

--
nosy: +Mariatta
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32969] Add more constants to zlib module

[Xiang Zhang]

Change by Xiang Zhang :


--
type: enhancement -> 
versions: +Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33006] docstring of filter function is incorrect

[Pierre Thibault]

New submission from Pierre Thibault :

> help(filter)

Help on built-in function filter in module __builtin__:

filter(...)
filter(function or None, sequence) -> list, tuple, or string

Return those items of sequence for which function(item) is true.  If
function is None, return the items that are true.  If sequence is a tuple
or string, return the same type, else return a list.
(END)

The second argument can be an iterable. Suggestion: Replace the docstring with 
the definition found at https://docs.python.org/2/library/functions.html#filter.

--
assignee: docs@python
components: Documentation
messages: 313302
nosy: Pierre Thibault, docs@python
priority: normal
severity: normal
status: open
title: docstring of filter function is incorrect
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: "except" and "subclasscheck" changed between CPython2 and 3

[Steven D'Aprano]

On Sat, 03 Mar 2018 04:28:24 +, 高岡陽太 wrote:

> Hello,
> 
> I found a difference of behavior about `except` statement between
> CPython 2.7 and 3.x .
> `except EXC_CLASS:` calls `__subclasscheck__` in 2.7, but does not in
> 3.x .


Python 3 does not accept virtual subclasses for exception handling. They 
have to be concrete subclasses of BaseException.

There is a feature-request to support that (as Python 2.7 does):

https://bugs.python.org/issue12029

but it is stalled.



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to access the help page of SRE_Pattern?

[Steven D'Aprano]

On Mon, 05 Mar 2018 18:13:59 -0600, Peng Yu wrote:

> Hi,
> 
 import re
 prog=re.compile('[a-f]+')
 help(prog)
> 
> I can use the above command to access SRE_Pattern. But this involves the
> creation of an object of the class.

If you're using help() interactively, the cost of creating the instance 
is about a millionth of the cost of actually reading the help text.

help(re.compile(''))

should be perfectly acceptable, performance-wise, even though it is a 
tiny bit longer to write than:

help(re)


If you really want a reference to the SRE_Pattern class, I'm afraid it is 
not public name. It is technically an implementation detail subject to 
change without notice. It could change its name, its internal details, 
its location, so long as it still offers the public regular expression 
interface.

I believe that SRE_Pattern is a built-in class, literally built into the 
interpreter.

The best way to get access to it is to do this once, at the beginning of 
your code:

SRE_Pattern = type(re.compile(''))


If you are doing this in the interactive interpreter, you might want to 
include that in your Python startup file.



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32919] csv.reader() to support QUOTE_ALL

[Pavel Shpilev]

Pavel Shpilev  added the comment:

I know that CSV specification says empty field and empty string are the same, 
however, I still believe there is practical use for unconventional processing 
of such fields.

In our specific case we parse CSVs produced by Amazon Athena (based on Presto) 
in which NULL and empty string values represented as above. Following CSV specs 
dogmatically, there's no way to distinguish between the two, but pragmatically 
you can tell them apart by simply looking at values.

Brief search shows we aren't the only ones facing the issue. After giving it 
some more thought, I'd agree that csv.QUOTE_ALL doesn't make much sense here, 
but may be an extra argument to csv.reader() will do the trick? Something like 
csv.reader(detect_none_values=False/True), with False being default, and 
emphasis in the documentation that True goes against CSV specification.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33004] Shutil module functions could accept Path-like objects

[Josh Rosenberg]

Josh Rosenberg  added the comment:

I think this falls under the umbrella of #30235, which posits that Path-like 
objects should be supported by shutil (and includes notes on doc validation).

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

How to access the help page of SRE_Pattern?

[Peng Yu]

Hi,

>>> import re
>>> prog=re.compile('[a-f]+')
>>> help(prog)

I can use the above command to access SRE_Pattern. But this involves
the creation of an object of the class.

I tried to directly access the class. But it does not work. Does
anybody know if there is a way to directly access the class help page?
Thanks.

>>> help(_sre.SRE_Pattern)
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'module' object has no attribute 'SRE_Pattern'
>>> help(SRE_Pattern)
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'SRE_Pattern' is not defined


-- 
Regards,
Peng
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to make Python run as fast (or faster) than Julia

[Dan Stromberg]

On Mon, Mar 5, 2018 at 3:53 PM, Python  wrote:
> On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote:
>> > Python is often a preferred solution because it is often fantastic for
>> > rapid implementation and maintainability.  The GIL's interference
>> > with threaded code performance has, for me at least, on several
>> > occasions been...  disappointing (perf costs of removing it aside)
>> > because it gets in the way of choosing Python for such solutions.
>> > Jython and IronPython are simply not feasible options for me, for
>> > multiple reasons that have zero to do with their technical
>> > suitability.
>>
>> Have you actually tried it and run into problems,
>
> Yes.  It was years ago and I forget the details, but I even posted
> some sample code here and was told (quite possibly by you) that it was
> the GIL that was eating my lunch.  Someone suggested writing the bits
> I wanted to thread as a C extension, which largely defeated the
> purpose of using Python.  In at least one case I just used C++, and in
> another I just ignored the problem until it went away.

So how about a little Cython?  It has decent GIL control, isn't much
different from Python syntactically, and can be used to create C
extension modules callable from CPython.  It allows you to pretty
freely intermix Python data types and C data types - just be careful
about implicit conversions from one to the other - they can slow
things down.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue28788] ConfigParser should be able to write config to a given filename, not only into file object

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +5765
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Chris Angelico]

On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano
 wrote:
> # Later.
> if __name__ = '__main__':
> # Enter the Kingdom of Nouns.

Don't you need a NounKingdomEnterer to do that for you?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to make Python run as fast (or faster) than Julia

[Python]

On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote:
> > Python is often a preferred solution because it is often fantastic for
> > rapid implementation and maintainability.  The GIL's interference
> > with threaded code performance has, for me at least, on several
> > occasions been...  disappointing (perf costs of removing it aside)
> > because it gets in the way of choosing Python for such solutions.
> > Jython and IronPython are simply not feasible options for me, for
> > multiple reasons that have zero to do with their technical
> > suitability.
> 
> Have you actually tried it and run into problems, 

Yes.  It was years ago and I forget the details, but I even posted
some sample code here and was told (quite possibly by you) that it was
the GIL that was eating my lunch.  Someone suggested writing the bits
I wanted to thread as a C extension, which largely defeated the
purpose of using Python.  In at least one case I just used C++, and in
another I just ignored the problem until it went away.

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue31226] shutil.rmtree fails when target has an internal directory junction (Windows)

[Vidar Fauske via Python-bugs-list]

Change by Vidar Fauske :


--
keywords: +patch
pull_requests: +5764
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Bitstream -- Binary Data for Humans

[Roel Schroeven]

Sébastien Boisgérault schreef op 5/03/2018 20:05:

I have released bitstream, a Python library to manage binary data (at the byte 
or bit level),

> hopefully without the pain that this kind of thing usually entails :)


If you have struggled with this topic in the past, please take a look at the 
documentation

> (http://boisgera.github.io/bitstream/) and tell me what you think.

Hi Sébastien,

At work I have some Python code to decode AIS[1] messages, for which I 
created my own code for handling binary data. It works, but is pretty 
slow. Not surprising, since handling data at the bit level is not 
exactly Python's strength. If I find the time, I'll try to replace my 
code with your bitstream and see if it does what I need it to do, and if 
it's any faster.


If/when I actually get around to it, I'll keep you informed.

[1] https://en.wikipedia.org/wiki/Automatic_identification_system


Best regards,
Roel

--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
  -- Isaac Asimov

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

[issue32991] AttributeError in doctest.DocTestFinder.find

[Jason R. Coombs]

Jason R. Coombs  added the comment:


New changeset b9650a04a81355c8a7dcd0464c28febfb4bfc0a9 by Jason R. Coombs in 
branch 'master':
bpo-32991: Restore expectation that inspect.getfile raises TypeError on 
namespace package (GH-5980)
https://github.com/python/cpython/commit/b9650a04a81355c8a7dcd0464c28febfb4bfc0a9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32991] AttributeError in doctest.DocTestFinder.find

[miss-islington]

Change by miss-islington :


--
pull_requests: +5763

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[miss-islington]

miss-islington  added the comment:


New changeset 96fdbacb7797a564249fd59ccf86ec153c4bb095 by Miss Islington (bot) 
in branch '3.7':
bpo-33001: Prevent buffer overrun in os.symlink (GH-5989)
https://github.com/python/cpython/commit/96fdbacb7797a564249fd59ccf86ec153c4bb095


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Steven D'Aprano]

On Mon, 05 Mar 2018 09:22:33 -0800, Ooomzay wrote:

[...]
>> Looking at that code, my major thought is  that there is far too much
>> OO design, not enough simplicity.
> 
> If this is far too much OO for you then RAII will be of no interest to
> you.

I think this is probably the wisest thing you have said in this entire 
discussion. OO design is often over-rated and elevated to the position of 
Holy Writ, full of over-engineered design patterns needed to overcome the 
flaws of OO design. I see no reason to design my application using OO 
principles when they aren't needed.

You say you have had great success with the RAII pattern in your own 
Python code. Great! I mean that sincerely. I'm really happy for you. But 
you now want to force that on *everyone*, even onto implementers where 
this will be an enormous burden to re-write their interpreter from the 
ground up to add a reference counter.

Why?

Do you have even the tiniest interest in running your RAII application 
under IronPython or Jython? Why do you care that your code, using CPython-
only features, must be portable to other implementations?

If your PEP is accepted, will you be volunteering your time for the next 
two or four years to re-write Jython and IronPython? To say nothing of 
PyPy and possibly Stackless?

CPython does what you want. So be satisfied that if you target CPython, 
you can use the RAII pattern to your heart's desire.


[...]
>> Perhaps I'm missing something, but I have no idea what benefit there is
>> in that style of code over:
>> 
>> with open('a') as a:
>> with open('b') as b:
>> process(a, b)
> 
> Encapsulation. Your application code is now managing details that should
> be hidden in the object. This PEP and RAII are unashamedly targeted at
> OO designs.

Encapsulation for the sake of encapsulation leads to monstrously over-
engineered heights of abstraction. Without a concrete use-case, I have no 
confidence that this *should* be "hidden in the object". Even if it 
should, I have no confidence that this specific design for encapsulation 
and/or data hiding[1] (they aren't the same thing) is the best design.


> If you would like to have a shot at coding this without RAII, but
> preserving the OO design, you will find that it is considerably
> _simpler_ than the with/context manager approach.

Preserving the OO design, you say? Okay, since my application apparently 
isn't allowed to know that it is processing two files, I'll simply 
delegate that to the object:

class C(A, B):
def run(self):
with open(self.a) as a:
with open(self.b) as b:
process(a, b)

# Later.
if __name__ = '__main__':
# Enter the Kingdom of Nouns.
c = C()
c.run()

There you go. Encapsulation and OO design.

Look, I'm sure that we could go back and forth for weeks trading more and 
more complicated, convoluted, esoteric or just plain unlikely scenarios 
leading up to the conclusion you want. I'm even willing to accept for the 
sake of discussion that in *your specific application's case*, you have 
come up with the best possible solution.

I'm not trying to dissuade you from using RAII in your own applications, 
if it works for you, great.

But I think it is unjustified to try to force that on all implementers 
unless you have a real need, not just as a matter of principle, to use 
RAII while still insisting on your code being implementation independent.


[...]
> If you choose RAII you will not be cavalier with your references.

You mean, "if you choose RAII, you cannot afford to be cavalier with your 
references, because if you fail to meet the rigorous demands of this 
pattern, your code will be buggy".

CPython has a long history of people relying on RAII and ending up with 
buggy code that doesn't close resources in a timely manner. The reason 
the with statement was invented was to be a simple and effective 
alternative to the RAII pattern, which promises the world in theory and 
fails to deliver in practice when it runs up against the reality that 
most people *are* cavalier with their references.

Not for you, obviously. I'm glad that it works for you. But as a 
community, we've been there and done that.




[1] They aren't the same thing.

-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "built-in" callables?

[Gregory Ewing]

Stefan Ram wrote:


  . So, what does "built-in" in the sense of »isbuiltin«
  actually mean?


It means "implemented in C".

Yes, this is confusing. The term "built-in" is used in two
different ways, and you just have to disambiguate them
from context.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

[issue33005] 3.7.0b2 Interpreter crash in dev mode (or with PYTHONMALLOC=debug) with 'python -X dev -c 'import os; os.fork()'

[Jason Madden]

Jason Madden  added the comment:

I built a local version of master (6821e73) and was able to get some line 
numbers (they're off by one for some reason, it appears):

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib  0x7fff78972e3e __pthread_kill + 10
1   libsystem_pthread.dylib 0x7fff78ab1150 pthread_kill + 333
2   libsystem_c.dylib   0x7fff788cf312 abort + 127
3   libsystem_malloc.dylib  0x7fff789cc866 free + 521
4   python.exe  0x000100fba715 _PyRuntimeState_Fini 
+ 37 (pystate.c:90)
5   python.exe  0x000100fb9d73 Py_FinalizeEx + 547 
(pylifecycle.c:1231)
6   python.exe  0x000100fddd80 pymain_main + 5808 
(main.c:2664)
7   python.exe  0x000100fdec82 _Py_UnixMain + 178 
(main.c:2697)
8   libdyld.dylib   0x7fff78823115 start + 1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33004] Shutil module functions could accept Path-like objects

[Marco Rougeth]

New submission from Marco Rougeth :

This is issue is to suggest an enhancement to the shutil module, I believe it's 
quiet similar to the issue32642.

I was using shutil.copytree to copy some files around and I tried to pass 
Path-like objects as input but got the exception "TypeError: argument should be 
string, bytes or integer, not PosixPath".

e.g.
build_path = BASE_DIR / 'build'
static_path = BASE_DIR / 'static'
shutil.copytree(static_path, build_path)


As said in issue32642, it "wasn't obvious because Path objects appear as 
strings in normal debug output". I had a look at the shutil source code and it 
seems that it wouldn't be to hard to implement. I'd love to do it, if it makes 
sense.

--
components: Library (Lib)
messages: 313295
nosy: rougeth
priority: normal
severity: normal
status: open
title: Shutil module functions could accept Path-like objects
type: enhancement
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33005] 3.7.0b2 Interpreter crash in dev mode (or with PYTHONMALLOC=debug) with 'python -X dev -c 'import os; os.fork()'

[Jason Madden]

New submission from Jason Madden :

At the request of Victor Stinner on twitter, I ran the gevent test suite with 
Python 3.7.0b2 with the new '-X dev' argument and discovered an interpreter 
crash. With a bit of work, it boiled down to a very simple command:

$ env -i .runtimes/snakepit/python3.7.0b2 -X dev -c 'import os; os.fork()'
*** Error in `.runtimes/snakepit/python3.7.0b2': munmap_chunk(): invalid 
pointer: 0x01c43a80 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f5a971607e5]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7f5a9716d698]
.runtimes/snakepit/python3.7.0b2(_PyRuntimeState_Fini+0x30)[0x515d90]
.runtimes/snakepit/python3.7.0b2[0x51445f]
.runtimes/snakepit/python3.7.0b2[0x42ce40]
.runtimes/snakepit/python3.7.0b2(_Py_UnixMain+0x7b)[0x42eaab]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f5a97109830]
.runtimes/snakepit/python3.7.0b2(_start+0x29)[0x42a0d9]
=== Memory map: 
0040-00689000 r-xp  08:01 177409 
//.runtimes/versions/python3.7.0b2/bin/python3.7
00888000-00889000 r--p 00288000 08:01 177409 
//.runtimes/versions/python3.7.0b2/bin/python3.7
00889000-008f3000 rw-p 00289000 08:01 177409 
//.runtimes/versions/python3.7.0b2/bin/python3.7
008f3000-00914000 rw-p  00:00 0
01b84000-01c64000 rw-p  00:00 0  [heap]
7f5a96052000-7f5a96068000 r-xp  08:01 265946 
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f5a96068000-7f5a96267000 ---p 00016000 08:01 265946 
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f5a96267000-7f5a96268000 rw-p 00015000 08:01 265946 
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f5a96268000-7f5a96273000 r-xp  08:01 268943 
/lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96273000-7f5a96472000 ---p b000 08:01 268943 
/lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96472000-7f5a96473000 r--p a000 08:01 268943 
/lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96473000-7f5a96474000 rw-p b000 08:01 268943 
/lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96474000-7f5a9647a000 rw-p  00:00 0
7f5a9647a000-7f5a96485000 r-xp  08:01 268947 
/lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96485000-7f5a96684000 ---p b000 08:01 268947 
/lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96684000-7f5a96685000 r--p a000 08:01 268947 
/lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96685000-7f5a96686000 rw-p b000 08:01 268947 
/lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96686000-7f5a9669c000 r-xp  08:01 268927 
/lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9669c000-7f5a9689b000 ---p 00016000 08:01 268927 
/lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9689b000-7f5a9689c000 r--p 00015000 08:01 268927 
/lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9689c000-7f5a9689d000 rw-p 00016000 08:01 268927 
/lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9689d000-7f5a9689f000 rw-p  00:00 0
7f5a9689f000-7f5a968a7000 r-xp  08:01 268938 
/lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a968a7000-7f5a96aa6000 ---p 8000 08:01 268938 
/lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a96aa6000-7f5a96aa7000 r--p 7000 08:01 268938 
/lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a96aa7000-7f5a96aa8000 rw-p 8000 08:01 268938 
/lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a96acc000-7f5a96b4c000 rw-p  00:00 0
7f5a96b4c000-7f5a96b4e000 r-xp  08:01 184551 
//.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96b4e000-7f5a96d4e000 ---p 2000 08:01 184551 
//.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96d4e000-7f5a96d4f000 r--p 2000 08:01 184551 
//.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96d4f000-7f5a96d51000 rw-p 3000 08:01 184551 
//.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96d51000-7f5a96e11000 rw-p  00:00 0
7f5a96e11000-7f5a970e9000 r--p  08:01 133586 
/usr/lib/locale/locale-archive
7f5a970e9000-7f5a972a9000 r-xp  08:01 268930 
/lib/x86_64-linux-gnu/libc-2.23.so
7f5a972a9000-7f5a974a9000 ---p 001c 08:01 268930 
/lib/x86_64-linux-gnu/libc-2.23.so
7f5a974a9000-7f5a974ad000 r--p 001c 08:01 268930 
/lib/x86_64-linux-gnu/libc-2.23.so
7f5a974ad000-7f5a974af000 

[issue33003] urllib: Document parse_http_list

[W. Trevor King]

New submission from W. Trevor King :

Python has had a parse_http_list helper since urllib2 landed in 6d7e47b8ea 
(EXPERIMENTAL, 2000-01-20).  With Python3 it was moved into urllib.request, and 
the implementation hasn't changed since (at least as of 4c19b9573, 2018-03-05). 
 External projects depend on the currently undocumented function [1,2], so it 
would be nice to have some user-facing documentation for it.  If that sounds 
appealing, I'm happy to work up a pull request.

[1]: https://github.com/requests/requests/blob/v2.18.4/requests/compat.py#L42
[2]: https://github.com/requests/requests/blob/v2.18.4/requests/compat.py#L58

--
assignee: docs@python
components: Documentation
messages: 313294
nosy: docs@python, labrat
priority: normal
severity: normal
status: open
title: urllib: Document parse_http_list
type: enhancement
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Steve Dower  added the comment:

Patches are merged, except for the ones that belong to @Larry.

Thanks again Alexey for the final round of feedback!

--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[miss-islington]

Change by miss-islington :


--
pull_requests: +5762

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Steve Dower  added the comment:


New changeset baa45079466eda1f5636a6d13f3a60c2c00fdcd3 by Steve Dower in branch 
'3.6':
[3.6] bpo-33001: Prevent buffer overrun in os.symlink (GH-5989) (GH-5990)
https://github.com/python/cpython/commit/baa45079466eda1f5636a6d13f3a60c2c00fdcd3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Steve Dower  added the comment:


New changeset 6921e73e33edc3c61bc2d78ed558eaa22a89a564 by Steve Dower in branch 
'master':
bpo-33001: Prevent buffer overrun in os.symlink (GH-5989)
https://github.com/python/cpython/commit/6921e73e33edc3c61bc2d78ed558eaa22a89a564


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29417] Sort entries in foo.dist-info/RECORD

[Éric Araujo]

Éric Araujo  added the comment:

Hello!  distutils does not write dist-info directories itself; wheel, 
setuptools and other build tools do that.  Look at https://github.com/pypa to 
find the bug trackers (I forget if wheel is there or on bitbucket)

--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29417] Sort entries in foo.dist-info/RECORD

[Dawei Wang]

Dawei Wang  added the comment:

This is important since for aws lambda the package can change every time.

--
nosy: +Dawei Wang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Ooomzay]

On Monday, 5 March 2018 19:14:05 UTC, Paul Rubin  wrote:
> Ooomzay writes:
> > If you want to use RAII objects then you will make sure you avoid
> > adding them to orphan cycles by design. If you don't know how to do
> > that then don't write applications that manage critical resources.
> 
> My claim is managing critical resources with refcounts is bug-prone in
> the case where the refcounts can become arbitrarily large at runtime.

What case is that? Don't go there! You may be jaded because python forgives 
bad design practices and peeps are inclined to leak resources all over 
the place for no good reason whatever and then complain when they have 
trouble cleaning up or exiting!

> You say you wrote a program that worked that way, but it sounds
> horrendous and I'd like to know how you tested it, maintained it, kept
> it maintainable by other programmers, etc.  
> It's painful to even think about.

I too used to suffer pain with python - until I saw that light and 
worked out how to use it for RAII. So here's the keys for much less pain:-

* Use CPython (or C++ ;)

* Use RAII for every resource-holding class: Implement __del__ to release any 
resources acquired in __init__.  (This is significantly less effort, and more 
reliable than adding __enter__ & __exit__ to every class).

* Wrap your try-except blocks in functions to prevent exceptions persisting 
outside the handler because in python they leak. This is typically a very 
natural factorization.

* Take care not to create Cyclic Exceptions in your Exception handling logic.
 This was the one that had me scratching my head for a couple of hours the 
first 
time I inadvertantly created a cycle.

* If you have no application-level requirement for persistent orphan cycles, 
and few, if any, applications do, then disable gc on entry and raise an 
exception on exit, or any other convenient moment if there is any garbage 
to be collected.

* Immediately plug/bug any leaks that you discover. Do not let them build 
up or you will drown and loose faith that there can be a better way.

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue33002] Making a class formattable as hex/oct integer with printf-style formatting requires both __int__ and __index__ for no good reason

[Josh Rosenberg]

Josh Rosenberg  added the comment:

To be clear, this is a problem with old-style (printf-style) formatting, and 
applies to both bytes formatting and str formatting. So a class like:

class Foo:
def __index__(self):
return 1

will fail with a TypeError should you do any of:

'%o' % Foo()
'%x' % Foo()
'%X' % Foo()
b'%o' % Foo()
b'%x' % Foo()
b'%X' % Foo()

even though hex(Foo()) and oct(Foo()) work without issue.

--
title: Making a class formattable as hex/oct integer requires both __int__ and 
__index__ for no good reason -> Making a class formattable as hex/oct integer 
with printf-style formatting requires both __int__ and __index__ for no good 
reason

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20092] type() constructor should bind __int__ to __index__ when __index__ is defined and __int__ is not

[Josh Rosenberg]

Josh Rosenberg  added the comment:

Pingback from #33002, which is caused by the fact that parts of the CPython 
code base that use PyNumber_Index for type conversion still pre-check for valid 
types with PyNumber_Check, meaning that a type with __index__ and not __int__ 
gets erroneously rejected by the pre-check.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33002] Making a class formattable as hex/oct integer requires both __int__ and __index__ for no good reason

[Josh Rosenberg]

Josh Rosenberg  added the comment:

Note: Obviously, defining __index__ without defining __int__ is a little 
strange (it's *equivalent* to int, but can't be *coerced* to int?), so yet 
another fix would be addressing #20092 so it wouldn't be possible for a type to 
define __index__ without (implicitly) defining __int__.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33002] Making a class formattable as hex/oct integer requires both __int__ and __index__ for no good reason

[Josh Rosenberg]

New submission from Josh Rosenberg :

In Python 2, making a user-defined class support formatting using the 
integer-specific type codes required that __int__ be defined and nothing else 
(that is, '%x' % Foo() only required Foo to provide a __int__ method). In 
Python 3, this was changed to perform the conversion via __index__ for the %o, 
%x and %X format types (to match how oct and hex behave), not __int__, but the 
pre-check for validity in unicodeobject.c's mainformatlong function is still 
based on PyNumber_Check, not PyIndex_Check, and PyNumber_Check is concerned 
solely with __int__ and __float__, not __index__.

This means that a class with __index__ but not __int__ can't be used with the 
%o/%x/%X format codes (even though hex(mytype) and oct(mytype) work just fine).

It seems to me that either:

1. PyNumber_Check should be a superset of PyIndex_Check (broader change, 
probably out of scope)

or

2. mainformatlong should restrict the scope of the PyNumber_Check test to only 
being used for the non-'o'/'x'/'X' tests (where it's needed to avoid coercing 
strings and the like to integer).

Change #2 should be safe, with no major side-effects; since PyLong and 
subclasses always passed the existing PyNumber_Check test anyway, and 
PyNumber_Index already performs PyIndex_Check, the only path that needs 
PyNumber_Check is the one that ends in calling PyNumber_Long.

--
components: Interpreter Core
messages: 313285
nosy: josh.r
priority: normal
severity: normal
status: open
title: Making a class formattable as hex/oct integer requires both __int__ and 
__index__ for no good reason
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32999] issubclass(obj, abc.ABC) causes a segfault

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

I've also checked that ABC.register() doesn't allow non-classes (and PEP 3119 
mentions that).

Looking at PyObject_IsSubclass in Objects/abstract.c, the only case in which 
its check_class() could be avoided is if there is a custom __subclasscheck__:

>>> class M(type):
...   def __subclasscheck__(cls, c):
... return c == 1 or super().__subclasscheck__(c)
...
>>> class A(metaclass=M):
...   pass
...
>>> issubclass(1, A)
True

If there is no need to support such weird __subclasscheck__, check_class() 
could be called earlier.

Note, however, that check_class() treats anything having __bases__ as a class, 
so moving the check alone is not enough to avoid the crash in all cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32642] add support for path-like objects in sys.path

[Jay Yin]

Jay Yin  added the comment:

I'm unsure how to regenerate the files that interact with the code for sys.path 
as travisCI states
"
Generated files not up to date
 M Python/importlib_external.h
"
since my code changes some of how the importing is handled

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Ooomzay]

On Monday, 5 March 2018 17:58:40 UTC, Chris Angelico  wrote:
> On Tue, Mar 6, 2018 at 4:53 AM, Ooomzay wrote:
> > On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico  wrote:
> >> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote:
> >> > Here is my fixed example, if someone else could try it in CPython and 
> >> > report back that would be interesting:-
> >> >
> >> > class RAIIFileAccess():
> >> > def __init__(self, fname):
> >> > print("%s Opened" % fname)
> >> > self.fname = fname
> >> >
> >> > def __del__(self):
> >> > print("%s Closed" % self.fname)
> >> >
> >> > class A():
> >> > def __init__(self):
> >> > self.res = RAIIFileAccess("a")
> >> >
> >> > class B():
> >> > def __init__(self):
> >> > self.res = RAIIFileAccess("b")
> >> >
> >> > class C():
> >> > def __init__(self):
> >> > self.a = A()
> >> > self.b = B()
> >> >
> >> > def main():
> >> > c = C()
> >> > c.dostuff()
> >> >
> >> > main()
> >>
> >> Here's how I'd do it with context managers.
> >>
> >> from contextlib import contextmanager
> >>
> >> @contextmanager
> >> def file_access(fname):
> >> try:
> >> print("%s Opened" % fname)
> >> yield
> >> finally:
> >> print("%s Closed" % fname)
> >>
> >> @contextmanager
> >> def c():
> >> try:
> >> print("Starting c")
> >> with file_access("a") as a, file_access("b") as b:
> >> yield
> >> finally:
> >> print("Cleaning up c")
> >>
> >> def main():
> >> with c():
> >> dostuff() # NameError
> >
> >
> > Thank you for having a go...
> >
> > However you have broken the encapsulation of class A and B. These
> > are trivial for the sake of example. I should have used
> > _underscores (i.e. self._res) to make the intent of
> > this example more obvious.
> >
> > Please try again but preserving the integrity/encapsulation
> > of class A & B & C, just as the RAII example does.
> 
> What is B? Is it something that's notionally a resource to be managed?

Yes. For example a supply of electrical power controlled via a serial protocol  
to a programmable power supply - the file is a private detail used for the 
communications. And lets imagine that this powersupply object needs to keep 
track of some state such as the voltage - and it has a long lifetime - not just 
created then destroyed in scope of one function i.e. it is a substantial object.

> If so, you can trivially add another level to the nesting.

Please illustrate. I really do want to be able to compare like for like.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bitstream -- Binary Data for Humans

[sum abiut]

Thanks

On 6/03/2018 7:13 AM, "Sébastien Boisgérault" <
sebastien.boisgera...@gmail.com> wrote:

Hi everyone,

I have released bitstream, a Python library to manage binary data (at the
byte or bit level), hopefully without the pain that this kind of thing
usually entails :)

If you have struggled with this topic in the past, please take a look at
the documentation (http://boisgera.github.io/bitstream/) and tell me what
you think.

Cheers,

Sébastien
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32993] urllib and webbrowser.open() can open w/ file: protocol

[Brett Cannon]

Change by Brett Cannon :


--
keywords: +security_issue
title: issue11662 Incomplete fix -> urllib and webbrowser.open() can open w/ 
file: protocol

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[Terry J. Reedy]

Change by Terry J. Reedy :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Eryk Sun]

Eryk Sun  added the comment:

>> As os.symlink requires administrative privileges on most versions 
>> of Windows
>
> The current implementation requires SeCreateSymbolicLinkPrivilege on 
> ALL versions of Windows because users must pass an additional flag to 
> CreateSymbolicLink to enable non-privileged symlinks on recent Windows
> 10, which os.symlink() doesn't do (see #31512).

The change in Windows 10 to allow unprivileged creation of links will be 
supported implicitly in 3.7, but this change is more for convenience than 
necessity. SeCreateSymbolicLinkPrivilege can be granted to standard users and 
groups. On my own systems, I grant this privilege to the "Authenticated Users" 
(S-1-5-11) well-known group. This even allows administrators to create symbolic 
links without having to elevate.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25427] Remove the pyvenv script in Python 3.8

[Brett Cannon]

Change by Brett Cannon :


--
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[Doug Hellmann]

Change by Doug Hellmann :


--
nosy: +doughellmann

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> As os.symlink requires administrative privileges on most versions of Windows

The current implementation requires SeCreateSymbolicLinkPrivilege on ALL 
versions of Windows because users must pass an additional flag to 
CreateSymbolicLink to enable non-privileged symlinks on recent Windows 10, 
which os.symlink() doesn't do (see #31512).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32305] Namespace packages have inconsistent __file__ and __spec__.origin

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
pull_requests: +5761

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[miss-islington]

miss-islington  added the comment:


New changeset 6935a511670797a3aaebdf96aad3dcff66baa76e by Miss Islington (bot) 
in branch '3.6':
bpo-32984: IDLE - set __file__ for startup files (GH-5981)
https://github.com/python/cpython/commit/6935a511670797a3aaebdf96aad3dcff66baa76e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Bitstream -- Binary Data for Humans

[Sébastien Boisgérault]

Hi everyone,

I have released bitstream, a Python library to manage binary data (at the byte 
or bit level), hopefully without the pain that this kind of thing usually 
entails :)

If you have struggled with this topic in the past, please take a look at the 
documentation (http://boisgera.github.io/bitstream/) and tell me what you think.

Cheers,

Sébastien
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

While judging by the source code it seems that bytes in 3.5 should be fine, 
I've got a crash with the latest binary from python.org:

Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)]
 on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.symlink(b'x\\' * 129, b'y\\' * 129)
(Windows pop-up here)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32305] Namespace packages have inconsistent __file__ and __spec__.origin

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Mar 5, 2018, at 10:33, Ned Batchelder  wrote:

> As is usual for me, I am here because some coverage.py code broke due to this 
> change.  A diff between b1 and b2 found me the code change (thanks for the 
> comment, btw!), but a What's New doesn't seem out of place.

Sounds good; I’ll work up a PR

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[miss-islington]

Change by miss-islington :


--
pull_requests: +5760

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Ways to make a free variable local to a function?

[Terry Reedy]

On 3/5/2018 9:34 AM, Chris Angelico wrote:

On Tue, Mar 6, 2018 at 12:52 AM, Terry Reedy  wrote:

On 3/5/2018 7:12 AM, Kirill Balunov wrote:

# 1. By passing through local variable's default values

  def func_local_1(numb, _int = int, _float = float, _range = range):



You are not required to mangle the names.

def func_local_1(numb, int = int, float = float, range = range):
...



Even so, this does mess up the function's signature,


Which I why I only said that using the original names solves the syntax 
highlighting issue (of marking built-ins as built-ins).



leaving your
callers wondering if they can call it with some sort of range
parameter. (Though in this particular instance, range() is only called
once, so it's pretty much useless to try to optimize it.)

In theory, the CPython bytecode compiler (don't know about other
Python implementations) could just add these as constants.  


Yes, what we really want for this sort of thing are unrebindable local 
constants.  A simple syntax change could do it.


 def func_local_1(numb; int = int, float = float, range = range):

The binding after ';' belong in the header because they should be done once.


They'd then
be bound at either compile time or function definition time (by
default the former, I think, but the latter would be more useful), and
be looked up as quickly as locals. I'm not sure how useful this would
be, though.


I believe that the occasional practice of re-binding built-in names to 
locals can be shown to speed up loops run enough times.



If PEP 572 [1] were to be accepted, you could do something like this:

def func(numb):
 if ((int as int), (float as float)):
 res = []
 for i in range(numb):
 res.append(int(i) + float(i))
 return res

Syntactically a bit clunky, but keeps everything inside the function,
and DOES create local variables. Not sure it's better than your other
options, but it is another option.


Code in the body should be executed everytime the function is called. 
Those binding should not be as they only need to be done once.



[1] PEP 572: https://www.python.org/dev/peps/pep-0572/


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

[issue32305] Namespace packages have inconsistent __file__ and __spec__.origin

[Ned Batchelder]

Ned Batchelder  added the comment:

As is usual for me, I am here because some coverage.py code broke due to this 
change.  A diff between b1 and b2 found me the code change (thanks for the 
comment, btw!), but a What's New doesn't seem out of place.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +5759

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +5758

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +5757

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32991] AttributeError in doctest.DocTestFinder.find

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Good catch Jason.  Your fix is exactly right.  I approved your PR, which is 
against master, so it should definitely be backported to 3.7.  No need to 
backport to 3.6; we reverted the change for that release.

--
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

Change by Steve Dower :


--
keywords: +patch
pull_requests: +5756
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33001] Buffer overflow vulnerability in os.symlink on Windows

[Steve Dower]

New submission from Steve Dower :

On February 27th, 2018, the Python Security Response team was notified of a 
buffer overflow issue in the os.symlink() method on Windows. The issue affects 
all versions of Python between 3.2 and 3.6.4, including the 3.7 beta releases. 
It will be patched for the next releases of 3.4, 3.5, 3.6 and 3.7.

Scripts may be vulnerable if they use os.symlink() on Windows and an attacker 
is able to influence the location where links are created. As os.symlink 
requires administrative privileges on most versions of Windows, exploits using 
this vulnerability are likely to achieve escalation of privilege.

Besides applying the fix to CPython, scripts can also ensure that the length of 
each path argument is less than 260, and if the source is a relative path, that 
its combination with the destination is also shorter than 260 characters. That 
is:

assert (len(src) < 260 and
len(dest) < 260 and
len(os.path.join(os.path.dirname(dest), src)) < 260)
os.symlink(src, dest)

Scripts that explicitly pass the target_is_directory argument as True are not 
vulnerable. Also, scripts on Python 3.5 that use bytes for paths are not 
vulnerable, because of a combination of stack layout and added parameter 
validation.

I will be requesting a CVE for this once the patches are applied to maintenance 
branches, and then notifying the security-announce list. The patch has been 
reviewed by the PSRT and reporter, and while it prevents the buffer overflow, 
it does not raise any new errors or enable the use of long paths when creating 
symlinks.

Many thanks to Alexey Izbyshev for the report, and helping us work through 
developing the patch.

--
assignee: steve.dower
components: Windows
keywords: security_issue
messages: 313275
nosy: izbyshev, paul.moore, steve.dower, tim.golden, zach.ware
priority: critical
severity: normal
status: open
title: Buffer overflow vulnerability in os.symlink on Windows
type: security
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Chris Angelico]

On Tue, Mar 6, 2018 at 4:53 AM, Ooomzay  wrote:
> On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico  wrote:
>> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote:
>> > Here is my fixed example, if someone else could try it in CPython and 
>> > report back that would be interesting:-
>> >
>> > class RAIIFileAccess():
>> > def __init__(self, fname):
>> > print("%s Opened" % fname)
>> > self.fname = fname
>> >
>> > def __del__(self):
>> > print("%s Closed" % self.fname)
>> >
>> > class A():
>> > def __init__(self):
>> > self.res = RAIIFileAccess("a")
>> >
>> > class B():
>> > def __init__(self):
>> > self.res = RAIIFileAccess("b")
>> >
>> > class C():
>> > def __init__(self):
>> > self.a = A()
>> > self.b = B()
>> >
>> > def main():
>> > c = C()
>> > c.dostuff()
>> >
>> > main()
>>
>> Here's how I'd do it with context managers.
>>
>> from contextlib import contextmanager
>>
>> @contextmanager
>> def file_access(fname):
>> try:
>> print("%s Opened" % fname)
>> yield
>> finally:
>> print("%s Closed" % fname)
>>
>> @contextmanager
>> def c():
>> try:
>> print("Starting c")
>> with file_access("a") as a, file_access("b") as b:
>> yield
>> finally:
>> print("Cleaning up c")
>>
>> def main():
>> with c():
>> dostuff() # NameError
>
>
> Thank you for having a go...
>
> However you have broken the encapsulation of class A and B. These
> are trivial for the sake of example. I should have used
> _underscores (i.e. self._res) to make the intent of
> this example more obvious.
>
> Please try again but preserving the integrity/encapsulation
> of class A & B & C, just as the RAII example does.

What is B? Is it something that's notionally a resource to be managed?
If so, you can trivially add another level to the nesting.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32305] Namespace packages have inconsistent __file__ and __spec__.origin

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I guess it depends on whether you think this is a new feature or a bug fix.  
Or, OTOH, since we had to revert for 3.6, maybe it makes sense either way since 
some code will be affected.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Ooomzay]

On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico  wrote:
> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote:
> > Here is my fixed example, if someone else could try it in CPython and 
> > report back that would be interesting:-
> >
> > class RAIIFileAccess():
> > def __init__(self, fname):
> > print("%s Opened" % fname)
> > self.fname = fname
> >
> > def __del__(self):
> > print("%s Closed" % self.fname)
> >
> > class A():
> > def __init__(self):
> > self.res = RAIIFileAccess("a")
> >
> > class B():
> > def __init__(self):
> > self.res = RAIIFileAccess("b")
> >
> > class C():
> > def __init__(self):
> > self.a = A()
> > self.b = B()
> >
> > def main():
> > c = C()
> > c.dostuff()
> >
> > main()
> 
> Here's how I'd do it with context managers.
> 
> from contextlib import contextmanager
> 
> @contextmanager
> def file_access(fname):
> try:
> print("%s Opened" % fname)
> yield
> finally:
> print("%s Closed" % fname)
> 
> @contextmanager
> def c():
> try:
> print("Starting c")
> with file_access("a") as a, file_access("b") as b:
> yield
> finally:
> print("Cleaning up c")
> 
> def main():
> with c():
> dostuff() # NameError
 

Thank you for having a go...

However you have broken the encapsulation of class A and B. These 
are trivial for the sake of example. I should have used
_underscores (i.e. self._res) to make the intent of
this example more obvious.

Please try again but preserving the integrity/encapsulation
of class A & B & C, just as the RAII example does.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32921] .pth files cannot contain folders with utf-8 names

[Steve Dower]

Steve Dower  added the comment:

Yes, it'll have significant side effects. The default file encoding on Windows 
is your configured code page (1252, in your case), and there's no good way 
around that default. The easiest immediate fix is to re-encode that file 
yourself.

Perhaps what we could do instead is allow the first line of a .pth file to be a 
coding comment? Then site.py can reopen the file with the specified encoding.

(FWIW, when I added the ._pth file, I explicitly made it UTF-8. But it had no 
history at that time so it was safe to do so.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33000] IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes / character stored

[Terry J. Reedy]

Terry J. Reedy  added the comment:

IDLEX is an independently developed and distributed set of IDLE eXtensions.  We 
have nothing to do with it.

IDLE, as the name suggests and the doc spells out, is an Integrated  
Development and Learning Environment.  It is not a production run environment.  
For interactive development, one can close the shell window and reclaim its 
memory while keeping an editor window open.

16 million lines of screen output way outside of IDLE's intended use.  Why are 
you (mis)using it this way?

As you already discovered, an effectively infinite output stream can be sent to 
a console.  Command Prompt has a similar behavior.  If you want more saved, 
output to disk.

Or write your own gui for your simulations.  IDLE (and hence IDLEX) uses the 
tkinter wrapper of the tcl/tk GUI framework.  tcl/tk is also separately 
developed and distributed.  We just use it.  Its text widget does not come with 
any automatic size management.  But you could write your own insert wrapper 
that would delete a line for each line added after you deem the widget 'full'.

When I ran your sample with 64 bit 3.7.0b2 on Win10, the memory reported by 
Task Manager increased from 30 to 195 Mb or 165 MB net, much less than you 
report.  I once printed about 500,000 60 char lines.  I had either 12 or 24 GB 
memory at the time.

I am inclined to close this issue as 'third party' (IDLEX and tcl/tk) but I 
will let you respond first.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Ooomzay]

On Monday, 5 March 2018 16:47:02 UTC, Steven D'Aprano  wrote:
> On Sun, 04 Mar 2018 16:58:38 -0800, Ooomzay wrote:
> 
> > Here is an example of a composite resource using RAII:-
> > 
> > class RAIIFileAccess():
> > def __init__(self, fname):
> > print("%s Opened" % fname)
> > def __del__(self):
> > print("%s Closed" % fname)
> > 
> > class A():
> > def __init__(self):
> > self.res = RAIIFileAccess("a")
> > 
> > class B():
> > def __init__(self):
> > self.res = RAIIFileAccess("b")
> > 
> > class C():
> > def __init__(self):
> > self.a = A()
> > self.b = B()
> >  
> > def main():
> > c = C()
> 
> Looking at that code, my major thought is  that there is far too much OO 
> design, not enough simplicity.

If this is far too much OO for you then RAII will be of no 
interest to you.

This example was specifically in response to a request to
illustrate the relative simplicity of RAII in the case of a 
composite (OO) resource.

> Perhaps I'm missing something, but I have no idea what benefit there is 
> in that style of code over:
> 
> with open('a') as a:
> with open('b') as b:
> process(a, b)

Encapsulation. Your application code is now managing details 
that should be hidden in the object. This PEP and RAII are
unashamedly targeted at OO designs.

If you would like to have a shot at coding this without RAII,
but preserving the OO design, you will find that it is 
considerably _simpler_ than the with/context manager approach.

> So long as you only process a or b inside the nested block, you are 
> guaranteed that they will be open.
> 
> And unlike your RAII example, they will be closed when you exit, 
> regardless of how many references to them you have, regardless of whether 
> an exception occurs or not, regardless of whether there are cycles or 
> whether they are globals or whether the interpreter is shutting down.

If you choose RAII you will not be cavalier with your references.

> I think that at this point, you have convinced me that you want to impose 
> enormous costs on all Python interpreters *and* Python developers, in 
> order to allow you to write C++ code in Python rather than learn Pythonic 
> idioms like the with statement.

On interpreters yes. On developers no. You can carry on exactly as you are.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32968] Fraction modulo infinity should behave consistently with other numbers

[Elias Zamaria]

Elias Zamaria  added the comment:

Done.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32829] Lib/ be more pythonic

[Дилян Палаузов]

Дилян Палаузов  added the comment:

The variables got_it in distutils/command/sdist and quote in 
email/_header_value_parser can be skipped making the code shorter and faster.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32999] issubclass(obj, abc.ABC) causes a segfault

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> Is there any sense in accepting non-types as the first argument of 
> issubclass()?

No, though it is not (clearly) documented. The docs mention TypeError, but only 
for the second argument if my reading is correct.

In practice, issubclass() raises a TypeError if the first argument is not a 
class object:

>>> issubclass(1, int)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: issubclass() arg 1 must be a class

Though, as I mentioned above, behavior for ABCs was always weird.

--
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Steven D'Aprano]

On Mon, 05 Mar 2018 07:31:57 -0800, Ooomzay wrote:

> We do not expect this to work in PyPy.

Or Jython, IronPython, possibly not Stackless either, or in the 
interactive interpreter of (probably) any implementation, or CPython if 
the object is garbage collected during interpreter shutdown or is in a 
cycle or has a global reference, or if the object simply doesn't go out 
of scope as quickly as you would like.


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32997] Catastrophic backtracking in fpformat

[James Davis]

James Davis  added the comment:

Equivalent, probably cleaner. Comment on the PR if you want a change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[miss-islington]

miss-islington  added the comment:


New changeset fd340bf9e308130736c76257ff9a697edbeb082d by Miss Islington (bot) 
in branch '3.7':
bpo-32984: IDLE - set __file__ for startup files (GH-5981)
https://github.com/python/cpython/commit/fd340bf9e308130736c76257ff9a697edbeb082d


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32999] issubclass(obj, abc.ABC) causes a segfault

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Is there any sense in accepting non-types as the first argument of 
issubclass()? I would add a check just in issubclass().

--
nosy: +inada.naoki, serhiy.storchaka
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Steven D'Aprano]

On Sun, 04 Mar 2018 16:58:38 -0800, Ooomzay wrote:

> Here is an example of a composite resource using RAII:-
> 
> class RAIIFileAccess():
> def __init__(self, fname):
> print("%s Opened" % fname)
> def __del__(self):
> print("%s Closed" % fname)
> 
> class A():
> def __init__(self):
> self.res = RAIIFileAccess("a")
> 
> class B():
> def __init__(self):
> self.res = RAIIFileAccess("b")
> 
> class C():
> def __init__(self):
> self.a = A()
> self.b = B()
>  
> def main():
> c = C()

Looking at that code, my major thought is  that there is far too much OO 
design, not enough simplicity.

Perhaps I'm missing something, but I have no idea what benefit there is 
in that style of code over:

with open('a') as a:
with open('b') as b:
process(a, b)

So long as you only process a or b inside the nested block, you are 
guaranteed that they will be open.

And unlike your RAII example, they will be closed when you exit, 
regardless of how many references to them you have, regardless of whether 
an exception occurs or not, regardless of whether there are cycles or 
whether they are globals or whether the interpreter is shutting down.

I think that at this point, you have convinced me that you want to impose 
enormous costs on all Python interpreters *and* Python developers, in 
order to allow you to write C++ code in Python rather than learn Pythonic 
idioms like the with statement.


I don't think this is a good tradeoff.


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue32997] Catastrophic backtracking in fpformat

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Wouldn't be easier to remove '0*' from the pattern? 0s could be stripped later 
by .lstrip('0').

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[mike bayer]

mike bayer  added the comment:

for those watching this would be the findall() case which is consistent between 
pythons:

import re

for reg in [
'VARCHAR(30) COLLATE "en_US"',
'VARCHAR(30)'
]:

print(re.findall(r'(?: COLLATE.*)?$', reg))


output (all pythons):

[' COLLATE "en_US"', '']
['']

so yes there are two matches for one and only one for the other.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33000] IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes / character stored

[John Brearley]

New submission from John Brearley :

While running a tensorflow script in the IDLEX GUI that runs for 8 million 
steps and produce 2 lines stdout per step, my PC used all 16GB RAM and crashed 
the python process, not to mention messed up other apps, like Firefox & Norton 
AntiVirus. While the RAM was recovered, Firefox started responding, but Norton 
Antivirus didn’t, so the PC had to be rebooted. 
 
The issue is easily reproduced with the short print loop that dumps 20K lines 
of stdout, at 171 characters / line on the IDLEX GUI window. When the script is 
run in the IDLEX GUI, the Windows Task Manager shows the python process start 
at 19MB RAM consumption, then grows to 569MB RAM consumption. If I run the 
script a second time in the same IDLEX GUI window, it grows to 1.1GB RAM 
consumption. 
 
So 20K lines off output at 171 characters / line (“i: n” prefix + 2 * 80 
byte string + newline) = 3.4M total characters stored in the scrollback buffer. 
The delta memory consumed was 569MB – 19MB = 550MB. The RAM consumed / 
character is 550MB / 3.4M = 161 bytes / character. This seems excessively 
inefficient.
 
I now understand how the tensorflow script would stop after 550K iterations and 
the 550K lines of stdout in the IDLEX GUI would consume all 16GB RAM on my PC.
 
BTW, when I run the same test script in the WinPython command prompt window, it 
only consumes 4MB RAM while it runs. However the scrollback buffer is limited 
to 10K lines, wrapped at the 80 character mark, so much less data saved.
 
I haven’t found any options in IDLEX GUI to limit the scrollback buffer size.
 
My request is to review the scrollback memory storage algorithms. If nothing 
can be done to improve them, then please add a circular buffer to limit the 
memory consumption.
 
# Print loop to test memory consumption in Python IDLEX GUI.
s1 = "0123456789"
s2 = s1+s1+s1+s1+s1+s1+s1+s1
for i in range(2):
   print("i:", i, s2, s2)

I am using Python 3.6.4 on Windows 7 PC, Intel i7-4770S, 3.1GHz, 16GB RAM.

--
assignee: terry.reedy
components: IDLE
messages: 313263
nosy: jbrearley, terry.reedy
priority: normal
severity: normal
status: open
title: IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes / 
character stored
type: resource usage
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[mike bayer]

mike bayer  added the comment:

for now the quickest solution is to add "count=1" so that it only replaces once.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32999] issubclass(obj, abc.ABC) causes a segfault

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

In debug mode, the following assertion fails:

python: ./Modules/_abc.c:642: _abc__abc_subclasscheck_impl: Assertion 
`((PyObject*)(mro))->ob_type))->tp_flags & ((1UL << 26))) != 0)' failed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

In your case you can just pass 1 as the fourth parameter of re.sub().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32969] Add more constants to zlib module

[Xiang Zhang]

Change by Xiang Zhang :


--
keywords: +patch
pull_requests: +5755
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32999] issubclass(obj, abc.ABC) causes a segfault

[Alexey Izbyshev]

New submission from Alexey Izbyshev :

Demo:

>>> from abc import ABC
>>> issubclass(1, ABC)
Segmentation fault (core dumped)

The stack trace is attached.

Before reimplementation of abc in C, the result was confusing too:

Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)]
 on win32
>>> from abc import ABC
>>> issubclass(1, ABC)
Traceback (most recent call last):
  File "", line 1, in 
  File "abc.py", line 230, in __subclasscheck__
  File "_weakrefset.py", line 84, in add
TypeError: cannot create weak reference to 'int' object

--
components: Extension Modules
files: stack-trace.txt
messages: 313259
nosy: izbyshev, levkivskyi
priority: normal
severity: normal
status: open
title: issubclass(obj, abc.ABC) causes a segfault
type: crash
versions: Python 3.8
Added file: https://bugs.python.org/file47470/stack-trace.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[mike bayer]

mike bayer  added the comment:

also, removing the "?" is not an option for me.   I need the brackets to be 
placed prior to the "COLLATE" subsection, but unconditionally even if the 
"COLLATE" section is not present. Looking at the change the behavior seems 
wrong to me.   The regexp states, "match the end of the string, plus an 
optional "COLLATE" clause, into a capturing expression.  replace everything 
here, e.g. the capturing part as well as the dollar sign part, with a single 
instance of FOO plus the captured part".   It is entirely unintuitive to me how 
a second replacement would be occurring here.   I cannot prove it but I think 
this change is wrong.

I will try to rewrite the expression.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Just see the re.sub() documentation for 3.7. There is also a note in the What's 
New document, in the "Changes in the Python API" section.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[mike bayer]

mike bayer  added the comment:

can you point me to the documentation?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This is intentional change.

Prior to 3.7 re.sub() didn't replace empty matches adjacent to a previous 
non-empty match. In 3.7 it does. Together with other changes this made all four 
functions that search multiple matches of the pattern (re.findall(), 
re.finditer(), re.split() and re.sub()) consistent.

In your example the pattern matches not only from " COLLATE" to the end of 
input string, but an empty string at the end of input string. If you do not 
want matching an empty string, just remove the '?' qualifier.

--
nosy: +serhiy.storchaka
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[miss-islington]

Change by miss-islington :


--
pull_requests: +5754

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[miss-islington]

Change by miss-islington :


--
pull_requests: +5753
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32984] IDLE: set and unset __file__ for startup files

[Terry J. Reedy]

Terry J. Reedy  added the comment:


New changeset 22c82be5df70c3d51e3f89b54fe1d4fb84728c1e by Terry Jan Reedy in 
branch 'master':
bpo-32984: IDLE - set __file__ for startup files (GH-5981)
https://github.com/python/cpython/commit/22c82be5df70c3d51e3f89b54fe1d4fb84728c1e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26707] plistlib fails to parse bplist with 0x80 UID values

[Jon Janzen]

Jon Janzen  added the comment:

@serhiy.storchaka: I've implemented a UID wrapper class

I've also updated the parser and writer classes to support the UID wrapper. The 
implementations for reading/writing XML UID tags match the implementations 
given by Apple's plutil distributed with macOS:

UID(x) becomes {'CF$UID': int(x)}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22102] Zipfile generates Zipfile error in zip with 0 total number of disk in Zip64 end of central directory locator

[Francisco Facioni]

Change by Francisco Facioni :


--
keywords: +patch
pull_requests: +5752
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[mike bayer]

mike bayer  added the comment:

correction, that's fedora 26, not 27

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32998] regular expression regression in python 3.7

[mike bayer]

New submission from mike bayer :

demo:

import re

inner = 'VARCHAR(30) COLLATE "en_US"'

result = re.sub(
r'((?: COLLATE.*)?)$',
r'FOO\1',
inner
)

print(inner)
print(result)


in all Python versions prior to 3.7:

VARCHAR(30) COLLATE "en_US"
VARCHAR(30)FOO COLLATE "en_US"

in Python 3.7.0b2:

VARCHAR(30) COLLATE "en_US"
VARCHAR(30)FOO COLLATE "en_US"FOO

platform: Fedora 27 
python build:
Python 3.7.0b2 (default, Mar  5 2018, 09:37:32) 
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux

--
components: Library (Lib)
messages: 313251
nosy: zzzeek
priority: normal
severity: normal
status: open
title: regular expression regression in python 3.7
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: RFC: Proposal: Deterministic Object Destruction

[Ooomzay]

On Monday, 5 March 2018 15:17:13 UTC, bartc  wrote:
> On 05/03/2018 13:58, Ooomzay wrote:
> > On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico  wrote:
> >> On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote:
> >>> Here is an example of a composite resource using RAII:-
> >>>
> >>> class RAIIFileAccess():
> >>>  def __init__(self, fname):
> >>>  print("%s Opened" % fname)
> >>>  def __del__(self):
> >>>  print("%s Closed" % fname)
> >>>
> >>> class A():
> >>>  def __init__(self):
> >>>  self.res = RAIIFileAccess("a")
> >>>
> >>> class B():
> >>>  def __init__(self):
> >>>  self.res = RAIIFileAccess("b")
> >>>
> >>> class C():
> >>>  def __init__(self):
> >>>  self.a = A()
> >>>  self.b = B()
> >>>
> >>> def main():
> >>>  c = C()
> >>>
> >>> Under this PEP this is all that is needed to guarantee that the files "a"
> >>> and "b" are closed on exit from main or after any exception has been 
> >>> handled.
> >>
> >> Okay. And all your PEP needs is for reference count semantics, right?
> >> Okay. I'm going to run this in CPython, with reference semantics. You
> >> guarantee that those files will be closed after an exception is
> >> handled? Right.
> >>
> > def main():
> >> ... c = C()
> >> ... c.do_stuff()
> >> ...
> > main()
> >> a Opened
> >> b Opened
> >> Traceback (most recent call last):
> >>File "", line 1, in 
> >>File "", line 3, in main
> >> AttributeError: 'C' object has no attribute 'do_stuff'
> >
> >>   
> >> Uhh I'm not seeing any messages about the files getting closed.
> > 
> > Then that is indeed a challenge. From CPython back in 2.6 days up to 
> > Python36-32 what I see is:-
> > 
> > a Opened
> > b Opened
> > Traceback (most recent call last):
> > ...
> > AttributeError: 'C' object has no attribute 'dostuff'
> > a Closed
> > b Closed
> > 
> >> Maybe exceptions aren't as easy to handle as you think?
> > 
> > Well there is a general issue with exceptions owing to the ease
> > with which one can create cycles that may catch out newbs. But
> > that is not the case here.
> > 
> >> Or maybe you
> >> just haven't tried any of this (which is obvious from the bug in your
> >> code
> > 
> > Or maybe I just made a typo when simplifying my test case and failed to 
> > retest?
> > 
> > Here is my fixed case, if someone else could try it in CPython and report 
> > back that would be interesting:-
> > 
> > class RAIIFileAccess():
> >  def __init__(self, fname):
> >  print("%s Opened" % fname)
> >  self.fname = fname
> > 
> >  def __del__(self):
> >  print("%s Closed" % self.fname)
> > 
> > class A():
> >  def __init__(self):
> >  self.res = RAIIFileAccess("a")
> > 
> > class B():
> >  def __init__(self):
> >  self.res = RAIIFileAccess("b")
> > 
> > class C():
> >  def __init__(self):
> >  self.a = A()
> >  self.b = B()
> > 
> > def main():
> >  c = C()
> >  c.dostuff()
> > 
> > main()
> 
> I get A and B closed messages when running on CPython 2.7 and 3.6, with 
> the code run from a .py file. (I never use interactive mode.)
> 
> But not when running on a PyPy version of 2.7 (however that is not CPython).

Thanks bartc, I have made the example more complete by adding an exception 
scope - this means it works as designed - in any context. See my reply to Chris.

We do not expect this to work in PyPy.
-- 
https://mail.python.org/mailman/listinfo/python-list