[issue27660] Replace size_t with Py_ssize_t as the type of local variable in list_resize

[Xiang Zhang]

Xiang Zhang added the comment:

v3 applies haypo's suggestion.

--
Added file: http://bugs.python.org/file44192/list_resize_v3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25825] AIX shared library extension modules installation broken

[David Edelsohn]

David Edelsohn added the comment:

> This bug was marked for 2.7 as well. Is there anything that needs to be done 
> for 2.7?

It would be great if both patches were applied to 2.7 also.

> How does patch 2 make the test_distutils situation worse? Is there anything 
> that should be done to improve things first, or should that just be handled 
> independently? I see that the 2.7 and 3.5 buildbots are currently failing 
> with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit 
status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' 
failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining 
about the plain “config” path.

Patch2 makes test_distutils worse because LDSHARED refers to a file that only 
exists when Python is installed.  If one tests in tree, the files don't exist.  
This may be related to the existence of BLDSHARED, used to build modules in the 
source tree, which can be overridden separately -- otherwise it defaults to 
LDSHARED.  If the rest of the testsuite ever is cleaned up for AIX, one either 
could skip the parts of test_distutils that require the LDSHARED files or add 
special hooks in the test for AIX.  As you point out, the change doesn't make 
the current testsuite results situation any worse.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25825] AIX shared library extension modules installation broken

[Martin Panter]

Martin Panter added the comment:

It looks like the change from plain config to config- was made in Issue 
9807 (r86731, 3.2). So the patch seems reasonable to me; it is just catching up 
with that change.

This bug was marked for 2.7 as well. Is there anything that needs to be done 
for 2.7?

How does patch 2 make the test_distutils situation worse? Is there anything 
that should be done to improve things first, or should that just be handled 
independently? I see that the 2.7 and 3.5 buildbots are currently failing with 
errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit 
status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' 
failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining 
about the plain “config” path.

--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27831] Python 3.4.5 leaks memory when attempting to load non-existent file

[geeknik]

New submission from geeknik:

While fuzzing Python 3.4.5 with AFL, ASAN and libdislocator, I fat fingered the 
keyboard ASAN pops up a LeakSanitizer error. Apparently trying to open a 
non-existent file causes Python 3.4.5 to leak >880KB of memory.

./python: can't open file 'id:000109': [Errno 2] No such file or directory

=
==16921==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 620904 byte(s) in 469 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12

Direct leak of 41800 byte(s) in 29 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x769b2f in PyEval_EvalCodeEx /root/python-3.4.5/Python/ceval.c:3588:14

Direct leak of 28160 byte(s) in 7 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x60459c in set_add_key /root/python-3.4.5/Objects/setobject.c:373:12

Direct leak of 26095 byte(s) in 33 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4dc54d in PyMem_RawMalloc /root/python-3.4.5/Objects/obmalloc.c:276:12
#2 0x4dc54d in _PyObject_Malloc /root/python-3.4.5/Objects/obmalloc.c:1314

Direct leak of 18176 byte(s) in 31 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4dc54d in PyMem_RawMalloc /root/python-3.4.5/Objects/obmalloc.c:276:12
#2 0x4dc54d in _PyObject_Malloc /root/python-3.4.5/Objects/obmalloc.c:1314
#3 0x8a80e9 in _PyObject_GC_NewVar 
/root/python-3.4.5/Modules/gcmodule.c:1765:26

Direct leak of 3584 byte(s) in 16 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x636d59 in type_call /root/python-3.4.5/Objects/typeobject.c:883:11

Direct leak of 1472 byte(s) in 15 object(s) allocated from:
#0 0x4bc5ce in realloc (/root/python-3.4.5/python+0x4bc5ce)
#1 0x4da38a in PyMem_Realloc /root/python-3.4.5/Objects/obmalloc.c:308:12

Direct leak of 448 byte(s) in 2 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x836b0e in _Py_InitializeEx_Private 
/root/python-3.4.5/Python/pythonrun.c:418:14
#3 0x4e2e82 in Py_Main /root/python-3.4.5/Modules/main.c:654:5
#4 0x4d9c8e in main /root/python-3.4.5/./Modules/python.c:69:11

Direct leak of 264 byte(s) in 2 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x4e2e82 in Py_Main /root/python-3.4.5/Modules/main.c:654:5
#3 0x4d9c8e in main /root/python-3.4.5/./Modules/python.c:69:11

Direct leak of 224 byte(s) in 1 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x546820 in _PyExc_Init /root/python-3.4.5/Objects/exceptions.c:2509:5
#3 0x836b09 in _Py_InitializeEx_Private 
/root/python-3.4.5/Python/pythonrun.c:416:5
#4 0x4e2e82 in Py_Main /root/python-3.4.5/Modules/main.c:654:5
#5 0x4d9c8e in main /root/python-3.4.5/./Modules/python.c:69:11

Direct leak of 224 byte(s) in 1 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x5468ad in _PyExc_Init /root/python-3.4.5/Objects/exceptions.c:2510:5
#3 0x836b09 in _Py_InitializeEx_Private 
/root/python-3.4.5/Python/pythonrun.c:416:5
#4 0x4e2e82 in Py_Main /root/python-3.4.5/Modules/main.c:654:5
#5 0x4d9c8e in main /root/python-3.4.5/./Modules/python.c:69:11

Direct leak of 224 byte(s) in 1 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x5e7712 in PyModule_Create2 
/root/python-3.4.5/Objects/moduleobject.c:105:9
#3 0x4e2e82 in Py_Main /root/python-3.4.5/Modules/main.c:654:5
#4 0x4d9c8e in main /root/python-3.4.5/./Modules/python.c:69:11

Direct leak of 224 byte(s) in 1 object(s) allocated from:
#0 0x4bc2db in malloc (/root/python-3.4.5/python+0x4bc2db)
#1 0x4da307 in PyMem_Malloc /root/python-3.4.5/Objects/obmalloc.c:299:12
#2 0x545fdf in _PyExc_Init /root/python-3.4.5/Objects/exceptions.c:2494:5
#3 0x836b09 in _Py_InitializeEx_Private 
/root/python-3.4.5/Python/pythonrun.c:416:5
#4 0x4e2e82 in Py_Main /root/python-3.4.5/Modules/main.c:654:5
#5 0x4d9c8e in main 

Raw Data from Website

[adam . j . kerr]

Hi, 

I am hoping someone is able to help me. 

Is there a way to pull as much raw data from a website as possible. The webpage 
that I am looking for is as follows: 
http://www.homepriceguide.com.au/Research/ResearchSeeFullList.aspx?LocationType=LGA=QLD=632

The main variable that is important is the "632" at the end, by adjusting this 
it changes the postcodes. Each postcode contains a large amount of data. Is 
there a way this all able to be exported into an excel document? 

Any help with this would be amazing. 

Thank you. 

Adam
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[Ethan Furman]

On 08/19/2016 06:11 PM, Wildman via Python-list wrote:


Since I am fairly new to Python, I realize there is much that I
still don't know but I don't understand how Windows can have
reserved names on a Linux system.  What am I missing?


A PureWindowsPath (and PurePosixPath and PurePath) is a theoretical -- it's 
still, basically, a string and hasn't been checked against the file system to 
see if it actually exists (or to create it, etc.).

A WindowsPath (PosixPath) does actually have a tie-in to the actual file 
system, and can only be instantiated on Windows (Posix) systems.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

[issue27435] ctypes library loading and AIX - also for 2.7.X (and later)

[Martin Panter]

Martin Panter added the comment:

I had understood that changing find_library() would only be useful in 
combination with the automatic RTDL_MEMBER detection. If you want to mention 
lack of support for AIX in the documentation, that is okay by me. If you want 
to propose an alternative find_library() implementation _without_ affecting the 
CDLL behaviour, that may be okay too.

For the libuuid problem, for 2.7 I would suggest adding an explicit call to 
load libc.a(. . .) or whatever is appropriate.

Personally I don’t have a specific interest in AIX, but I am happy to commit 
patches for AIX if I think they are sensible. I think I mentioned before in the 
other bug that your changes seemed okay for the next Python version. The beta 
deadline for 3.6 is in a few weeks (PEP 494), although after that there is 
still 3.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[Roundup Robot]

Roundup Robot added the comment:

New changeset 620b2e554be4 by Victor Stinner in branch 'default':
Issue #27809: builtin___build_class__() uses fast call
https://hg.python.org/cpython/rev/620b2e554be4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27830] Add _PyObject_FastCallKeywords(): pass keyword arguments as (key, value) pairs

[STINNER Victor]

STINNER Victor added the comment:

Here is a first patch:

* Add _PyObject_FastCallKeywords(): main entry point
* Add _PyCFunction_FastCallKeywords(): implementation for C function, but 
currently it's just an empty skeleton
* Rename fast_function() to PyFunction_FastCallKeywords() and ajust its API

In this first implementation, _PyObject_FastCallKeywords() has a full 
implementation for Python function, otherwise it only uses 
_PyObject_FastCallDict() fast path if the function has no keyword argument. In 
the last resort, it creates a temporary tuple and maybe also a temporary dict, 
to call PyObject_Call().

To test the patch, I modified do_call() to call _PyObject_FastCallKeywords(). 
I'm not sure that it makes sense in term of performance at this stage.

Serhiy: You can use this first implementation to experimental for fast argument 
parser for keyword arguments passed as (key, value) pairs ;-)

--
keywords: +patch
Added file: http://bugs.python.org/file44191/fastcall_keywords.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27643] test_ctypes fails on AIX with xlc

[Martin Panter]

Martin Panter added the comment:

Now I am confused. In  we have

[Me] If your compiler does not support “signed short” bitfields, maybe we just 
have to accept that ctypes supports it even though the compiler doesn’t, and 
skip the test.

[Michael] Looks like this may be the approach: as signed int, signed short give 
the same message - it seems that xlc (and maybe Sun C) does not accept "short" 
as a bitfield type.

Apparently XLC doesn’t accept signed short bitfields for Michael, but does for 
Eric. What’s going on? Maybe different versions?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27824] update ConfigParser docs regarding in-line comments

[Joshua Haas]

Joshua Haas added the comment:

I'm not sure it's deprecated so much as moved. Python 3.5 includes an 
additional init parameter, so setting inline_comment_prefixes=';' allows inline 
comments even for values containing the ';' character, displaying the "expected 
behavior" described in my original bug report. They are, however, disabled by 
default, unlike in Python 2.7.

I also have a different proposal I think might be more clear:

Configuration files may include comments, prefixed by specific characters. 
Comments may appear on their own in an otherwise empty line using either '#' or 
';'. In addition, inline comments can be added to the end of a line after a ';' 
as long as the ';' is preceded by a whitespace character. Only the first ';' in 
a line is checked, so an inline comment cannot be added to lines that already 
contain any ';'s.

I'm also wondering if an example would be a good idea, as I don't think any of 
the examples on the doc page include comments:

# this is a comment
; this is also a comment
foo = bar ; inline comment
password = jgfw;as1 ; this will be parsed as part of 'password'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27810] Add METH_FASTCALL: new calling convention for C functions

[STINNER Victor]

STINNER Victor added the comment:

According to discussions in the issue #27809, require a Python dict leads to 
inefficient code because a temporary dict may be required. The issue #27830 
proposes to pass keyword arguments as (key, value) pairs.

It should be used for the new METH_FASTCALL calling convention.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27830] Add _PyObject_FastCallKeywords(): pass keyword arguments as (key, value) pairs

[STINNER Victor]

New submission from STINNER Victor:

In the issue #27809, I added support for keyword arguments to the function 
_PyObject_FastCallDict() (previously called _PyObject_FastCall). Keyword 
arguments are passed as a Python dictionary (dict).

_PyObject_FastCallDict() is efficient when keyword arguments are not used. But 
when keyword arguments are used, the creation of a temporary dictionary to pass 
keyword arguments can defeat the "fastcall" optimization for positional 
arguments.

See Serhiy Storchaka's comment:
http://bugs.python.org/issue27809#msg273367

Extract: "It may be more efficient to push values and names as continuous 
arrays (issue27213). PyArg_ParseTupleAndKeywords() accepts a tuple and a dict, 
but private function _PyArg_ParseTupleAndKeywordsFast() (issue27574) can be 
changed to accept positional and keyword arguments as continuous arrays: (int 
nargs, PyObject **args, int nkwargs, PyObject **kwnames, PyObject **kwargs)."

And Stefan Behnel's comment:
http://bugs.python.org/issue27809#msg273391

Serhiy proposed to pass keyword arguments as (key, value) pairs in C array. We 
can probably use the simple "stack" format used in Python/ceval.c:

   PyObject* _PyObject_FastCallKeywords(PyObject *func, PyObject **stack, int 
nargs, int nkwargs)

where nargs is number of positional arguments at the beginning of stack, and 
nkwargs is the number of (key, value) pairs at the end of stack.

This function would be the fundation for a new C calling convention: 
METH_FASTCALL, see the issue #27810.

--
components: Interpreter Core
messages: 273408
nosy: haypo, scoder, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Add _PyObject_FastCallKeywords(): pass keyword arguments as (key, value) 
pairs
type: performance
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[Roundup Robot]

Roundup Robot added the comment:

New changeset ef0110a52e24 by Victor Stinner in branch 'default':
PyEval_CallObjectWithKeywords() uses fast call with kwargs
https://hg.python.org/cpython/rev/ef0110a52e24

New changeset 5587d0dfab4c by Victor Stinner in branch 'default':
Issue #27809: Use _PyObject_FastCallDict()
https://hg.python.org/cpython/rev/5587d0dfab4c

New changeset 0c65a2089f00 by Victor Stinner in branch 'default':
Add _PyErr_CreateException()
https://hg.python.org/cpython/rev/0c65a2089f00

New changeset c53c532de995 by Victor Stinner in branch 'default':
Issue #27809: PyErr_SetImportError() uses fast call
https://hg.python.org/cpython/rev/c53c532de995

New changeset 614dd914c21e by Victor Stinner in branch 'default':
Issue #27809: tzinfo_reduce() uses fast call
https://hg.python.org/cpython/rev/614dd914c21e

New changeset f48ae71e8a8f by Victor Stinner in branch 'default':
Issue #27809: _csv: _call_dialect() uses fast call
https://hg.python.org/cpython/rev/f48ae71e8a8f

New changeset 60660890459f by Victor Stinner in branch 'default':
Issue #27809: methodcaller_reduce() uses fast call
https://hg.python.org/cpython/rev/60660890459f

New changeset bf2d88cf039a by Victor Stinner in branch 'default':
PyEval_CallObjectWithKeywords() doesn't inc/decref
https://hg.python.org/cpython/rev/bf2d88cf039a

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[STINNER Victor]

STINNER Victor added the comment:

Stefan Behnel: "I like the oneArg/noArg etc. macros. We use something similar 
in Cython."

Oh, nice :-) Since they macros are private, I pushed fastcall_macros.patch. We 
can still rework them later if needed.


"You can even use them to inline the METH_O and METH_NOARGS call cases 
(although I use inline functions for that in Cython)."

Hum, I'm not sure that it's worth it.


"I just took a quick look at the fastcall_kwargs-2.patch for now. It looks ok 
in general, but it also adds quite some special code for the dict-to-locals 
mapping. Is the keyword argument calling case really that important? I mean, it 
requires creating a dict and filling it, after all. People would rather avoid 
that in the first place if they care about performance. I wouldn't object, I'm 
merely asking..."

Hey, it's nothing new: Python 3.5 does the same in function_call()!? Oh in 
fact, function_call() code is very short compared to fastcall_kwargs-2.patch 
and it doesn't require to modify the hot code of Python/ceval.c! So I pushed a 
very small change (change 1aefb4c4a7b4) which adds support for keyword with the 
sample function_call() code.

Serhiy already told me that creating a dict is overkill... but please read 
again my message msg273370. We cannot break all APIs, we need a transition from 
the existing C APIs using Python dictionaries for keyword arguments.

I reworked my new fastcall APIs to keep the door open for a faster API to pass 
keyword arguments as pairs of (key, value) in an unique C array.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27820] Possible bug in smtplib when initial_response_ok=False

[Mario Colombo]

Mario Colombo added the comment:

Yes, this (or something similar) totally bit me, when for another unrelated 
reason 'AUTH PLAIN' authentication failed:

https://gist.github.com/macolo/bf2811c14d985d013dda0741bfd339e0

Python then tries auth_login, but doesn't send 'AUTH LOGIN' to the mail server. 
The second auth method also fails.

--
nosy: +Mario Colombo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: saving octet-stream png file

[Jon Ribbens]

On 2016-08-22, Larry Martell  wrote:
> On Mon, Aug 22, 2016 at 1:25 PM, Jon Ribbens  
> wrote:
>> On 2016-08-22, Larry Martell  wrote:
>>> (Pdb) type(request.POST[key])
>>>
>>> (Pdb) request.encoding = "iso-8859-1"
>>> (Pdb) type(request.POST[key])
>>> *** MultiValueDictKeyError:
>>> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
>>
>> Sounds like you should be filing a bug report with Django.
>
> Turns out that is expected behavior -- request.encoding clears
> existing GET/POST data. Not sure how it's useful to set that then.

Yeah it clears the cached parsed data, but when you fetch a value
afterwards it should automatically recalculate the data, not fail.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: saving octet-stream png file

[Larry Martell]

On Mon, Aug 22, 2016 at 1:25 PM, Jon Ribbens  wrote:
> On 2016-08-22, Larry Martell  wrote:
>> On Sun, Aug 21, 2016 at 5:24 PM, Jon Ribbens  
>> wrote:
>>> On 2016-08-19, Larry Martell  wrote:
 fd.write(request.POST[key])
>>>
>>> You could try:
>>>
>>>   request.encoding = "iso-8859-1"
>>>   fd.write(request.POST[key].encode("iso-8859-1"))
>>>
>>> It's hacky and nasty and there might be a better "official" method
>>> but I think it should work.
>>
>> For some reason that messes up the request structure:
>>
>> (Pdb) type(request.POST[key])
>>
>> (Pdb) request.encoding = "iso-8859-1"
>> (Pdb) type(request.POST[key])
>> *** MultiValueDictKeyError:
>> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
>
> Sounds like you should be filing a bug report with Django.

Turns out that is expected behavior -- request.encoding clears
existing GET/POST data. Not sure how it's useful to set that then.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[Roundup Robot]

Roundup Robot added the comment:

New changeset 1aefb4c4a7b4 by Victor Stinner in branch 'default':
_PyFunction_FastCallDict() supports keyword args
https://hg.python.org/cpython/rev/1aefb4c4a7b4

New changeset 7924eac912fe by Victor Stinner in branch 'default':
Issue #27809: Cleanup _PyEval_EvalCodeWithName()
https://hg.python.org/cpython/rev/7924eac912fe

New changeset 15eab21bf934 by Victor Stinner in branch 'default':
Issue #27809: Use _PyObject_FastCallDict()
https://hg.python.org/cpython/rev/15eab21bf934

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[Roundup Robot]

Roundup Robot added the comment:

New changeset 263334a652ac by Victor Stinner in branch 'default':
Rename _PyObject_FastCall() to _PyObject_FastCallDict()
https://hg.python.org/cpython/rev/263334a652ac

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Dynamically import specific names from a module vs importing full module

[Random832]

On Mon, Aug 22, 2016, at 16:21, Malcolm Greene wrote:
> Python 3.5: Is there a way to dynamically import specific names from a
> module vs importing the full module?
> 
> By dynamic I mean via some form of importlib machinery, eg. I'm looking
> for the dynamic "from  import " equivalent of "import
> "'s importlib.import_module.

You do know that "from  import " still loads the whole
module into sys.modules, right? And you're free to assign the result
from importlib.import_module to a local variable and delete it once
you've gotten the names you want out of it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dynamically import specific names from a module vs importing full module

[Ned Batchelder]

On Monday, August 22, 2016 at 4:22:09 PM UTC-4, Malcolm Greene wrote:
> Python 3.5: Is there a way to dynamically import specific names from a
> module vs importing the full module?
> 
> By dynamic I mean via some form of importlib machinery, eg. I'm looking
> for the dynamic "from  import " equivalent of "import
> "'s importlib.import_module.
> 
> Thank you,
> Malcolm

You can use:

the_mod = importlib.import_module("module")
a = the_mod.a
b = the_mod.b
# or
a = getattr(the_mod, 'a')
b = getattr(the_mod, 'b')

--Ned.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27128] Add _PyObject_FastCall()

[STINNER Victor]

STINNER Victor added the comment:

Ok, I see much better with concrete commits. I'm really happy that
Cython also benefits from these enhancements.

Note: handling keywords is likely to change quickly ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27660] Replace size_t with Py_ssize_t as the type of local variable in list_resize

[STINNER Victor]

STINNER Victor added the comment:

I reviewed list_resize_v2.patch.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Dynamically import specific names from a module vs importing full module

[Malcolm Greene]

Python 3.5: Is there a way to dynamically import specific names from a
module vs importing the full module?

By dynamic I mean via some form of importlib machinery, eg. I'm looking
for the dynamic "from  import " equivalent of "import
"'s importlib.import_module.

Thank you,
Malcolm
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27814] contextlib.suppress: Add whitelist argument

[Michael Büsch]

Michael Büsch added the comment:

>an implementation of the vastly *less* common pattern:

Ok, here are some numbers.
My codebase is about 32 kLOC.

$ git grep suppressAllExc |wc -l
20
$ git grep contextlib\\.suppress |wc -l
17

(suppressAllExc being my local version to suppress Exception, but exclude a few)

So _neither_ one is pretty common. And that's not surprising, as exceptions 
ought to be properly handled most of the time.
However the ratio between the two forms is about 1:1. So I don't agree that it 
would be "vastly *less* common".

But I do accept if such a thing is not wanted upsteam, Altough it basically is 
a twoliner. I'll simply have to ship my own (I'll have to do that anyway for 
older Python versions. meh :)

Thanks for the discussion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27826] Null-pointer dereference in tuplehash() function

[Raymond Hettinger]

Raymond Hettinger added the comment:

Closing as won't fix.  It is impractical to make marshal resilient against 
bytecode hacks and it is likewise impractical to put a NULL pointer check 
in-front of every dereference in the language.

--
resolution:  -> wont fix
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: saving octet-stream png file

[Wildman via Python-list]

On Mon, 22 Aug 2016 13:21:43 -0400, Larry Martell wrote:

> On Fri, Aug 19, 2016 at 4:51 PM, Lawrence D’Oliveiro
>  wrote:
>> On Saturday, August 20, 2016 at 6:03:53 AM UTC+12, Terry Reedy wrote:
>>>
>>> An 'octet' is a byte of 8 bits.
>>
>> Is there any other size of byte?
> 
> Many, many years ago, probably c. 1982 my Dad came into my house and
> saw a Byte Magazine laying on the coffee table. He asked "What is a
> byte?" I replied "Half a word." He then asked "What is the other half
> of the word?" I said "That is also a byte." He thought for a moment,
> then said "So the full word is 'byte byte'?"

LOL!  Did you explain to him that a full word could also be
'nibble nibble nibble nibble' or 'bit bit bit bit bit bit
bit bit bit bit bit bit bit bit bit bit'?

-- 
 GNU/Linux user #557453
The cow died so I don't need your bull!
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27660] Replace size_t with Py_ssize_t as the type of local variable in list_resize

[Raymond Hettinger]

Raymond Hettinger added the comment:

This is low priority.  I will look at it during the sprint.  As far as I know 
there is no bug here or performance problem; instead, the patch alters stable 
code and transfers responsibilities.

--
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Does This Scare You?

[Wildman via Python-list]

On Mon, 22 Aug 2016 17:27:13 +, Jon Ribbens wrote:

> On 2016-08-22, Chris Angelico  wrote:
>> I tried things like "con.txt" and it simply failed (no such file or
>> directory), without printing anything to the console.
> 
> I'm not sure how you got that to fail, but writing to "con.txt"
> certainly does write to the console in Windows 10 - I just tried it:
> 
> C:\>echo hello >con.txt
> hello

I got the same result with Windows 7, Vista, XP, 2000, NT4, 98SE
and 3.11.  Yes, I have a lot VM's although I don't have 8.x.  I
would, however, expect the same result.

On Linux a file is created named con.txt that contains hello/n
as expected.

-- 
 GNU/Linux user #557453
"The Constitution only gives people the right to
pursue happiness. You have to catch it yourself."
  -Benjamin Franklin
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ANN: pygene - genetic algorithms package

[saeed . fazayeli]

On Tuesday, December 6, 2005 at 11:10:56 AM UTC+3:30, aum wrote:
> Hi all,
> 
> I looked at a few genetic algorithms/genetic programming packages for
> Python, and found them somewhat convoluted, complicated and
> counter-intuitive to use.
> 
> So I've written a genetic algorithms package which I hope will be more
> approachable to beginners.
> 
> The first release of pygene is up at:
> http://www.freenet.org.nz/python/pygene
> 
> The package includes full api documentation, and an implementation of
> the travelling salesman problem, plus a couple of simpler cases.
> 
> -- 
> 
> Cheers
> aum

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: DED processing

[Chris Angelico]

On Tue, Aug 23, 2016 at 3:58 AM, Larry Martell  wrote:
> On Mon, Aug 22, 2016 at 1:52 PM, Gary Sublett  wrote:
>> I have to go out for a while, so for DED processing two options from
>> my end:
>>
>> 1. Process as you all have been in the past for now.  If you all do
>> this, the records that have not been mailed prior to the latest list
>> are contained in a MailManage Job name DED_master.  If you chose to
>> process as in the past, these records should be exported to a
>> spreadsheet and added to whatever new file you have for today.
>>
>> 2. I can come in tomorrow morning at about 9:00 and walk Greg thru
>> the way I process the DED job.
>>
>> Let me know.
>
> I think you should let Greg know. What if you get hit by a bus, or win
> the lottery?

DED Processing Disclaimer: This e-mail (and files transmitted with it)
is intended only for the person or entity to which it is addressed. If
you are not the intended recipient, then you have received this e-mail
by mistake and any use, dissemination, forwarding, printing or copying
of this e-mail and its file attachments is prohibited; please print
this email out on minimum 100 GSM paper, cross-cut shred it, burn the
remains, and send the ashes back to the sender. Failure to do so will
result in the internet police being informed of your IP address,
street address, ICBM address, and t-shirt size.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27643] test_ctypes fails on AIX with xlc

[Eric N. Vander Weele]

Changes by Eric N. Vander Weele :


Removed file: http://bugs.python.org/file44189/ctypes_test_sign_bitfields.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27643] test_ctypes fails on AIX with xlc

[Eric N. Vander Weele]

Eric N. Vander Weele added the comment:

> I believe that there is a specific reason that M, N, O, P, Q, R and S 
are "short". [...]

Oops - this was an oversight when I created the patch.  I just uploaded 
ctypes_test_sign_bitfields_2.diff, which is what I originally intended.

--
Added file: http://bugs.python.org/file44190/ctypes_test_sign_bitfields_2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27794] setattr a read-only property; the AttributeError should show the attribute that failed

[Antti Haapala]

Antti Haapala added the comment:

I've got one idea about how to implement this, but it would require adding a 
new flag field to PyExc_AttributeError type.

This flag, if set, would tell that the AttributeError in question was raised in 
C descriptor code or under similar circumstances, and that the attribute name 
was not known, and thus it is OK for setattr/delattr and attribute lookups to 
append ": attributename" to the end of the message, then clear the flag; then  
all those places that raise AttributeError in __get__, __set__, __del__ would 
just need to set this flag.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12713] argparse: allow abbreviation of sub commands by users

[Vinay Sajip]

Vinay Sajip added the comment:

Mean culpa - I have  jumped the gun on this. Sorry to all for the 
inconvenience. I have just got back after being away for a few days, and will 
revert the change shortly, if no one beats me to it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12713] argparse: allow abbreviation of sub commands by users

[Vinay Sajip]

Vinay Sajip added the comment:

*Mea culpa. Autocorrect :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27643] test_ctypes fails on AIX with xlc

[Michael Felt]

Michael Felt added the comment:

On 22-Aug-16 19:43, Eric N. Vander Weele wrote:
> Eric N. Vander Weele added the comment:
>
> I came across this issue while researching where to post my patch (having 
> come across this while building Python 2.7 & 3.x on AIX via xlc).
>
>> In general in C, if a bit-field has type “int” without a signed or unsigned 
>> qualifier, it is up to the implementation which mode is chosen.
> Unfortunately, the ISO C standard leaves it up to the compiler to decide 
> whether to default to 'signed' or 'unsigned' for non-qualified bit-field 
> declarations.  gcc defaults to signed 
> (https://gcc.gnu.org/onlinedocs/gcc-3.3.6/gcc/Non_002dbugs.html); however, 
> xlc defaults to unsigned 
> (https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.3/com.ibm.xlc1313.aix.doc/compiler_ref/opt_bitfields.html).
>
>> [...] having bitfields working regardless of the compiler should be 
>> preferred.
> However, ctypes_test assumes an unqualified bit-field will be signed.  To 
> achieve bit-fields working regardless of the compiler, declarations must 
> explicitly qualifying the sign for bit-fields.  This makes the intent of what 
> is expected explicit and avoids implementation-defined behavior that will 
> differ from one compiler to the next.
>
> With patch ctypes_test_sign_bitfields.diff provided, I have verified 
> ctypes_test passes on Python 2.7, Python 3.4, and Python 3.5 (on AIX & 
> Solaris).  If need be, I'm happy to provide before-&-after output of 
> ctypes_test with & without the patch applied (or verification from others 
> would be greatly appreciated).
>
> --
> Added file: http://bugs.python.org/file44189/ctypes_test_sign_bitfields.diff
I believe that there is a specific reason that M, N, O, P, Q, R and S 
are "short". For AIX/xlc at least "signed" is equivalent to "signed int" 
and not the same size as "unsigned short".

> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27435] ctypes library loading and AIX - also for 2.7.X (and later)

[Michael Felt]

Michael Felt added the comment:

On 22-Aug-16 04:04, Martin Panter wrote:
> Martin Panter added the comment:
>
> The ctypes tests all seem to be protected with code that checks for None, and 
> explicitly skip the test in that case. The skip message should be visible 
> when you run the test in verbose mode.
So at least test/test_something.py run cleanly - but why accept code 
that always returns None - for find_library().

Rather than "hard-code" proper strings in a test - shouldn't the test be 
demonstrating the "failure" of find_library() to return a suitable 
response - when it should not be None.

For example, on AIX the IBM provided /usr/lib/libm.a only has static 
members. So, what has always been wrong (and noone noticed, or is it 
noone cared) the "built-in" test in util.py has, and does, always fail:

root@x064:[/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes]../../python 
util.py
None
None
None
Traceback (most recent call last):
   File "util.py", line 271, in 
 test()
   File "util.py", line 266, in test
 print cdll.LoadLibrary("libm.so")
   File 
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 
440, in LoadLibrary
 return self._dlltype(name)
   File 
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 
362, in __init__
 self._handle = _dlopen(self._name, mode)
OSError:0509-022 Cannot load module .
 0509-026 System error: A file or directory in the path name 
does not exist.

The documentation says:

The exact functionality is system dependent.

On Linux, |find_library()| tries to run external programs 
(|/sbin/ldconfig|, |gcc|, and |objdump|) to find the library file. It 
returns the filename of the library file.
...
On OS X, |find_library()| tries several predefined naming schemes and 
paths to locate the library, and returns a full pathname if successful:
...
On Windows, |find_library()| searches along the system search path, and 
returns the full pathname, but since there is no predefined naming 
scheme a call like |find_library("c")| will fail and return |None|.

Are you saying the documentation should be updated to read:
On AIX, find_library tries to use the Linux code which generally fails 
and will return None.

And the closing bit:
If wrapping a shared library with |ctypes| 
, it /may/ 
be better to determine the shared library name at development time, and 
hardcode that into the wrapper module instead of using |find_library()| 
to locate the library at runtime (so forget about developing code for 
multiple platforms - find_library() cannot give you that support.
p.s. On AIX, you must use a hard-coded name to ctypes.CDLL() - and also 
add an extra flag - because even though we said "the exact functionality 
is system dependent" we do not permit new systems to add corrections.

I apologize that my frustration is more than peeking through. But I am 
frustrated. find_library(), unless I completely misunderstand the 
documentation, is a helper function to releave the programmer of needing 
to know the specifics of multiple platforms. My frustration is 
heightened by what I read about the issues with "uuid" - which also 
calls find_library() - so again, on AIX it always returns None -- a) 
find_library is broken; b) there is no libuuid.so or libuuid.a on AIX 
(and c) 4 of the 5 programs also called do not exist on AIX, and the one 
that does generates a different string than is compared to).

I understand - AIX has not been a major (user) platform for python. I am 
amazed that noone from IBM (AIX) development has never shown more interest.
Disclaimer: yes, I work for IBM - but am not a developer. Customers do 
ask me about "why is python so old, why does it not work with XXX" - and 
only because I am inclined to "get to the bottom" did I even start on this.

If the root cause is: "IBM showed no interest X years ago and we, 
python, have no interest now in IBM or AIX" then please just say so - 
and I shall do other things with my time.

In closing - I thought python was multi-platform and I had hoped to be 
of some help in improving python integration with the platform AIX. 
Please, one way or the other, help me lessen my frustration.

Thank you for your time and patience with my rant!

> We could avoid skipping these tests in 2.7 by adding a special case for AIX 
> that loads the library via CDLL("libc.a(shr[_64].o)", RTLD_MEMBER). Like an 
> extension of the special cases already at 
> ,
>  but we have to set the RTLD_ flag as well as the library name.
>
> --
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

Re: Does This Scare You?

[Jon Ribbens]

On 2016-08-22, Steve D'Aprano  wrote:
> I'm not really sure what the question is -- we've established that there's a
> bug in the non-Windows implementation that tries to emulate Window's
> behaviour. What else is there to argue about?

It doesn't seem to be "the non-Windows implementation", it seems to be
"the implementation".

> - Does anyone wish to argue that Python shouldn't provide
>   PureWindowsPath.is_reserved on non-Windows systems? For what reason?

It shouldn't provide it at all unless it works. The reason I'm putting
it that way is that making it work may be a very great deal of effort
(What is the actual full list of special filenames? Does it vary
between supported Windows versions? How can anyone tell test that
the function is correct?)

> - Is anyone still arguing that there's a new security vulnerability
>   here because of the pathlib functions? If so, how do you see this
>   attack working? (Existing filename-based attacks are not new.)

Already answered.

> I don't see what the issue is. Eryksun found a bug in pathlib, well done. (I
> mean that, I'm not being sarcastic.) I still don't understand why Lawrence
> posed his question in the first place.

Presumably because of the security implications as described.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[Stefan Behnel]

Stefan Behnel added the comment:

I just took a quick look at the fastcall_kwargs-2.patch for now. It looks ok in 
general, but it also adds quite some special code for the dict-to-locals 
mapping. Is the keyword argument calling case really that important? I mean, it 
requires creating a dict and filling it, after all. People would rather avoid 
that in the first place if they care about performance. I wouldn't object, I'm 
merely asking...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Does This Scare You?

[Steve D'Aprano]

On Tue, 23 Aug 2016 03:13 am, eryk sun wrote:

> But if they open files
> like "C:\Users\JoeUser\Documents\Nul.20160822.doc", I want to make
> sure they know that they just asked to save to "\\.\NUL". It's not a
> common problem. I just find the system's behavior abhorrent. I'd like
> to have a manifest setting that opts the process out of this stupid
> DOS legacy behavior.


Aren't we getting further and further away from the original topic?

I don't think this is about Window's support for legacy behaviour from
ancient DOS days. Yes, it would be nice if Windows dropped support for
these horrid magic file names, but that's nothing to do with Python.

I'm not really sure what the question is -- we've established that there's a
bug in the non-Windows implementation that tries to emulate Window's
behaviour. What else is there to argue about?

- Does anyone wish to argue that Python shouldn't provide
  PureWindowsPath.is_reserved on non-Windows systems? For what reason?

- Is anyone still arguing that there's a new security vulnerability
  here because of the pathlib functions? If so, how do you see this
  attack working? (Existing filename-based attacks are not new.)

I don't see what the issue is. Eryksun found a bug in pathlib, well done. (I
mean that, I'm not being sarcastic.) I still don't understand why Lawrence
posed his question in the first place.





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Something wrong with the PIP, lots of friend have the same problem

[Wentao Liang]

even though I try to upgrade PIP[cid:beb9e7c5-7d98-4506-ba67-9e67890edec1]


From: Wentao Liang
Sent: Sunday, 21 August 2016 4:35:31 PM
To: python-list@python.org
Subject: Something wrong with the PIP, lots of friend have the same problem


[cid:1c20248f-a678-40c0-9563-da4f8a1ef3bd]
Is it the problem from PIP itself, or I used it in a wrong way? []

Looking forwards to your reply!

Thank you

Wentao Liang
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[Stefan Behnel]

Stefan Behnel added the comment:

I like the oneArg/noArg etc. macros. We use something similar in Cython. You 
can even use them to inline the METH_O and METH_NOARGS call cases (although I 
use inline functions for that in Cython).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: DED processing

[Larry Martell]

On Mon, Aug 22, 2016 at 1:52 PM, Gary Sublett  wrote:
> I have to go out for a while, so for DED processing two options from
> my end:
>
> 1. Process as you all have been in the past for now.  If you all do
> this, the records that have not been mailed prior to the latest list
> are contained in a MailManage Job name DED_master.  If you chose to
> process as in the past, these records should be exported to a
> spreadsheet and added to whatever new file you have for today.
>
> 2. I can come in tomorrow morning at about 9:00 and walk Greg thru
> the way I process the DED job.
>
> Let me know.

I think you should let Greg know. What if you get hit by a bus, or win
the lottery?
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue25825] AIX shared library extension modules installation broken

[Eric N. Vander Weele]

Eric N. Vander Weele added the comment:

> Patch1 looks fine to me, though I will have to rely on you people to verify 
> that it does what it’s supposed to. Do you want me to commit it straight 
> away, or wait for your follow-up patch?

Independently, I have created a similar patch as well and can also verify that 
it does what it's suppose to do.

> The second of two patches. This patch changes the definition of LDSHARED for 
> AIX in configure to reference the matching installed location as defined in 
> Makefile.pre.in by Patch1. [...]

I can also confirm that Patch2 is necessary as well and does what it's suppose 
to do, having creating this patch independently as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

DED processing

[Gary Sublett]

I have to go out for a while, so for DED processing two options from
my end:

1. Process as you all have been in the past for now.  If you all do
this, the records that have not been mailed prior to the latest list
are contained in a MailManage Job name DED_master.  If you chose to
process as in the past, these records should be exported to a
spreadsheet and added to whatever new file you have for today.

2. I can come in tomorrow morning at about 9:00 and walk Greg thru
the way I process the DED job. 

Let me know.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27128] Add _PyObject_FastCall()

[Stefan Behnel]

Stefan Behnel added the comment:

> What do you mean by "I copied your (no-kwargs) implementation"?

I copied what you committed into CPython for _PyFunction_FastCall():

https://github.com/cython/cython/commit/8f3d3bd199a3d7f2a9fdfec0af57145b3ab363ca

and then enabled its usage in a couple of places:

https://github.com/cython/cython/commit/a3cfec8f7bd6d585831dd6669f6dad5f88303c71

especially for all function/method calls that we generate for user code:

https://github.com/cython/cython/commit/a51df339f395634f57b77e3ec13cecb3a28a5462

Note that PyMethod objects get unpacked into function+self right before the 
PyFunction_Check(), so the tuple avoidance optimisation also applies to Python 
method calls.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Chris Angelico]

On Tue, Aug 23, 2016 at 3:32 AM, Tim Chase
 wrote:
> On 2016-08-23 02:20, Chris Angelico wrote:
>> It generally will (or rather, only if the file has one of a
>> particular set of extensions). Automatic thumbnailing is usually
>> done only for certain file names. I don't know of anything that
>> opens every single file to see if it has a JFIF signature (etc for
>> PNG and whatever other types).
>
> How about a web server that opens arbitrary files.  Compare any of
>
> https://technet.microsoft.com/en-us/library/nonexistent.aspx
> https://technet.microsoft.com/en-us/library/doesnotexist.aspx
> https://technet.microsoft.com/en-us/library/asdf.aspx
>
> vs
>
> https://technet.microsoft.com/en-us/library/con.aspx
> https://technet.microsoft.com/en-us/library/lpt1.aspx
> https://technet.microsoft.com/en-us/library/com1.aspx
> https://technet.microsoft.com/en-us/library/nul.aspx
>
> This is FREAKING MICROSOFT and it breaks things.  It's not like
> anybody would open arbitrarily-named files...

Oh, brilliant. Brilliant brilliant brilliant.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27643] test_ctypes fails on AIX with xlc

[Eric N. Vander Weele]

Eric N. Vander Weele added the comment:

I came across this issue while researching where to post my patch (having come 
across this while building Python 2.7 & 3.x on AIX via xlc).

> In general in C, if a bit-field has type “int” without a signed or unsigned 
> qualifier, it is up to the implementation which mode is chosen.

Unfortunately, the ISO C standard leaves it up to the compiler to decide 
whether to default to 'signed' or 'unsigned' for non-qualified bit-field 
declarations.  gcc defaults to signed 
(https://gcc.gnu.org/onlinedocs/gcc-3.3.6/gcc/Non_002dbugs.html); however, xlc 
defaults to unsigned 
(https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.3/com.ibm.xlc1313.aix.doc/compiler_ref/opt_bitfields.html).

> [...] having bitfields working regardless of the compiler should be preferred.

However, ctypes_test assumes an unqualified bit-field will be signed.  To 
achieve bit-fields working regardless of the compiler, declarations must 
explicitly qualifying the sign for bit-fields.  This makes the intent of what 
is expected explicit and avoids implementation-defined behavior that will 
differ from one compiler to the next.

With patch ctypes_test_sign_bitfields.diff provided, I have verified 
ctypes_test passes on Python 2.7, Python 3.4, and Python 3.5 (on AIX & 
Solaris).  If need be, I'm happy to provide before-&-after output of 
ctypes_test with & without the patch applied (or verification from others would 
be greatly appreciated).

--
Added file: http://bugs.python.org/file44189/ctypes_test_sign_bitfields.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27794] setattr a read-only property; the AttributeError should show the attribute that failed

[Xiang Zhang]

Xiang Zhang added the comment:

One solution I can think of is alter the constructor of property() and add an 
optional name attribute to it. If users really care about the exception 
message, they can set the attribute to the property's name. If they don't care, 
everything remains the same as now. This is just like the doc parameter of 
property() right now and the first parameter of type() when it is used to 
create a class dynamically. This is just a suggestion, I am not sure it's worth 
to do it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Does This Scare You?

[Jon Ribbens]

On 2016-08-22, Steve D'Aprano  wrote:
> On Mon, 22 Aug 2016 09:50 pm, Jon Ribbens wrote:
>> I don't know what purpose you are envisaging this function being used
>> for, but the only one I can think of is input sanitisation. e.g. a web
>> form where you receive a file from the Internet and store it somewhere,
>> and you want to use the filename given to you rather than choose your
>> own randomly-generated one.
>> 
>> Under Unix all you need to do is check for the filename starting with
>> "." or containing "/." (or "/", depending on your requirements).
>> Under Windows you would use this function, which apparently doesn't
>> work, hence: security hole.
>
> That's backwards: it works under Windows, but is buggy under non-Windows.

I'm not sure what you're getting at there, but you appear to be wrong:

C:\>python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathlib
>>> pathlib.WindowsPath("con .txt").is_reserved()
False

> For starters, even on Unix I probably wouldn't want to
> use "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\.txt" as a
> file name, even if it is technically legal.

That might be your personal preference, but it's not a security hole
per se.

> No, not at all. I can't assume that just because a file name is legal that I
> can write to it. Even after sanitising the name (or deciding I don't need
> to bother) I still need to be prepared to catch I/O errors when writing the
> file -- including attempts to write to reserved file names.

As mentioned by others, this is nothing to do with lack of error-checking.

> Technically that could be called a DoS attack, but the same could be said
> for ANY bug that raises an exception. I think that it takes more than that
> to be a security bug.

It doesn't raise an exception. It interacts with the hardware of the
server.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27794] setattr a read-only property; the AttributeError should show the attribute that failed

[R. David Murray]

R. David Murray added the comment:

Not worth it.  It would feel like boilerplate, and the situation where you want 
the information is almost invariably going to be one you didn't anticipate and 
so didn't "bother" with the name.  Either that or you always have to do it, and 
the elegance of your python code takes a hit not commensurate with the benefit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Does This Scare You?

[Tim Chase]

On 2016-08-22 22:39, Chris Angelico wrote:
> Nope. On Windows, you would try/except it. There are myriad other
> ways something could fail, and the only correct action is to
> attempt it. Most of the reserved names will simply give an error;

The problem is that when opening such a pseudo-file, you can get
unexpected behavior.  In the Unix world, we're used to
files-that-aren-t-files (such as things in /dev ).  But a lot of
Windows developers don't handle these cases, and so opening something
like COM1 can end up hanging a program indefinitely instead of
actually returning either an error or a file-handle.  If you have a
web-server running on Windows and can manage to coerce a file to have
such a name, you might be able to hang the web-server process while
it tries to read from (or write to) the serial port.  And poof,
near-instant DoS.

-tkc




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Tim Chase]

On 2016-08-23 02:20, Chris Angelico wrote:
> It generally will (or rather, only if the file has one of a
> particular set of extensions). Automatic thumbnailing is usually
> done only for certain file names. I don't know of anything that
> opens every single file to see if it has a JFIF signature (etc for
> PNG and whatever other types).

How about a web server that opens arbitrary files.  Compare any of

https://technet.microsoft.com/en-us/library/nonexistent.aspx
https://technet.microsoft.com/en-us/library/doesnotexist.aspx
https://technet.microsoft.com/en-us/library/asdf.aspx

vs

https://technet.microsoft.com/en-us/library/con.aspx
https://technet.microsoft.com/en-us/library/lpt1.aspx
https://technet.microsoft.com/en-us/library/com1.aspx
https://technet.microsoft.com/en-us/library/nul.aspx

This is FREAKING MICROSOFT and it breaks things.  It's not like
anybody would open arbitrarily-named files...

-tkc


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: saving octet-stream png file

[Jon Ribbens]

On 2016-08-22, Larry Martell  wrote:
> On Sun, Aug 21, 2016 at 5:24 PM, Jon Ribbens  
> wrote:
>> On 2016-08-19, Larry Martell  wrote:
>>> fd.write(request.POST[key])
>>
>> You could try:
>>
>>   request.encoding = "iso-8859-1"
>>   fd.write(request.POST[key].encode("iso-8859-1"))
>>
>> It's hacky and nasty and there might be a better "official" method
>> but I think it should work.
>
> For some reason that messes up the request structure:
>
> (Pdb) type(request.POST[key])
>
> (Pdb) request.encoding = "iso-8859-1"
> (Pdb) type(request.POST[key])
> *** MultiValueDictKeyError:
> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"

Sounds like you should be filing a bug report with Django.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[Jon Ribbens]

On 2016-08-22, Chris Angelico  wrote:
> I tried things like "con.txt" and it simply failed (no such file or
> directory), without printing anything to the console.

I'm not sure how you got that to fail, but writing to "con.txt"
certainly does write to the console in Windows 10 - I just tried it:

C:\>echo hello >con.txt
hello
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[Chris Angelico]

On Tue, Aug 23, 2016 at 3:13 AM, eryk sun  wrote:
> On Mon, Aug 22, 2016 at 4:18 PM, Chris Angelico  wrote:
>>
>>> The CON device should work if the process is attached to a console
>>> (i.e. a conhost.exe instance).
>>
>> No, I used Pike (to avoid any specifically-Python issues or
>> protections) running in a console. Attempting to write to "Logs/con"
>> wrote to the console, so I know the console device is active.
>> Attempting to write to "Logs/con.txt" failed as described.
>
> What version of Windows is this? If it's Windows 7 I'll have to check
> that later. If "Logs" is an existing directory, then both "Logs/con"
> and "Logs/con.txt" should refer to the console. If "Logs" doesn't
> exist, then both should fail. Virtual DOS devices only exist in
> existing directories.

Yes, it was Windows 7 (running in a VM under Debian Jessie, though I
doubt that makes any difference). The Logs directory did exist (that's
why I used that otherwise-odd choice of name).

>> Windows won't let you put a colon or question mark in a file name, for 
>> instance,
>> which annoys my brother no end when I give him files like
>> "What's Up, Doc?.mkv" or "Operation: Rabbit.mkv")
>
> A colon is reserved for NTFS streams, so it can silently bite you. If
> you get an error for "Operation: Rabbit.mkv", you're probably using
> FAT32. On NTFS that creates a file named "Operation" with a $DATA
> stream named " Rabbit.mkv".

I think it was FAT32 that we were using - it was a USB stick for
sharing files between my Linux systems and his Windows. Linux was
quite happy to put files into that FS that Windows refused to touch.
(Also, due to some renamings, I ended up giving him two copies of a
file that differed only in case - something like "Name The File
Something.mkv" and "Name the File Something.mkv" - and Windows took
issue with that too.)

> As to question mark and other wildcard characters, those are all
> unambiguously handled as exceptions. No Microsoft filesystem allows
> them because wildcard matching is baked into the system, such as the
> FileName parameter of NtQueryDirectoryFile when listing the contents
> of a directory. I guess no one on the NT I/O team wanted to bother
> with the possibility of escaping these wildcards since they weren't
> allowed in DOS filenames.

Yeah. They're nice straight-forward errors, as long as you're running
on Windows. Detecting that on POSIX platforms and avoiding those names
is why we have PureWindowsPath. If it doesn't currently have a quick
little "is_valid", it could possibly benefit from one - PurePosixPath
would simply "return '\0' not in name", while PureWindowsPath can do
the more extensive checks with wildcard characters and stuff.

>> or over-long names or anything like that.
>
> Good news. In Windows 10, systems and applications can opt in to using
> long paths with up to about 32760 characters. Python 3.6 will be
> opting in.
>
> [1]: https://msdn.microsoft.com/en-us/library/ff546850

Great. So just as soon as all previous Windowses die...

Remind me how many people are still using Python 3.4 because they're
still on Windows XP?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: saving octet-stream png file

[Larry Martell]

On Fri, Aug 19, 2016 at 4:51 PM, Lawrence D’Oliveiro
 wrote:
> On Saturday, August 20, 2016 at 6:03:53 AM UTC+12, Terry Reedy wrote:
>>
>> An 'octet' is a byte of 8 bits.
>
> Is there any other size of byte?

Many, many years ago, probably c. 1982 my Dad came into my house and
saw a Byte Magazine laying on the coffee table. He asked "What is a
byte?" I replied "Half a word." He then asked "What is the other half
of the word?" I said "That is also a byte." He thought for a moment,
then said "So the full word is 'byte byte'?"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[eryk sun]

On Mon, Aug 22, 2016 at 4:18 PM, Chris Angelico <ros...@gmail.com> wrote:
>
>> The CON device should work if the process is attached to a console
>> (i.e. a conhost.exe instance).
>
> No, I used Pike (to avoid any specifically-Python issues or
> protections) running in a console. Attempting to write to "Logs/con"
> wrote to the console, so I know the console device is active.
> Attempting to write to "Logs/con.txt" failed as described.

What version of Windows is this? If it's Windows 7 I'll have to check
that later. If "Logs" is an existing directory, then both "Logs/con"
and "Logs/con.txt" should refer to the console. If "Logs" doesn't
exist, then both should fail. Virtual DOS devices only exist in
existing directories.

>> > I wouldn't accept file names from untrusted sources on *any* system
>>
>> There are still desktop applications that ask users to name their files.
> If the user owns the computer, s/he should be allowed to attempt any
> name, and there'd simply be some that fail - same as any other invalid
> characters

Silently writing a file to a (possibly hidden) console or NUL device
is not the same thing as failing with an error. If users explicitly
open "NUL" or "\\.\NUL", I can detect that, and I have no problem with
it (with some reservations about the former). But if they open files
like "C:\Users\JoeUser\Documents\Nul.20160822.doc", I want to make
sure they know that they just asked to save to "\\.\NUL". It's not a
common problem. I just find the system's behavior abhorrent. I'd like
to have a manifest setting that opts the process out of this stupid
DOS legacy behavior.

> Windows won't let you put a colon or question mark in a file name, for 
> instance,
> which annoys my brother no end when I give him files like
> "What's Up, Doc?.mkv" or "Operation: Rabbit.mkv")

A colon is reserved for NTFS streams, so it can silently bite you. If
you get an error for "Operation: Rabbit.mkv", you're probably using
FAT32. On NTFS that creates a file named "Operation" with a $DATA
stream named " Rabbit.mkv".

Every NTFS file has at least the anonymous stream, e.g.
"filename::$DATA". Every NTFS directory has at least the $I30 stream,
e.g. "dirname:$I30:$INDEX_ALLOCATION". Directories can also have named
$DATA streams.

As to question mark and other wildcard characters, those are all
unambiguously handled as exceptions. No Microsoft filesystem allows
them because wildcard matching is baked into the system, such as the
FileName parameter of NtQueryDirectoryFile when listing the contents
of a directory. I guess no one on the NT I/O team wanted to bother
with the possibility of escaping these wildcards since they weren't
allowed in DOS filenames.

FsRtlIsNameInExpression [1] is the kernel-mode filesystem function
that implements wildcard matching. Note that DOS wildcard semantics
are implemented by DOS_STAR (<), DOS_QM (>), and DOS_DOT (").
FindFirstFile has to convert DOS wildcard patterns to NT semantics
before calling NtQueryDirectoryFile. It's not as simple as
substituting the corresponding DOS_ wildcard.

Pipe and ASCII control characters are also reserved, except not in
NTFS stream names. Reserving pipe is just a DOS legacy. I don't think
it's used for anything internally.

> or over-long names or anything like that.

Good news. In Windows 10, systems and applications can opt in to using
long paths with up to about 32760 characters. Python 3.6 will be
opting in.

[1]: https://msdn.microsoft.com/en-us/library/ff546850
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue26331] PEP 515: Tokenizer: allow underscores for grouping in numeric literals

[Xiang Zhang]

Xiang Zhang added the comment:

Hi Georg, I left several comments on Rietveld. Hope it helps.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: saving octet-stream png file

[Larry Martell]

On Mon, Aug 22, 2016 at 10:36 AM, Larry Martell  wrote:
> On Fri, Aug 19, 2016 at 4:24 PM, Chris Kaynor  
> wrote:
>> On Fri, Aug 19, 2016 at 12:00 PM, Larry Martell 
>> wrote:
>>
>>> On Fri, Aug 19, 2016 at 1:24 PM, Chris Angelico  wrote:
>>> > On Sat, Aug 20, 2016 at 3:10 AM, Larry Martell 
>>> wrote:
>>> >> I have some python code (part of a django app) that processes a
>>> >> request that contains a png file. The request is send with
>>> >> content_type = 'application/octet-stream'
>>> >>
>>> >> In the python code I want to write this data to a file and still have
>>> >> it still be a valid png file.
>>> >>
>>> >> The data I get looks like this:
>>> >>
>>> >> u'\ufffdPNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\ufffd\
>>> x00\x00\x01\ufffd
>>> >> ..'
>>> >>
>>> >> If I try and write that to a file it fails with a UnicodeEncodeError.
>>> >> If I write it with encode('utf8') it writes the file, but then it's no
>>> >> longer a valid png file.
>>> >>
>>> >> Anyone know how I can do this?
>>> >
>>> > At that point, you've already lost information. Each U+FFFD (shown as
>>> > "\ufffd" above) is a marker saying "a byte here was not valid UTF-8"
>>> > (or whatever was being used). Something somewhere took the .png file's
>>> > bytes and tried to interpret them as text, which they're not.
>>> >
>>> > What sent you that data? How did you receive it?
>>>
>>> The request is sent by a client app written in C++ with Qt. It's
>>> received by a django based server. I am trying to port a falcon server
>>> to django. The falcon server code did this:
>>>
>>> form = cgi.FieldStorage(fp=req.stream, environ=req.env)
>>>
>>> and then wrote the png like this:
>>>
>>> fd.write(form[key].file.read())
>>>
>>> Whereas in the django server I am doing:
>>>
>>> fd.write(request.POST[key])
>>>
>>> I've never used the cgi module. I guess I can try that. I've written a
>>> lot with django but never had to receive a PNG file.
>>>
>>>
>> I don't know Django, however a quick search makes it seem like you might
>> need to use request.FILES[key] (1) rather than request.POST[key]. You may
>> also be able to use request.POST if you set request.encoding first (2). If
>> both of those fail, you may need to use request.body and parse the HTTP
>> form data manually, though I'd imagine there is an easier way.
>>
>> [1]
>> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.FILES
>>
>> [2]
>> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.encoding
>
> Thanks for the reply. When I get the request, request.FILES is empty.
> Yet the content type is multipart/form-data and the method is POST:
>
> (Pdb) print request.META['CONTENT_TYPE']
> multipart/form-data;
> boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="
>
> (Pdb) print request.META['REQUEST_METHOD']
> POST
>
> (Pdb) print request.FILES
> 
>
> Tried setting request.encoding, but that messes up the request structure:
>
> (Pdb) type(request.POST[key])
> 
> (Pdb) request.encoding = "iso-8859-1"
> (Pdb) type(request.POST[key])
> *** MultiValueDictKeyError:
> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"


Thanks to everyone for they replied. I solved this by changing the
client to set the file and filename fields each part of the multipart
and then I was able it iterate through request.FILES and successfully
write the files as PNG.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman

[Chris Angelico]

On Tue, Aug 23, 2016 at 2:08 AM, Marko Rauhamaa  wrote:
> Random832 :
>
>> On Mon, Aug 22, 2016, at 11:40, Chris Angelico wrote:
>>> Windows has some other issues, including that arbitrary files can
>>> become executable very easily (eg if %PATHEXT% includes its file
>>> extension), and since the current directory is always at the
>>> beginning of your path, this can easily turn into a remote code
>>> execution exploit.
>>
>> I didn't include dot in my example whitelist, and there's no mechanism
>> for an attacker to add random extensions to your PATHEXT.
>
> Years back, my FTP server was hacked by exploiting a buffer overflow.
> The anonymous input directory contained a very long filename that
> apparently contained some valid x86 code.
>
> Did you vet your whitelist so it couldn't possibly be interpreted by the
> CPU as meaningful instructions?

Step 1: Don't have buffers.
Step 2: Profit!

Anyone who's using fixed-sized buffers for application-level code
deserves to be exploited. A program designed to be accessed via the
internet is never (well, hardly ever) going to need so much
performance that it can't afford to be written in a high level
language - it's going to spend most of its time waiting for the
network. The rare exceptions (*maybe* DNS, but even there, I'd be
quite happy to replace my DNS server with one written in Pike, if
BIND9 ever becomes a major threat vector) should be monitored closely
- preferably statically checked with something like Coverity - because
they're remotely-accessible and thus a major risk.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[eryk sun]

On Mon, Aug 22, 2016 at 3:40 PM, Chris Angelico  wrote:
> Windows has some other issues, including that arbitrary files can
> become executable very easily (eg if %PATHEXT% includes its file
> extension),

cmd uses PATHEXT to augment its search by appending each extension in
the list, in addition to searching for the exact filename. cmd will
always attempt to run any match, regardless of the extension.

You must be thinking of PowerShell, which for some reason reinterprets
how this environment variable has worked since probably OS/2 in the
late 80s. PowerShell only executes files found in PATH that have an
extension that's listed in PATHEXT.

CreateProcess checks the user's execute access in the file security,
which can prevent the execution of .BAT/.CMD files and PE executables,
regardless of extension. But ShellExecute(Ex) has an MS-DOS brain (so
much of the entire Explorer/shell32 implementation has an MS-DOS
brain; it's like they think they're still supporting Windows 9x), so
scripts and data files are always 'executable'. You get some help here
from cmd, which always tries CreateProcess, regardless of extension,
and won't continue to ShellExecuteEx if CreateProcess failed because
access was denied. PowerShell... not so much.

> and since the current directory is always at the beginning
> of your path, this can easily turn into a remote code execution
> exploit.

Since Vista, both CreateProcess and cmd.exe support the environment
variable NoDefaultCurrentDirectoryInExePath. If this is set, you have
to explicitly reference the current directory. PowerShell always
required this.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Chris Angelico]

On Tue, Aug 23, 2016 at 1:56 AM, Random832  wrote:
>> And any GUI that automatically calculates thumbnails from
>> image files (this includes Windows, Mac OS, and more than one Linux
>> window manager) could potentially be attacked via a malformed file,
>> simply by having it appear on the file system.
>
> This has nothing to do with the filename, unless you additionally assume
> that this will only happen if the file is called .jpg

It generally will (or rather, only if the file has one of a particular
set of extensions). Automatic thumbnailing is usually done only for
certain file names. I don't know of anything that opens every single
file to see if it has a JFIF signature (etc for PNG and whatever other
types).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[Chris Angelico]

On Tue, Aug 23, 2016 at 1:54 AM, eryk sun  wrote:
> On Mon, Aug 22, 2016 at 1:17 PM, Chris Angelico  wrote:
>> I tried things like "con.txt" and it simply failed (no such file or
>> directory), without printing anything to the console.
>
> Are you using IDLE or some other IDE that uses pythonw.exe instead of
> python.exe? If so, first use ctypes to allocate a console:
>
> import ctypes
> ctypes.WinDLL('kernel32').AllocConsole()
>
> The CON device should work if the process is attached to a console
> (i.e. a conhost.exe instance).

No, I used Pike (to avoid any specifically-Python issues or
protections) running in a console. Attempting to write to "Logs/con"
wrote to the console, so I know the console device is active.
Attempting to write to "Logs/con.txt" failed as described.

> > I wouldn't accept file names from untrusted sources on *any* system
>
> There are still desktop applications that ask users to name their files.

A person running a desktop application is generally considered a
trusted source. In a kiosk environment, you have a lot more to worry
about than special device names (eg someone could overwrite a key
file), so again, allowing an untrusted user to name a file in that
situation would be inappropriate. If the user owns the computer, s/he
should be allowed to attempt any name, and there'd simply be some that
fail - same as any other invalid characters (Windows won't let you put
a colon or question mark in a file name, for instance, which annoys my
brother no end when I give him files like "What's Up, Doc?.mkv" or
"Operation: Rabbit.mkv") or over-long names or anything like that.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman

[Marko Rauhamaa]

Random832 :

> On Mon, Aug 22, 2016, at 11:40, Chris Angelico wrote:
>> Windows has some other issues, including that arbitrary files can
>> become executable very easily (eg if %PATHEXT% includes its file
>> extension), and since the current directory is always at the
>> beginning of your path, this can easily turn into a remote code
>> execution exploit.
>
> I didn't include dot in my example whitelist, and there's no mechanism
> for an attacker to add random extensions to your PATHEXT.

Years back, my FTP server was hacked by exploiting a buffer overflow.
The anonymous input directory contained a very long filename that
apparently contained some valid x86 code.

Did you vet your whitelist so it couldn't possibly be interpreted by the
CPU as meaningful instructions?


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27828] Regression in http.cookies parsing with single key without value

[Yuriy Zhuravlev]

Yuriy Zhuravlev added the comment:

>Any RFC I'm looking at says that the value is required, so your syntax is 
>simply not valid and is rejected by the parser.

Perhaps but this cookies sets not by me (maybe JS ads widget or etc). And I 
can't fix it. I can't auth users because their cookies wrong. And I can't 
detect this situation because all process is silent without exception.

>I am not the biggest expert on cookie syntaxes, but maybe these reports are 
>related:

I suppose you are right.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Random832]

On Mon, Aug 22, 2016, at 11:40, Chris Angelico wrote:
> Windows has some other issues, including that arbitrary files can
> become executable very easily (eg if %PATHEXT% includes its file
> extension), and since the current directory is always at the beginning
> of your path, this can easily turn into a remote code execution
> exploit.

I didn't include dot in my example whitelist, and there's no mechanism
for an attacker to add random extensions to your PATHEXT.

> And any GUI that automatically calculates thumbnails from
> image files (this includes Windows, Mac OS, and more than one Linux
> window manager) could potentially be attacked via a malformed file,
> simply by having it appear on the file system.

This has nothing to do with the filename, unless you additionally assume
that this will only happen if the file is called .jpg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[eryk sun]

On Mon, Aug 22, 2016 at 1:17 PM, Chris Angelico  wrote:
> I tried things like "con.txt" and it simply failed (no such file or
> directory), without printing anything to the console.

Are you using IDLE or some other IDE that uses pythonw.exe instead of
python.exe? If so, first use ctypes to allocate a console:

import ctypes
ctypes.WinDLL('kernel32').AllocConsole()

The CON device should work if the process is attached to a console
(i.e. a conhost.exe instance).

You can detect any of the classic DOS devices via isatty(), since
they're all character devices. Below I'm using Windows 10, so the
console uses a kernel device (condrv.sys was added in Windows 8), for
which the name can be queried:

>>> f = open('con.txt')
>>> h = msvcrt.get_osfhandle(f.fileno())
>>> devname =  UNICODE_STRING()
>>> ctypes.resize(devname, sizeof(devname) + 32767*2)
>>> ntdll.NtQueryObject(h, 1, byref(devname), sizeof(devname))
0
>>> print(devname.Buffer)
\Device\ConDrv

> I wouldn't accept file names from untrusted sources on *any* system

There are still desktop applications that ask users to name their files.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27784] Random failure of test_TCPServer() of test.test_socketserver.SocketServerTest and test_handle_accept() of test.test_asyncore.TestAPI_UseIPv6Select on FreeBSD buildbots

[Berker Peksag]

Berker Peksag added the comment:

Here is another one on koobs-freebsd10: 
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/4809/steps/test/logs/stdio
 test_handle_accept was fine this time.

And this one is from koobs-freebsd9:

==
ERROR: test_handle_accept (test.test_asyncore.TestAPI_UseIPv6Select)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncore.py", 
line 500, in tearDown
asyncore.close_all()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/asyncore.py", 
line 561, in close_all
x.close()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/asyncore.py", 
line 397, in close
self.socket.close()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/socket.py", line 
407, in close
self._real_close()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/socket.py", line 
401, in _real_close
_ss.close(self)
ConnectionResetError: [Errno 54] Connection reset by peer

--

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/4705/steps/test/logs/stdio

--
components: +Tests
nosy: +berker.peksag
stage:  -> needs patch
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Chris Angelico]

On Tue, Aug 23, 2016 at 12:21 AM, Ben Finney  wrote:
>
> So yes, filenames from arbitrary sources should be *completely*
> untrusted, and never used to access any file on the system. Throw the
> entire filename away and make a filename locally, without using any part
> of the original name.

Oh, and I wish you could convince some other parts of the world about
this. When you mix file uploads with Apache+PHP web applications, you
basically get remote code execution right there. As sysadmin, I have
to constantly play whack-a-mole with stupid exploits that just
wouldn't happen if Joomla's web uploads followed this model.
Seriously, how hard is it for something that *already has a database*
to simply insert a row into jos_uploaded_files and then use that row's
ID eg "uploads/file_"+id as the file name? In one stroke, you
eliminate code execution (it can't be ".php" or any variant thereof),
collisions (two uploads with the same file name will get different
IDs), and even detritus from incomplete transactions (if you find
"uploads/file_12345" but there's no row with ID 12345, you can assume
the transaction got rolled back, and safely delete the file).

S'not that hard, folks. Be safe. Be smart.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Chris Angelico]

On Tue, Aug 23, 2016 at 12:34 AM, Random832  wrote:
> On Mon, Aug 22, 2016, at 10:21, Ben Finney wrote:
>> So yes, filenames from arbitrary sources should be *completely*
>> untrusted, and never used to access any file on the system. Throw the
>> entire filename away and make a filename locally, without using any part
>> of the original name.
>
> To be fair, this particular case is unique in presenting a possibility
> to cause problems even for a filename that consists only of whitelisted
> characters (for a reasonable-sounding whitelist such as "ASCII letters
> and numbers and underscore only; all other characters to be scrubbed and
> replaced with {underscore, hex escape, nothing}"). I don't think there's
> any other precedent.

Windows has some other issues, including that arbitrary files can
become executable very easily (eg if %PATHEXT% includes its file
extension), and since the current directory is always at the beginning
of your path, this can easily turn into a remote code execution
exploit. And any GUI that automatically calculates thumbnails from
image files (this includes Windows, Mac OS, and more than one Linux
window manager) could potentially be attacked via a malformed file,
simply by having it appear on the file system. So the idea that some
file names are dangerous is far FAR broader than "a file called
prn.txt will get saved to the printer".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Tim Chase]

On 2016-08-23 00:21, Ben Finney wrote:
> So yes, filenames from arbitrary sources should be *completely*
> untrusted, and never used to access any file on the system. Throw
> the entire filename away and make a filename locally, without using
> any part of the original name.

Sadly, this ideal advice too often conflicts with the shoddy Code
Other People Wrote In Our "Enterprise" System™. :-/

-tkc


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP suggestion: Uniform way to indicate Python language version

[Chris Angelico]

On Tue, Aug 23, 2016 at 12:29 AM, Random832  wrote:
> On Mon, Aug 22, 2016, at 09:21, Steve D'Aprano wrote:
>> Rather, you just use the features you rely on, document the minimum
>> supported version, and if somebody is silly enough to try running your
>> code
>> under Python 1.4, they'll get a SyntaxError or an exception when you try
>> to
>> do something that is not supported.
>
> Receiving a SyntaxError or whatever other exception, which provides no
> suggestion about how to actually fix the issue (install a later version
> of python / run with "python3" instead of "python"), is a bad user
> experience. It will continue to be a bad user experience when people are
> using features that only work on python 5.0 and later and other people
> are trying to run their scripts under python 4.0, so it not having
> existed all along is not a sufficient justification to not consider
> adding it,.

How bad, exactly? If you installed the program using a package manager
(apt, yum, brew, etc etc etc), there should be versioning directives
in that, just the same as if there's some particular feature of libc
or openssl or libjpeg that you need. And if you go to someone's web
site to download it, you should get told "This requires Python X.Y".

Sometimes, when there's a really common failure, you can toss in a
try/except, or even just a comment:

import tkinter # ImportError? You may need Python 3

which will then show up if something goes wrong. There are much, MUCH
better ways of solving this than trying to magically make older
Pythons, uhh uhh do what, exactly? Nobody's yet said what
ought to happen instead of SyntaxError/ImportError. Should the process
be terminated with an error message? Should a different exception be
raised (InsufficientVersionError) and a check added to the default
error handler so it displays differently (the way SystemExit does, or
rather doesn't display at all)? And above all, how is all this an
improvement over the status quo, which is either a
SyntaxError/ImportError straight away, or the ability to put a
straight-forward run-time check that does anything you like?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue18139] email module should have a way to prepend and insert headers

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

I meant .items()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18139] email module should have a way to prepend and insert headers

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

I've updated this to Python 3.6, but I don't know if there's time to design an 
API and implementation in the time left before beta 1.  But a use case has come 
up, so I want to reboot this discussion (yes, it should go to email-sig too).

Apparently ARC headers  requires a header order.  In 
Mailman we have a GSoC student who is working on an implementation, and he's 
stuck because he needs to *prepend* ARC headers on the message, but the API 
gives him no direct way to do this.

We do have all the API problems that RDM points out.  I'm not going to suggest 
to the student that they use Message._headers.  It seems like the only way to 
do what he needs is to .get() all the headers in order, then del them all, 
manipulate the list out-of-band, and then re-add them all in the original order 
with the ARC headers at the front.  Yuck.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18139] email module should have a way to prepend and insert headers

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27587] Issues, reported by PVS-Studio static analyzer

[Berker Peksag]

Changes by Berker Peksag :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27587] Issues, reported by PVS-Studio static analyzer

[Roundup Robot]

Roundup Robot added the comment:

New changeset 51627344fc58 by Berker Peksag in branch '3.5':
Issue #27587: Move null pointer check earlier in _PyState_AddModule()
https://hg.python.org/cpython/rev/51627344fc58

New changeset 7d90bf4780ff by Berker Peksag in branch 'default':
Issue #27587: Merge from 3.5
https://hg.python.org/cpython/rev/7d90bf4780ff

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: saving octet-stream png file

[Larry Martell]

On Fri, Aug 19, 2016 at 4:24 PM, Chris Kaynor  wrote:
> On Fri, Aug 19, 2016 at 12:00 PM, Larry Martell 
> wrote:
>
>> On Fri, Aug 19, 2016 at 1:24 PM, Chris Angelico  wrote:
>> > On Sat, Aug 20, 2016 at 3:10 AM, Larry Martell 
>> wrote:
>> >> I have some python code (part of a django app) that processes a
>> >> request that contains a png file. The request is send with
>> >> content_type = 'application/octet-stream'
>> >>
>> >> In the python code I want to write this data to a file and still have
>> >> it still be a valid png file.
>> >>
>> >> The data I get looks like this:
>> >>
>> >> u'\ufffdPNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\ufffd\
>> x00\x00\x01\ufffd
>> >> ..'
>> >>
>> >> If I try and write that to a file it fails with a UnicodeEncodeError.
>> >> If I write it with encode('utf8') it writes the file, but then it's no
>> >> longer a valid png file.
>> >>
>> >> Anyone know how I can do this?
>> >
>> > At that point, you've already lost information. Each U+FFFD (shown as
>> > "\ufffd" above) is a marker saying "a byte here was not valid UTF-8"
>> > (or whatever was being used). Something somewhere took the .png file's
>> > bytes and tried to interpret them as text, which they're not.
>> >
>> > What sent you that data? How did you receive it?
>>
>> The request is sent by a client app written in C++ with Qt. It's
>> received by a django based server. I am trying to port a falcon server
>> to django. The falcon server code did this:
>>
>> form = cgi.FieldStorage(fp=req.stream, environ=req.env)
>>
>> and then wrote the png like this:
>>
>> fd.write(form[key].file.read())
>>
>> Whereas in the django server I am doing:
>>
>> fd.write(request.POST[key])
>>
>> I've never used the cgi module. I guess I can try that. I've written a
>> lot with django but never had to receive a PNG file.
>>
>>
> I don't know Django, however a quick search makes it seem like you might
> need to use request.FILES[key] (1) rather than request.POST[key]. You may
> also be able to use request.POST if you set request.encoding first (2). If
> both of those fail, you may need to use request.body and parse the HTTP
> form data manually, though I'd imagine there is an easier way.
>
> [1]
> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.FILES
>
> [2]
> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.encoding

Thanks for the reply. When I get the request, request.FILES is empty.
Yet the content type is multipart/form-data and the method is POST:

(Pdb) print request.META['CONTENT_TYPE']
multipart/form-data;
boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="

(Pdb) print request.META['REQUEST_METHOD']
POST

(Pdb) print request.FILES


Tried setting request.encoding, but that messes up the request structure:

(Pdb) type(request.POST[key])

(Pdb) request.encoding = "iso-8859-1"
(Pdb) type(request.POST[key])
*** MultiValueDictKeyError:
"u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: saving octet-stream png file

[Larry Martell]

On Sun, Aug 21, 2016 at 5:24 PM, Jon Ribbens  wrote:
> On 2016-08-19, Larry Martell  wrote:
>> fd.write(request.POST[key])
>
> You could try:
>
>   request.encoding = "iso-8859-1"
>   fd.write(request.POST[key].encode("iso-8859-1"))
>
> It's hacky and nasty and there might be a better "official" method
> but I think it should work.

For some reason that messes up the request structure:

(Pdb) type(request.POST[key])

(Pdb) request.encoding = "iso-8859-1"
(Pdb) type(request.POST[key])
*** MultiValueDictKeyError:
"u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Random832]

On Mon, Aug 22, 2016, at 10:21, Ben Finney wrote:
> So yes, filenames from arbitrary sources should be *completely*
> untrusted, and never used to access any file on the system. Throw the
> entire filename away and make a filename locally, without using any part
> of the original name.

To be fair, this particular case is unique in presenting a possibility
to cause problems even for a filename that consists only of whitelisted
characters (for a reasonable-sounding whitelist such as "ASCII letters
and numbers and underscore only; all other characters to be scrubbed and
replaced with {underscore, hex escape, nothing}"). I don't think there's
any other precedent.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP suggestion: Uniform way to indicate Python language version

[Random832]

On Mon, Aug 22, 2016, at 09:21, Steve D'Aprano wrote:
> Rather, you just use the features you rely on, document the minimum
> supported version, and if somebody is silly enough to try running your
> code
> under Python 1.4, they'll get a SyntaxError or an exception when you try
> to
> do something that is not supported.

Receiving a SyntaxError or whatever other exception, which provides no
suggestion about how to actually fix the issue (install a later version
of python / run with "python3" instead of "python"), is a bad user
experience. It will continue to be a bad user experience when people are
using features that only work on python 5.0 and later and other people
are trying to run their scripts under python 4.0, so it not having
existed all along is not a sufficient justification to not consider
adding it,.
-- 
https://mail.python.org/mailman/listinfo/python-list

The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)

[Ben Finney]

Chris Angelico  writes:

> […] to be honest, I wouldn't accept file names from untrusted sources
> on *any* system […]

That's one of the wiser things said in this whole thread.

> I'd use arbitrary numbers or hashes as the file names, and store the
> originally-submitted file name in some sort of metadata repository,
> like a Postgres table.)

The failure modes of using filenames from untrusted input are shockingly
diverse, as Tom Eastman describes:

The scope for abuse is eye-widening: The contents of the file, the
type of the file, the size and encoding of the file, even the *name*
of the file can be a potent vector for attacking your system.

The scariest part? Even the best and most secure web-frameworks
(yes, I'm talking about Django) can't protect you from all of it.

In this talk, I'll show you every scary thing I know about that can
be done with a file upload, and how to protect yourself from --
hopefully -- most of them.



Tom presented to us at this year's PyCon AU
.

So yes, filenames from arbitrary sources should be *completely*
untrusted, and never used to access any file on the system. Throw the
entire filename away and make a filename locally, without using any part
of the original name.

-- 
 \“I saw a sign: ‘Rest Area 25 Miles’. That's pretty big. Some |
  `\  people must be really tired.” —Steven Wright |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[Steve D'Aprano]

On Mon, 22 Aug 2016 10:56 pm, Random832 wrote:

> On Mon, Aug 22, 2016, at 08:39, Chris Angelico wrote:
>> Nope. On Windows, you would try/except it.
> 
> No, you can't, because the failure mode often isn't "file refuses to
> open" but "data is written to a serial port".

Ah, that's a good point. I hadn't thought of that.

But... what are the consequences if you write to the serial port? Unless you
actually have an external device plugged into it, isn't that equivalent to
writing to /dev/null? (Bytes go into the serial port, and just disappear.)
The user uploads their file, and cleverly fools you into discarding their
file? I'm not seeing how this is an attack.


I suppose they could write to CON and display a lot of garbage on the
screen. But if you're running this on Windows, surely you've already dealt
with these issues, in which case it's a non-issue. Or you haven't dealt
with them, in which case it's an existing bug and the code Lawrence
demonstrated doesn't change anything.



>> There are myriad other ways
>> something could fail, and the only correct action is to attempt it.
>> Most of the reserved names will simply give an error; the only way
>> you'd actually get incorrect behaviour is if the file name, including
>> extension, is exactly a device name.
> 
> I think the reason you believe this can be traced back to the
> "C:\con\con" trick, which crashed the system by trying to use the name
> as a directory.

\con\con hasn't been an issue since Windows 98. If you're running your web
application under Win 98, you deserve to be blue-screened :-)




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP suggestion: Uniform way to indicate Python language version

[Steve D'Aprano]

On Mon, 22 Aug 2016 10:52 pm, Random832 wrote:

> On Mon, Aug 22, 2016, at 08:44, Chris Angelico wrote:
>> However, I don't think it's particularly necessary. Explicit version
>> number checks should be very rare, and shouldn't be encouraged.
>> Instead, encourage feature checks, as Steve gave some examples of.
> 
> The problem is when you want to write a large body of code that just
> *uses* lots of features (including syntactic features), *without*
> checking for them. Steve's examples were of how to support earlier
> versions, not how *not* to. At some point you want to just write Python
> 3 code and say to hell with Python 2, or the same for any particular
> version of Python 3.

Surely that's not a problem -- it's what people have been doing with Python
code for decades. When was the last time you saw somebody do an explicit
version check?

if sys.version_info() < (2, 2):
raise RuntimeError('Python too old')


Rather, you just use the features you rely on, document the minimum
supported version, and if somebody is silly enough to try running your code
under Python 1.4, they'll get a SyntaxError or an exception when you try to
do something that is not supported.

Maybe if Python supported this from the beginning it would be worth the
bother. But it seems of very marginal utility to me.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does This Scare You?

[Steve D'Aprano]

On Mon, 22 Aug 2016 09:50 pm, Jon Ribbens wrote:

> On 2016-08-22, Chris Angelico  wrote:
>> On Mon, Aug 22, 2016 at 8:33 PM, Jon Ribbens 
>> wrote:
>>> On 2016-08-22, Steve D'Aprano  wrote:
 On Mon, 22 Aug 2016 10:38 am, eryk sun wrote:
> To me it's scary that this check misses cases because it's trying to
> be cross-platform instead of simply relying on GetFullPathName to do
> the work. For example, it misses at least the following cases:

 Instead of shaking in your boots over a simple bug in a non-critical
 library, how about reporting these cases on the bug tracker with an
 explanation of the problem?
>>>
>>> That seems a rather unnecessarily harsh response.
>>> Also, it's not "non-critical", this is a security bug.
>>
>> Explain how?
> 
> I don't know what purpose you are envisaging this function being used
> for, but the only one I can think of is input sanitisation. e.g. a web
> form where you receive a file from the Internet and store it somewhere,
> and you want to use the filename given to you rather than choose your
> own randomly-generated one.
> 
> Under Unix all you need to do is check for the filename starting with
> "." or containing "/." (or "/", depending on your requirements).
> Under Windows you would use this function, which apparently doesn't
> work, hence: security hole.

That's backwards: it works under Windows, but is buggy under non-Windows.

In any case, I don't think that what you suggest is even close to
sufficient.

For starters, even on Unix I probably wouldn't want to
use "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\.txt" as a
file name, even if it is technically legal. Nor would I allow 

"foo.jpg

   .exe"

either, especially not on Windows.

But let's suppose my application can handle newlines and other control
characters in filenames (I never `ls` the storage directory, all my scripts
are newline safe, etc.) And I don't care about users uploading malware with
disguised file names. Am I done?

No, not at all. I can't assume that just because a file name is legal that I
can write to it. Even after sanitising the name (or deciding I don't need
to bother) I still need to be prepared to catch I/O errors when writing the
file -- including attempts to write to reserved file names.

None of this makes the issue a security issue. How would you exploit it? The
attacker would have to ensure that the file name is sanitised under Linux
but then written to a file under Windows, in which case the worst that
happens is that I get an unexpected I/O error.

Technically that could be called a DoS attack, but the same could be said
for ANY bug that raises an exception. I think that it takes more than that
to be a security bug.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27660] Replace size_t with Py_ssize_t as the type of local variable in list_resize

[Xiang Zhang]

Xiang Zhang added the comment:

Ping.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[STINNER Victor]

STINNER Victor added the comment:

fastcall_macros.patch:

* Rename _PyObject_FastCall() to _PyObject_FastCallDict()
* Add _PyObject_FastCall(func, args, nargs) macro
* Add _PyObject_CallArg1(func, arg) macro
* Add _PyObject_CallNoArg(func) macro

The 3 new macros call _PyObject_FastCallDict().

Statistics on calls to these functions and macros:

* _PyObject_CallNoArg: 11
* _PyObject_CallArg1: 8
* _PyObject_FastCall: 5
* _PyObject_FastCallDict: 0   (the function doesn't work yet, it's the purpose 
of this issue, so it's expected that it's unused yet ;-))

_PyObject_CallNoArg() and _PyObject_CallArg1() are much more common and the 
more complex format _PyObject_FastCall() format (more than 1 argument). I like 
having macros with a simple prototype.

--

The C API has a wide choice of functions to call a function, but I still want 
to add  _PyObject_CallNoArg() and _PyObject_CallArg1() to get the best 
performances (avoid layers before reaching the final _PyObject_FastCallDict) 
and have a simple and well defined prototype for the most common function calls.

kwargs:

* PyObject_Call(): root of all call functions in Python 3.5
* PyEval_CallObjectWithKeywords(func, args, kwargs) -> PyObject_Call()

args:

* PyEval_CallObject(func, args)
* PyObject_CallObject(func, args) -> PyEval_CallObjectWithKeywords()

vargs:

* PyObject_CallFunctionObjArgs(func, ...) -> PyObject_Call()

format:

* PyObject_CallFunction(func, format, ...) -> PyObject_Call()
* PyObject_CallMethod(func, method, format, ...) -> PyObject_Call()
* _PyObject_CallMethodId(func, method_id, format, ...) -> PyObject_Call()
* PyEval_CallFunction(func, format, ...) -> PyEval_CallObject()
* PyEval_CallMethod(func, method, format, ...) -> PyEval_CallObject()

Fast call:

* _PyObject_CallNoArg(func) -> _PyObject_FastCallDict()
* _PyObject_CallArg1(func, arg) -> _PyObject_FastCallDict()
* _PyObject_FastCall(func, arg) -> _PyObject_FastCallDict()

--
Added file: http://bugs.python.org/file44188/fastcall_macros.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27824] update ConfigParser docs regarding in-line comments

[R. David Murray]

R. David Murray added the comment:

The python3 docs do not mention inline comments.  Perhaps we even want a 
documented deprecation here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27824] update ConfigParser docs regarding in-line comments

[R. David Murray]

R. David Murray added the comment:

Alternate proposal.  Replace that whole paragraph with:

Configuration files may include comments, prefixed by specific characters (# 
and ;). Comments may appear on their own in an otherwise empty line.  If a line 
has a value or section name that does not include a ';', an inline comment can 
be added by entering at least one space followed by a ';' and the comment.  
While supported, inline comments are fragile in the face of values that may 
contain ';'s, and thus are not recommended.

Of course, I'm making a (new) value judgement in that last sentence so lukasz 
may override me :)

--
nosy: +lukasz.langa, r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: PEP suggestion: Uniform way to indicate Python language version

[Rustom Mody]

On Monday, August 22, 2016 at 7:00:36 PM UTC+5:30, Steve D'Aprano wrote:
> Realistically, by the time you convince people this is a useful feature,
> write a patch and have the patch reviewed, you'll be looking at Python 3.7.
> So 3.7 will have this new syntax "use version", and *no other version*.
> That means you can't use it until you are ready to abandon Python 2.4
> through 3.6.

Yes 

First-class version detection and management needs to be put into the language
right at the outset if it is to work
If you do it late in the history (like now for python), you need to
1 First detect if first-class version management is supported
2 If yes use it
3 If no use the old ad-hoc ways

So might as well use 3 alone!!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP suggestion: Uniform way to indicate Python language version

[Steve D'Aprano]

On Mon, 22 Aug 2016 08:32 pm, rocky wrote:

> On Monday, August 22, 2016 at 1:36:07 AM UTC-4, Steven D'Aprano wrote:
[...]
>> But... I don't understand what this proposal actually is. We already have
>> a uniform way to indicate the Python language version: check sys.version,
>> or sys.version_info, or sys.hexversion, whichever is more appropriate for
>> your needs.
> 
> The fact that you mention 3 different ways says to me this isn't uniform.

What do you mean? Of course it is uniform -- every compliant Python
interpreter must support sys.version, sys.hexversion and sys.version_info.

sys.version and sys.hexversion go back to at least Python 1.5 and possibly
older. I'm not sure when version_info was introduced, but anything offering
Python 2.4 or better must support it. So as far as current and future
versions of Python go, you can trust that *any* version of *any*
implementation will support those three attributes, be it CPython, PyPy,
Nuitika, Jython, IronPython, Stackless, or something else. 

Perhaps you mean "unique" rather than uniform -- but of course you cannot
have a unique way of doing this, since people can simply test for the
existence of features, or inspect the version number, whichever they
prefer. Adding something like Perl's "use" won't change that -- it still
won't be unique.


>> Could somebody (the OP?) please explain what is the purpose of this
>> proposal, what it does, how it works, and when would people use it?
> 
> All of the ways you mention involve running the program. Suppose you have
> a tool that analyzes source code. It reads the code, but doesn't run it.
> For example possibly code climate. 

What is "code climate"?


> How it it figure out which dialect of Python is being used?

In general, you can't, because Python code will run under many different
versions and implementations.

I'm not strongly opposed to this suggestion. I just don't see the point.
I've never found myself missing this feature. And I don't like version
testing except as an absolute last resort: I much prefer feature detection.

But the biggest problem with this suggestion is the practical issue that the
absolute earliest it could be introduced will be Python 3.6, and more
likely 3.7. Python 3.6 has only a couple of weeks left before feature
freeze, and all older versions are likewise frozen or out of support.

Realistically, by the time you convince people this is a useful feature,
write a patch and have the patch reviewed, you'll be looking at Python 3.7.
So 3.7 will have this new syntax "use version", and *no other version*.
That means you can't use it until you are ready to abandon Python 2.4
through 3.6.

I'm assuming you want a new syntactic feature like Perl's "use version". If
its just a function, then you can use a version check to test for it first:

if sys.version_info > (3, 7):
sys.use("v3.7")

but that makes "use" completely redundant. (If you have to test the version
before you can test the version, what's the point of the second test?)

If this feature had been built into Python from the start, it would merely
be an attractive nuisance, encouraging people to test the version instead
of doing feature-detection. But since it wasn't, I don't see any point to
add it now: if we added it to Python 3.7, it wouldn't be until (probably)
2028 or significantly later that folks could rely on it being available.

(Currently, I think that Red Hat offer the longest paid commercial support
for Python versions: ten years. Suppose they start using Python 3.6 as
their standard Python next year, it won't fall out of support until 2028.
That's the absolute earliest pre-3.7 versions could be considered out of
support.)



Earlier, in a previous email, Rocky also wrote:

> Right. People are focusing on specific code instead of the problem: a
> simple and uniform way to indicate a specific Python dialect in force for
> that program. 

That's not describing the problem. That's describing one specific solution
to some unknown problem. The *solution* is "a simple and uniform way to
indicate..." but what is the problem? Under what circumstances would people
use this? Why would they use it?

Normally, there's no question about what version of Python a module is meant
to use:

- if it is a script, it has a hash-bang line at the top which specifies the
location of the Python interpreter;

- if its a library, it will be installed in the site-packages directory of a
specific version of the interpreter;

- and more importantly, many (possibly even most) Python code will run under
multiple versions.

There are exceptions to this of course. Sometimes libraries will be just
dumped any old place, and scripts may not have hash-bang lines. Neither of
these are compulsory. But neither will your proposed "use version".


> The language continues to evolve over time: there are many 
> Python 2.7 programs that won't work on Python 2.5 or earlier and vice
> versa. When you expand the range from Python 1.5 to Python 3.6 the
> 

[issue27809] _PyObject_FastCall(): add support for keyword arguments

[STINNER Victor]

STINNER Victor added the comment:

> _PyObject_FastCallKeywords(PyObject **args, int nargs, PyObject **kwargs, int 
> nkwargs): nkwargs is the number of (key, value) pairs

Hum, I don't know if it's worth to "split" the C array into two parts. Maybe it 
can be just:

_PyObject_FastCallKeywords(PyObject **args, int nargs, int nkwargs);

As current used in Python/ceval.c.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Does This Scare You?

[Chris Angelico]

On Mon, Aug 22, 2016 at 10:56 PM, Random832  wrote:
>> Most of the reserved names will simply give an error; the only way
>> you'd actually get incorrect behaviour is if the file name, including
>> extension, is exactly a device name.
>
> I think the reason you believe this can be traced back to the
> "C:\con\con" trick, which crashed the system by trying to use the name
> as a directory.

I tried things like "con.txt" and it simply failed (no such file or
directory), without printing anything to the console. But as Eryk
says, adding an underscore is safe; and to be honest, I wouldn't
accept file names from untrusted sources on *any* system - at very
least, I'd prefix/suffix them with something to ensure uniqueness,
which would deal with this issue as a convenient side effect. (Or
alternatively, I'd use arbitrary numbers or hashes as the file names,
and store the originally-submitted file name in some sort of metadata
repository, like a Postgres table.)

So I still don't see this as a security problem, just a practicality one.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP suggestion: Uniform way to indicate Python language version

[Marko Rauhamaa]

Random832 :
> The problem is when you want to write a large body of code that just
> *uses* lots of features (including syntactic features), *without*
> checking for them.

Ordinarily, that's the job of package management. The installer will
perform the necessary checks for you.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27818] Speed up number format spec parsing

[STINNER Victor]

STINNER Victor added the comment:

-while (pos

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27213] Rework CALL_FUNCTION* opcodes

[STINNER Victor]

STINNER Victor added the comment:

See also the issue #27809 "_PyObject_FastCall(): add support for keyword 
arguments" where we discuss if we can completely avoid the creation of a 
temporary dictionary to pass keyword arguments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Does This Scare You?

[eryk sun]

On Mon, Aug 22, 2016 at 12:39 PM, Chris Angelico  wrote:
>
> Nope. On Windows, you would try/except it. There are myriad other ways

No, I would not rely on exceptions in this case. Both \\.\CON and
\\.\NUL can be opened for both reading and writing, so you may not
detect the problem.

I think it's best to check for this via os.path.abspath, which calls
GetFullPathName, which is implemented by RtlGetFullPathName_Ustr,
which is exactly what RtlDosPathNameToNtPathName_U uses to normalize a
path before it converts it to the NT namespace.

> or just slap some extra text onto the beginning or end of the file name
> (beginning meaning "after the last slash", not the beginning of the file
> *path*) and you're safe.

Adding to the beginning is the safe bet without having to worry about
the rules for trailing spaces, dots, and colons at the end of the
name. Adding a single underscore prefix will suffice (or however many
to make a unique name) if for some reason you can't ask the user for a
new name.
-- 
https://mail.python.org/mailman/listinfo/python-list