[issue29326] Blank lines in ._pth file are not ignored

2017-01-27 Thread Alexey Izbyshev

Alexey Izbyshev added the comment:

I've encountered this issue too. (FYI, the official 3.6.0 embeddable zip from 
https://www.python.org/downloads/windows/ does contain a blank line in its 
._pth, so all its users get an invalid entry in sys.path).

The patch is attached. I couldn't fix tests in more straightforward way because 
both of them were broken:
* Both of them generated ._pth with 'import site'
* Both of them checked for zero exit code instead of non-zero one, but both 
passed because predicates in sys.exit() were incorrect

I've strengthened the check in 'nosite' test because it is trivial to fully 
emulate sys.path calculation logic in this case. I've preserved weaker checks 
in the other test because I didn't want to emulate site.py logic.

--
nosy: +izbyshev
Added file: http://bugs.python.org/file46433/underpth_blank_lines.diff

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29326>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28896] Embeddable zip allows Windows registry to override module location

2016-12-08 Thread Alexey Izbyshev

Alexey Izbyshev added the comment:

Thanks to Steve and everyone for quick and decisive action!

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28896>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28896] Embeddable zip allows Windows registry to override module location

2016-12-07 Thread Alexey Izbyshev

New submission from Alexey Izbyshev:

The docs claim: "... the embedded distribution is (almost) fully isolated from 
the user’s system, including environment variables, system registry settings, 
and installed packages."

Via ProcessMonitor tool I've discovered that python.exe still accesses keys 
like "HKLM\Software\Python\PythonCore\3.5\Modules\collections" on every module 
import, allowing registry settings to override the location of any non-builtin 
module.

Digging into the 3.5.2 code revealed that WindowsRegistryFinder is 
unconditionally added to sys.meta_path (Lib/importlib/_bootstrap_external.py, 
line 1422):

if _os.__name__ == 'nt':
sys.meta_path.append(WindowsRegistryFinder)

It can also be confirmed in runtime:

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.meta_path)
[, , , ]

Is this behavior intended? It seems to be against doc claims and the goal of 
embeddability.

--
components: Windows
messages: 282632
nosy: izbyshev, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Embeddable zip allows Windows registry to override module location
type: behavior
versions: Python 3.5

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28896>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32236] open() shouldn't silently ignore buffering=1 in binary mode

2017-12-18 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

I had similar thoughts when I was fixing tests that broke due to ValueError. 
I've updated the PR to issue a RuntimeWarning instead.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32369] test_subprocess: last part of test_close_fds() doesn't check what's intended

2017-12-18 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
nosy: +gregory.p.smith, pitrou

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32369>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32369] test_subprocess: last part of test_close_fds() doesn't check what's intended

2017-12-18 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

The last part of test_close_fds() doesn't match its own comment. The following 
assertion always holds because fds_to_keep and open_fds are disjoint by 
construction.

self.assertFalse(remaining_fds & fds_to_keep & open_fds,
 "Some fds not in pass_fds were left open")

--
components: Tests
messages: 308581
nosy: izbyshev
priority: normal
severity: normal
status: open
title: test_subprocess: last part of test_close_fds() doesn't check what's 
intended
type: behavior
versions: Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32369>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32369] test_subprocess: last part of test_close_fds() doesn't check what's intended

2017-12-18 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +4815
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32369>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32369] test_subprocess: last part of test_close_fds() doesn't check what's intended

2017-12-18 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32369>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32236] open() shouldn't silently ignore buffering=1 in binary mode

2017-12-13 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +4730
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21332] subprocess bufsize=1 docs are misleading

2017-12-13 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
pull_requests: +4732

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue21332>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue10344] codecs.open() buffering doc needs fix

2017-12-13 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +4731
stage: needs patch -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue10344>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32256] Make patchcheck.py work for out-of-tree builds

2017-12-08 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

Despite some steps made in issue 9860 make patchcheck still doesn't work for 
out-of-tree builds because it runs git and hg in the current directory instead 
of the source directory (msg169465).

--
messages: 307875
nosy: izbyshev
priority: normal
severity: normal
status: open
title: Make patchcheck.py work for out-of-tree builds
type: enhancement
versions: Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32256>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32256] Make patchcheck.py work for out-of-tree builds

2017-12-08 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +4663
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32256>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue13322] buffered read() and write() does not raise BlockingIOError

2017-12-06 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

For added fun: at least one part of the standard library doesn't expect None 
returns from read() in the buffering layer.

>>> import os
>>> r, w = os.pipe2(os.O_NONBLOCK)
>>> f = os.fdopen(r, 'r')
>>> f.read()
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/izbyshev/workspace/cpython/Lib/codecs.py", line 321, in decode
data = self.buffer + input
TypeError: can't concat NoneType to bytes

Note that nonblock-none.patch doesn't seem to address that.

--
nosy: +izbyshev
versions: +Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue13322>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue13322] buffered read() and write() does not raise BlockingIOError

2017-12-06 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Yes, your claim is confirmed by the fact that there have been little interest 
in this issue since 2011. Still, non-blocking behavior is incorrectly specified 
in the docs and is inconsistent (as investigated by Martin). And obscure errors 
like in my example or in Issue 13858 show that I/O stack is confused too. To 
prevent people from tripping on it, would you consider recommending against 
usage of I/O stack for non-blocking operations in io module docs?

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue13322>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32236] open() shouldn't silently ignore buffering=1 in binary mode

2017-12-06 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

The fact that "buffering=1" is usable only in text mode is documented for 
open(). In binary mode, setting buffering to 1 is silently ignored and 
equivalent to using default buffer size. I argue that such behavior is:

1. Inconsistent
For example, attempts to use buffering=0 in text mode or to specify encoding in 
binary mode raise ValueError.

2. Dangerous
Consider the following code fragment running on *nix (inspired by real-world 
code):

with open("fifo", "wb", buffering=1) as out:
for line in lines:
out.write(line)

"fifo" refers to a FIFO (named pipe). For each line, len(line) <= PIPE_BUF. 
Because of line buffering, such code must be able to safely assume that each 
write() is atomic, so that even in case of multiple writers no line read by a 
FIFO reader will ever get intermixed with another line. And it's so in Python 
2. After migration to Python 3 (assuming that line is now bytes), this code 
still works on Linux because of two factors:
a) PIPE_BUF is 4096, and so is the default IO buffer size (usually)
b) When a write() is going to overflow the buffer, BufferedWriter first flushes 
and then processes the new write() (based on my experiment). So, each OS 
write() is called with complete lines only and is atomic per (a).

But PIPE_BUF is 512 on Mac OS X (I don't know about default buffer size). So, 
we are likely to have a subtle 2-to-3 migration issue.

I suggest to raise a ValueError if buffering=1 is used for binary mode. Note 
that issue 10344 (msg244354) and issue 21332 would have been uncovered earlier 
if it was done from the beginning.

--
components: IO
messages: 307775
nosy: izbyshev, pitrou
priority: normal
severity: normal
status: open
title: open() shouldn't silently ignore buffering=1 in binary mode
type: behavior
versions: Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue30718] open builtin function: specifying the size of buffer has no effect for text files

2017-12-06 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Yes, clarifying buffering for text mode in open() would be nice.

@direprobs: just in case you didn't know, you can achieve what you want with 
something like the following in pre-3.7:

with open("/dev/null", "wb", buffering=10) as outb, \
io.TextIOWrapper(outb, write_through=True) as out:
out.write("x" * 20)

Sadly, write_through can't be passed to open(), but it can be changed on 
existing TextIOWrapper since 3.7 (via new reconfigure() method).

--
nosy: +izbyshev

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue30718>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32270] subprocess closes redirected fds even if they are in pass_fds

2017-12-10 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

Demonstration:

$ cat test.py
import os
import subprocess
import sys

fd = os.dup(sys.stdout.fileno())
subprocess.call([sys.executable, '-c',
 'import sys;'
 'print("Hello stdout");'
 'print("Hello fd", file=open(int(sys.argv[1]), "w"))',
 str(fd)],
stdout=fd,
pass_fds=[fd])


$ python3 test.py
Hello stdout
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 9] Bad file descriptor

I see two issues here:

1. The fact that file descriptors specified for redirection are closed (even 
with close_fds=False) unless in range [0, 2] is not documented and not tested. 
If I change the corresponding code in _posixsubprocess.c to close them only if 
they are ends of pipes created by subprocess itself, all tests still pass. 
Also, shells behave differently: e.g., in command 'echo 3>&1' fd 3 is 
duplicated but not closed in the child, so subprocess behavior may be not what 
people expect.

2. pass_fds interaction with (1) should be fixed and documented. We may either 
raise ValueError if pass_fds contains a redirected fd or don't close such a 
redirected fd.

Please provide your thoughts.

--
components: Library (Lib)
files: test.py
messages: 307962
nosy: gregory.p.smith, izbyshev, pitrou, vstinner
priority: normal
severity: normal
status: open
title: subprocess closes redirected fds even if they are in pass_fds
type: behavior
versions: Python 3.6, Python 3.7
Added file: https://bugs.python.org/file47329/test.py

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32270>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32270] subprocess closes redirected fds even if they are in pass_fds

2017-12-10 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Regarding fixing (1), I'm worrying about backward compatibility a bit. Some 
people who discovered that behavior might rely on such "move" semantics and 
expect that the redirected descriptor is not leaked into the child. OTOH, since 
descriptors are non-inheritable by default since 3.4, it might be less of an 
issue now. What do you think?

Otherwise, yes, I'd like to fix this, but fixing is a bit trickier than it may 
seem because sometimes descriptors are duplicated in _posixsubprocess.c (in 
case of low fds), so we need to track the ownership properly. I've already 
performed some work and discovered another issue: in a corner case involving 
closed standard fds, pipes created by subprocess may leak into the child and/or 
an incorrect redirection may occur. Since I can't completely fix this issue 
without affecting a discovered one, I think I'll do the opposite: file a new 
issue, fix it, and then go back to this one.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32270>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32236] open() shouldn't silently ignore buffering=1 in binary mode

2017-12-10 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

I'm in favor of raising an exception because it'll expose existing code with 
incorrect assumptions. I'll check whether it breaks any tests and submit a PR.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32236] open() shouldn't silently ignore buffering=1 in binary mode

2018-01-25 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Any feedback on the updated PR?

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32256] Make patchcheck.py work for out-of-tree builds

2018-01-25 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Any objections to the PR?

--
nosy: +belopolsky, eric.araujo

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32256>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32777] subprocess: child_exec() uses _Py_set_inheritable() which is not async-signal-safe

2018-02-05 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

_Py_set_inheritable() raises a Python-level exception on error and thus is  not 
async-signal-safe, but child_exec() must use only async-signal-safe functions 
because it's executed between fork() and exec().

Since a non-raising version is already implemented in Python/fileutils.c for 
internal use (set_inheritable), I suggest to simply expose it via another 
public function (similar to _Py_open_noraise(), etc.).

--
components: Library (Lib)
messages: 311699
nosy: izbyshev, vstinner
priority: normal
severity: normal
status: open
title: subprocess: child_exec() uses _Py_set_inheritable() which is not 
async-signal-safe
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32777>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32777] subprocess: child_exec() uses _Py_set_inheritable() which is not async-signal-safe

2018-02-05 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5382
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32777>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32844] subprocess may incorrectly redirect a low fd to stderr if another low fd is closed

2018-02-14 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5482
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32844>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32844] subprocess may incorrectly redirect a low fd to stderr if another low fd is closed

2018-02-14 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Note that the PR doesn't attempt to fix leaking of low dup'ed fds to the child. 
I'll file a separate report for that in a while.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32844>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32844] subprocess may incorrectly redirect a low fd to stderr if another low fd is closed

2018-02-14 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

When redirecting, subprocess attempts to achieve the following state: each fd 
to be redirected to is less than or equal to the fd it is redirected from, 
which is necessary because redirection occurs in the ascending order of 
destination descriptors. It fails to do so if a low fd (< 2) is redirected to 
stderr and another low fd is closed, which may lead to an incorrect 
redirection, for example:

$ cat test.py
import os
import subprocess
import sys

os.close(0)

subprocess.call([sys.executable, '-c',
 'import sys; print("Hello", file=sys.stderr)'],
stdin=2,
stderr=1)

$ python3 test.py 2>/dev/null
$ python3 test.py >/dev/null
Hello

Expected behavior:
$ python3 test.py >/dev/null
$ python3 test.py 2>/dev/null
Hello

--
components: Extension Modules, Library (Lib)
messages: 312181
nosy: gregory.p.smith, izbyshev
priority: normal
severity: normal
status: open
title: subprocess may incorrectly redirect a low fd to stderr if another low fd 
is closed
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32844>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32777] subprocess: child_exec() uses _Py_set_inheritable() which is not async-signal-safe

2018-02-05 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

> out of curiosity, did you actually diagnose a process deadlocked due to this 
> or was it noted via code inspection?

The latter. I noted it while working on another issue. While it was easy to 
trigger a malloc() in child_exec() by e.g. specifying an invalid fd in pass_fds 
for Popen, I haven't tried to arrange a deadlock.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32777>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32862] os.dup2(fd, fd, inheritable=False) behaves inconsistently

2018-02-16 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5503
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32862] os.dup2(fd, fd, inheritable=False) behaves inconsistently

2018-02-16 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

os.dup2(fd, fd, inheritable=False) may fail or change fd inheritability in 
following ways:

1) POSIX without F_DUP2FD_CLOEXEC
1.1) dup3() is available (a common case for Linux): OSError (EINVAL, dup3() 
doesn't allow equal descriptors)
1.2) dup3() is not available: fd made non-inheritable

2) POSIX with F_DUP2FD_CLOEXEC (FreeBSD): inheritability is not changed

3) Windows: fd made non-inheritable

In contrast, os.dup2(fd, fd, inheritable=True) never changes fd inheritability 
(same as before PEP 446 landed). I suggest to make os.dup2(fd, fd, 
inheritable=False) behave the same.

--
components: Extension Modules, Library (Lib)
messages: 312266
nosy: benjamin.peterson, izbyshev, vstinner
priority: normal
severity: normal
status: open
title: os.dup2(fd, fd, inheritable=False) behaves inconsistently
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32849] Fatal Python error: Py_Initialize: can't initialize sys standard streams

2018-02-17 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

ktrace shows that dup(0) succeeded but fstat(0) failed. The symptom is the same 
as in #30225. Could you check whether any of the following quick tests produces 
the same error?

python3 -c 'import os, subprocess, sys; r, w = os.pipe(); os.close(w); 
subprocess.call([sys.executable, "-c", ""], stdin=r)'

python3 -c 'import os, subprocess, sys; r, w = os.pipe(); os.close(r); 
subprocess.call([sys.executable, "-c", ""], stdin=w)'

--
nosy: +izbyshev, vstinner

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32865] os.pipe creates inheritable FDs with a bad internal state on Windows

2018-02-17 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

>Also, it has to skip this check if the FD is flagged as a pipe, because a pipe 
>is likely opened in synchronous mode and blocked on a read in the parent, in 
>which case calling GetFileType would deadlock.

Does an FD get flagged as a pipe by calling GetFileType in _open_osfhandle?

Also, it's totally unclear to me how something like GetFileType can interfere 
with I/O and cause a deadlock.

My summary of this report. There are two issues.

The first is a bug in os.pipe (creation of an inheritable descriptor with 
non-inheritable underlying handle). This can be fixed by using 
_open_osfhandle() correctly.

The second is that os.set_inheritable() / _Py_set_inheritable() is broken on 
Windows because it may put descriptor and handle heritability out-of-sync (as 
happens e.g. in os.dup(), os.dup2(), _Py_fopen and friends). This is hard to 
fix because msvcrt doesn't have something like fcntl(F_SETFD) to change 
_O_NOINHERIT flag (though one nasty thought is to poke msvcrt internal 
structures like in _PyVerify_fd removed in #23524).

Both issues cause out-of-sync which is usually hidden and harmless because 
people prefer subprocess.Popen() which doesn't have the ability to propagate 
msvcrt-level descriptors at all. The issue surfaces itself if pipes and 
os.spawn* are involved, leading to two types of inconsistency in the child:
1) Bogus "valid" descriptors referring to wrong handles
2) Unexpectedly invalid descriptors and leaked handles

--
nosy: +izbyshev

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32865>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32862] os.dup2(fd, fd, inheritable=False) behaves inconsistently

2018-02-17 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

@eryksun: Thank you for the note! I've commented on #32865. This adds even more 
inconsistency to this corner case.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32849] Fatal Python error: Py_Initialize: can't initialize sys standard streams

2018-02-19 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

> I think that we can even drop dup-based validation from is_valid_fd()

For POSIX, that is. There is no fstat on Windows, and dup is probably OK there 
(or, even better, dup2(fd, fd) -- no need to close).

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32849] Fatal Python error: Py_Initialize: can't initialize sys standard streams

2018-02-19 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

OK, never mind with the test. I've finally got to a FreeBSD box and reproduced 
the problem. It has to do with 'revoke' feature of *BSD. When revoke is called 
on a terminal device (as part of logout process, for example), all descriptors 
associated with it are invalidated. They can be dup'ed, but any I/O (including 
fstat) will fail with EBADF. The attached 'repro.c' demonstrates the same 
behavior as Python in your ktrace log.

# sleep 5; ./repro > &
# exit
(login again)
# cat err.txt
isatty: Inappropriate ioctl for device
dup ok: 3
fstat: Bad file descriptor

So it seems that in your case the parent of your Python processes passed a 
descriptor referring to the terminal as fd 0, and then terminal got revoked at 
some point. People have stumbled on that, for example, 
https://bitbucket.org/tildeslash/monit/issues/649/init_env-fails-if-open-2-returns-an

As for Python, it seems OK to fix it as in #30225 since the fd is unusable for 
I/O anyway. I think that we can even drop dup-based validation from 
is_valid_fd() since there is a corner case for Linux too: if a descriptor 
opened with O_PATH inherited as a standard one, dup() will succeed but fstat() 
will fail in kernels before 3.6. And we do fstat() almost immediately after 
is_valid_fd() to get blksize, so the dup-based optimization doesn't seem worth 
the trouble.

Victor, do you have an opinion on that?

--
components: +FreeBSD, IO
nosy: +koobs
versions: +Python 3.7
Added file: https://bugs.python.org/file47452/repro.c

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32890] os: Some functions may report bogus errors on Windows

2018-02-20 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

Demo:

>>> os.execve('', ['a'], {})
Traceback (most recent call last):
  File "", line 1, in 
OSError: [WinError 0] The operation completed successfully: ''

The reason is that path_error() used throughout os module always uses 
GetLastError() on Windows, but some functions are implemented via CRT calls 
which report errors via errno. It seems that commit 292c83554 caused this issue.

--
messages: 312446
nosy: izbyshev, vstinner
priority: normal
severity: normal
status: open
title: os: Some functions may report bogus errors on Windows
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32890>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32890] os: Some functions may report bogus errors on Windows

2018-02-20 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5565
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32890>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue31106] os.posix_fallocate() generate exception with errno 0

2018-02-20 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

The failed test is test_posix_fallocate, not test_posix_fallocate_errno. The 
former special-cases Solaris for EINVAL already, and the latter merely passes 
an invalid descriptor.

It seems that this particular issue is still open only because there is no NEWS 
entry in the merged commits.

--
nosy: +izbyshev

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue31106>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32904] os.chdir() may crash on Windows in presence of races

2018-02-21 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

win32_wchdir() retries GetCurrentDirectory() with a larger buffer if necessary, 
but doesn't check whether the new buffer is large enough. Another thread could 
change the current directory in meanwhile, so the buffer could turn out to be 
still not large enough, left in an uninitialized state and passed to 
SetEnvironmentVariable() afterwards.

--
components: Extension Modules, Windows
messages: 312524
nosy: izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
priority: normal
severity: normal
status: open
title: os.chdir() may crash on Windows in presence of races
type: crash
versions: Python 2.7, Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32904>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32903] os.chdir() may leak memory on Windows

2018-02-21 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5579
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32903>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32904] os.chdir() may crash on Windows in presence of races

2018-02-21 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5580
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32904>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32904] os.chdir(), os.getcwd() may crash on Windows in presence of races

2018-02-21 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

The same issue applies to os.getcwd().

--
title: os.chdir() may crash on Windows in presence of races -> os.chdir(), 
os.getcwd() may crash on Windows in presence of races

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32904>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32903] os.chdir() may leak memory on Windows

2018-02-21 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
versions: +Python 2.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32903>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32903] os.chdir() may leak memory on Windows

2018-02-21 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

'new_path' is not freed if the new directory is a UNC path longer than MAX_PATH.

--
components: Extension Modules, Windows
messages: 312522
nosy: izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
priority: normal
severity: normal
status: open
title: os.chdir() may leak memory on Windows
type: resource usage
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32903>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32907] pathlib: test_resolve_common fails on Windows

2018-02-22 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5587
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32907>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32907] pathlib: test_resolve_common fails on Windows

2018-02-22 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

==
FAIL: test_resolve_common (test.test_pathlib.PathTest)
--
Traceback (most recent call last):
  File "C:\workspace\cpython-3.8a\lib\test\test_pathlib.py", line 1538, in 
test_resolve_common
self._check_resolve_relative(p, P(d, 'foo', 'in', 'spam'), False)
  File "C:\workspace\cpython-3.8a\lib\test\test_pathlib.py", line 1477, in 
_check_resolve
self.assertEqual(q, expected)
AssertionError: 
WindowsPath('C:/Users/longusername/AppData/Local/Temp/tmpbenaiqaa-[
13 chars]pam') != WindowsPath('C:/Users/LONGUS~1/AppData/Local/Temp/tmpbenaiqaa-
dirD/foo/in/spam')

==

The problem is that the temporary directory path returned by tempfile.mkdtemp() 
contains the username in "short" (8.3) format, but Path.resolve() converts 
short names to long ones (thanks to ntpath._getfinalpathname()).

Since os.path.realpath() still doesn't resolve symlinks on Windows (#9949, 
#14094), and users of ntpath._getfinalpathname() have to deal with '\\?\' 
prefix, I think I'll just use Path.resolve() for the tmp dir path as a 
workaround.

--
components: Tests, Windows
messages: 312545
nosy: izbyshev, paul.moore, pitrou, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: pathlib: test_resolve_common fails on Windows
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32907>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32869] Incorrect dst buffer size for MultiByteToWideChar in _Py_fopen_obj

2018-02-18 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32869>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32849] Fatal Python error: Py_Initialize: can't initialize sys standard streams

2018-02-20 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5552
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32849] Fatal Python error: Py_Initialize: can't initialize sys standard streams

2018-02-20 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

> But I'm not sure why it can be solved, sometimes, by restarting the the 
> daemon.

Could it be simply because daemon is respawned from a process that does have a 
valid stdin at the time of respawn?

Note that daemon has an option to redirect std streams to /dev/null.

--

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32887] os: Users of path_converter don't handle fd == -1 properly

2018-02-20 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

Demo:
>>> import os
>>> os.chdir(-1)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 14] Bad address: -1
>>> os.chdir(-2)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 9] Bad file descriptor: -2

Functions in os supporting either path or file descriptor argument 
(os.supports_fd) usually use the following code pattern to distinguish between 
those cases:

if (path->fd != -1)
result = fchdir(path->fd);
else
result = chdir(path->narrow);

However, _fd_converter used by path_converter internally doesn't give any 
special meaning to -1 and allows any negative file descriptors. Therefore, if a 
user passes -1 to such function, path->narrow, which is NULL, will be used.

I see two ways to fix this.
1) Make some flag in path_t indicating that it should be treated as fd and make 
all users check that flag.
2) Make _fd_converter raise an exception for negative descriptors.

Also, I have to mention an inconsistency in reporting of bad descriptors. A 
handful of os functions uses fildes_converter for descriptors, which uses 
PyObject_AsFileDescriptor, which in turn is used in other places in Python as 
well (e.g. in fcntl module). PyObject_AsFileDescriptor raises a ValueError for 
negative descriptors instead of OSError raised by most os functions in this 
case.

>>> os.fchdir(-1)
Traceback (most recent call last):
  File "", line 1, in 
ValueError: file descriptor cannot be a negative integer (-1)

--
messages: 312421
nosy: izbyshev
priority: normal
severity: normal
status: open
title: os: Users of path_converter don't handle fd == -1 properly
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32887>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32887] os: Users of path_converter don't handle fd == -1 properly

2018-02-20 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
nosy: +larry

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32887>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32859] os.dup2() tests dup3() availability on each call

2018-02-16 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

os.dup2() tests for dup3() system call availability at runtime, but doesn't 
remember the result across calls, repeating the test on each call with 
inheritable=False even if the test fails.

Judging by the code, 'dup3_works' was intended to be static (the first time its 
value is checked it's always an initial one).

--
components: Extension Modules, Library (Lib)
messages: 312256
nosy: benjamin.peterson, izbyshev, vstinner
priority: normal
severity: normal
status: open
title: os.dup2() tests dup3() availability on each call
type: performance
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32859>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32859] os.dup2() tests dup3() availability on each call

2018-02-16 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5497
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32859>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32849] Fatal Python error: Py_Initialize: can't initialize sys standard streams

2018-02-19 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

Thank you for checking. If this issue happens even when Python is run manually 
from an ordinary shell, fixing it in the same way as in #30225 is probably not 
what you want because while the error message will be gone the corresponding 
std stream will be None (sys.stdin in the case that you ktrace'd). However, if 
fd 0 really becomes unusable for some reason, there isn't anything Python can 
do.

Given your description and ktrace log, I can't imagine why fd 0 would behave 
strangely only in Python. I've attached a small C program to check fd 0. Could 
you compile it and run in an infinite loop from the shell in an attempt to 
reproduce this?

--
Added file: https://bugs.python.org/file47451/fstat0.c

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32869] Incorrect dst buffer size for MultiByteToWideChar in _Py_fopen_obj

2018-02-18 Thread Alexey Izbyshev

New submission from Alexey Izbyshev <izbys...@ispras.ru>:

MultiByteToWideChar expects the destination buffer size to be given in wide 
characters, not bytes.

This is currently not a real issue since _Py_fopen_obj is only used internally 
with mode being a short constant string in all call sites I've found.

--
components: IO, Windows
messages: 312314
nosy: izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
priority: normal
severity: normal
status: open
title: Incorrect dst buffer size for MultiByteToWideChar in _Py_fopen_obj
type: enhancement
versions: Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32869>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32869] Incorrect dst buffer size for MultiByteToWideChar in _Py_fopen_obj

2018-02-18 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
keywords: +patch
pull_requests: +5517
stage:  -> patch review

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32869>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue32920] Implement PEP 529 for os.getcwdb on Windows

2018-02-23 Thread Alexey Izbyshev

Change by Alexey Izbyshev <izbys...@ispras.ru>:


--
nosy: +izbyshev

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32920>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue23808] Symlink to directory on Windows 8

2018-02-26 Thread Alexey Izbyshev

Alexey Izbyshev <izbys...@ispras.ru> added the comment:

This behavior is not specific to just Windows 8. Symlinks to directories are 
treated as directories on Windows, in particular, they should be removed with 
RemoveDirectory, not DeleteFile.

Is there any reason this issue is still open?

--
nosy: +izbyshev

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue23808>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34441] NULL dereference when issubclass() is called on a class with bogus __subclasses__

2018-08-20 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue34441>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34457] Missing NULL check in alias_for_import_name() from Python/ast.c

2018-08-21 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The return value of PyUnicode_InternFromString() is dereferenced without 
checking at 
https://github.com/python/cpython/blob/ef8861c112ed1dac9351958c121bc24ca4ecdb08/Python/ast.c#L3256

Reported by Svace static analyzer.

--
components: Interpreter Core
messages: 323854
nosy: izbyshev, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Missing NULL check in alias_for_import_name() from Python/ast.c
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34457>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34454] datetime: NULL dereference in fromisoformat() on PyUnicode_AsUTF8AndSize() failure

2018-08-21 Thread Alexey Izbyshev


Alexey Izbyshev  added the comment:

I will be glad to rebase my PR and remove the try/except from the test if you 
remove the dependency of separator searching code on PyUnicode_AsUTF8AndSize() 
as you suggest. Or we can go the other way and merge mine first -- whatever you 
prefer.

Note that technically a difference between C and Python implementation of 
fromisoformat() will still remain: if a part of the input string before or 
after the separator contains surrogates, the C code will throw a 
UnicodeEncodeError while the Python code -- ValueError. But since the former 
error is a subclass of the latter, I guess it's OK, what do you think?

Also, note that the other discovered C/Python impl difference (for strftime, 
handled by another try/catch in tests) won't go away, of course, unless someone 
is ready to fix that as well.

--

___
Python tracker 
<https://bugs.python.org/issue34454>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34457] Missing NULL check in alias_for_import_name() from Python/ast.c

2018-08-21 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8324
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34457>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34457] Missing NULL check in alias_for_import_name() from Python/ast.c

2018-08-21 Thread Alexey Izbyshev


Alexey Izbyshev  added the comment:

The NULL dereference seems to be hard to trigger due to caching for 
one-character Latin-1 strings, but being safe probably won't hurt.

--

___
Python tracker 
<https://bugs.python.org/issue34457>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34456] pickle: Missing NULL check in save_global()

2018-08-21 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The return value of Py_BuildValue() is dereferenced without checking at 
https://github.com/python/cpython/blob/ef8861c112ed1dac9351958c121bc24ca4ecdb08/Modules/_pickle.c#L3457

Reported by Svace static analyzer.

--
components: Extension Modules
messages: 323853
nosy: izbyshev, serhiy.storchaka
priority: normal
severity: normal
status: open
title: pickle: Missing NULL check in save_global()
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34456>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34456] pickle: Missing NULL check in save_global()

2018-08-21 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8323
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34456>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34474] Python/bltinmodule.c: Missing NULL check in builtin_sum_impl()

2018-08-23 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8350
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34474>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34477] Objects/typeobject.c: Missing NULL check in type_init()

2018-08-23 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8351
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34477>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34474] Python/bltinmodule.c: Missing NULL check in builtin_sum_impl()

2018-08-23 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The return value of PyFloat_FromDouble() at 
https://github.com/python/cpython/blob/48ce4897f8f8d91d948ecd1241ffab002df2be9e/Python/bltinmodule.c#L2403
 is not checked for NULL and then dereferenced.

Reported by Svace static analyzer.

A similar issue in an 'if' statement above was fixed in #18560. Note that 2.7 
is affected by both issues, so I can fix both if needed.

--
components: Interpreter Core
messages: 323946
nosy: berker.peksag, izbyshev, pitrou, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Python/bltinmodule.c: Missing NULL check in builtin_sum_impl()
type: behavior
versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34474>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34477] Objects/typeobject.c: Missing NULL check in type_init()

2018-08-23 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The return value of PyTuple_GetSlice() at 
https://github.com/python/cpython/blob/48ce4897f8f8d91d948ecd1241ffab002df2be9e/Objects/typeobject.c#L2297
 is not checked for NULL and then dereferenced.

Reported by Svace static analyzer.

Currently, PyTuple_GetSlice() for a zero-length slice can fail only if it's the 
first zero-length tuple ever created. If there is an invariant that at least 
one zero-length tuple exists on any type_init() call, the check may be replaced 
with an assert.

--
components: Interpreter Core
messages: 323951
nosy: berker.peksag, izbyshev, pitrou, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Objects/typeobject.c: Missing NULL check in type_init()
type: behavior
versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34477>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34468] An always-false condition in range_repr() from Objects/rangeobject.c

2018-08-22 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

range_repr() contains the following snippet:

/* Check for special case values for printing.  We don't always
   need the step value.  We don't care about errors
   (it means overflow), so clear the errors. */
istep = PyNumber_AsSsize_t(r->step, NULL);
if (istep != 1 || (istep == -1 && PyErr_Occurred())) {
PyErr_Clear();
}

The second condition in 'if' statement is always false (reported by Svace 
static analyzer). Moreover, the comment is incorrect. It implies that any error 
from PyNumber_AsSsize_t() means that an overflow occurred, but according to its 
implementation and documentation, any error is guaranteed *not* to be an 
OverflowError if the second parameter is NULL (the result is clamped on 
overflow).

In practice, 'range' invariants currently guarantee that no error can occur in 
this case because 'r->step' is a subtype of int (enforced in validate_step()). 
So the 'if' statement can be replaced either with an assert:

assert(!(istep == -1 && PyErr_Occurred()))

or with a conservative check:

if (istep == -1 && PyErr_Occurred())) {
// may also add assert(!PyErr_ExceptionMatches( PyExc_OverflowError);
return NULL;
}

Suggestions are appreciated.

--
components: Interpreter Core
messages: 323912
nosy: izbyshev, ncoghlan, serhiy.storchaka
priority: normal
severity: normal
status: open
title: An always-false condition in range_repr() from Objects/rangeobject.c
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34468>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34481] Different behavior of C and Python impls of datetime.strftime with non-UTF-8-encodable strings

2018-08-23 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The C datetime implementation uses PyUnicode_AsUTF8AndSize() in wrap_strftime() 
and rejects strings containing surrogate code points (0xD800 - 0xDFFF) since 
they can't be encoded in UTF-8. On the other hand, the pure-Python datetime 
implementation doesn't have this restriction:

>>> import sys
>>> sys.modules['_datetime'] = None # block C implementation
>>> from datetime import time
>>> time().strftime('\ud800')
'\ud800'
>>> del sys.modules['datetime']
>>> del sys.modules['_datetime']
>>> from datetime import time
>>> time().strftime('\ud800')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 
0: surrogates not allowed

--
components: Extension Modules
messages: 323963
nosy: belopolsky, izbyshev, pitrou, serhiy.storchaka, taleinat
priority: normal
severity: normal
status: open
title: Different behavior of C and Python impls of datetime.strftime with 
non-UTF-8-encodable strings
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34481>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34482] datetime: Tests for potential crashes due to non-UTF-8-encodable strings

2018-08-23 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
pull_requests: +8355

___
Python tracker 
<https://bugs.python.org/issue34482>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34482] datetime: Tests for potential crashes due to non-UTF-8-encodable strings

2018-08-23 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

This is a follow-up of #34454. 'datetime' extension module attempts to encode 
input strings into UTF-8 in several places, which requires special care because 
some valid Python strings can't be represented in UTF-8. It makes sense to add 
more tests for methods dealing with strings.

Note that my PR doesn't attempt to deal with #34481. In cases where behavior 
differs between C and Python datetime impls the tests check only for  absence 
of crashes.

--
components: Extension Modules
messages: 323964
nosy: belopolsky, izbyshev, p-ganssle, serhiy.storchaka, taleinat
priority: normal
severity: normal
status: open
title: datetime: Tests for potential crashes due to non-UTF-8-encodable strings
type: security
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34482>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34482] datetime: Tests for potential crashes due to non-UTF-8-encodable strings

2018-08-23 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8353
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34482>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34454] datetime: NULL dereference in fromisoformat() on PyUnicode_AsUTF8AndSize() failure

2018-08-23 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
pull_requests: +8354

___
Python tracker 
<https://bugs.python.org/issue34454>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34468] An always-false condition in range_repr() from Objects/rangeobject.c

2018-08-23 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8357
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34468>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34468] An always-false condition in range_repr() from Objects/rangeobject.c

2018-08-23 Thread Alexey Izbyshev


Alexey Izbyshev  added the comment:

Thank you, Brett, I've submitted a PR.

--

___
Python tracker 
<https://bugs.python.org/issue34468>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34454] datetime: NULL dereference in fromisoformat() on PyUnicode_AsUTF8AndSize() failure

2018-08-21 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

A failure of PyUnicode_AsUTF8AndSize() in various fromisoformat() functions in 
Modules/_datetimemodule.c leads to NULL dereference due to the missing check, 
e.g.:

>>> from datetime import date
>>> date.fromisoformat('\ud800')
Segmentation fault (core dumped)

This is similar to msg123474. The missing NULL check was reported by Svace 
static analyzer.

While preparing tests for this issue, I've discovered a deeper problem. The C 
datetime implementation uses PyUnicode_AsUTF8AndSize() in several places, 
making some functions reject strings containing surrogate code points (0xD800 - 
0xDFFF) since they can't be encoded in UTF-8. On the other hand, the 
pure-Python datetime implementation doesn't have this restriction. For example:

>>> import sys
>>> sys.modules['_datetime'] = None # block C implementation
>>> from datetime import time
>>> time().strftime('\ud800')
'\ud800'
>>> del sys.modules['datetime']
>>> del sys.modules['_datetime']
>>> from datetime import time
>>> time().strftime('\ud800')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 
0: surrogates not allowed

My PR (coming soon) doesn't address this difference but focuses on fixing the 
immediate problem instead. Suggestions are appreciated.

--
components: Extension Modules
messages: 323844
nosy: belopolsky, izbyshev, p-ganssle, serhiy.storchaka
priority: normal
severity: normal
status: open
title: datetime: NULL dereference in fromisoformat() on 
PyUnicode_AsUTF8AndSize() failure
type: crash
versions: Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34454>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34454] datetime: NULL dereference in fromisoformat() on PyUnicode_AsUTF8AndSize() failure

2018-08-21 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8322
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34454>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34493] Objects/genobject.c: Missing NULL check in compute_cr_origin()

2018-08-24 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The return value of PyTuple_New() is not checked for NULL at 
https://github.com/python/cpython/blob/ef8861c112ed1dac9351958c121bc24ca4ecdb08/Objects/genobject.c#L1130
 and then dereferenced.

--
components: Interpreter Core
messages: 324022
nosy: berker.peksag, izbyshev, njs, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Objects/genobject.c: Missing NULL check in compute_cr_origin()
type: behavior
versions: Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34493>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34493] Objects/genobject.c: Missing NULL check in compute_cr_origin()

2018-08-24 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8383
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34493>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34492] Python/coreconfig.c: Missing NULL check in _Py_wstrlist_copy()

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
pull_requests: +8395

___
Python tracker 
<https://bugs.python.org/issue34492>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34492] Python/coreconfig.c: Missing NULL check in _Py_wstrlist_copy()

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
versions: +Python 3.7

___
Python tracker 
<https://bugs.python.org/issue34492>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34492] Python/coreconfig.c: Fix _Py_wstrlist_copy()

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
title: Python/coreconfig.c: Missing NULL check in _Py_wstrlist_copy() -> 
Python/coreconfig.c: Fix _Py_wstrlist_copy()

___
Python tracker 
<https://bugs.python.org/issue34492>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34501] PyType_FromSpecWithBases: spec->name is dereferenced before checking for NULL

2018-08-25 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

In the following snippet from PyType_FromSpecWithBases() in 
Objects/typeobject.c, spec->name is dereferenced by strrchr() but then is 
checked for NULL:

/* Set the type name and qualname */
s = strrchr(spec->name, '.');
if (s == NULL)
s = (char*)spec->name;
else
s++;

[snip]

type->tp_name = spec->name;
if (!type->tp_name)
goto fail;

This was reported by Svace static analyzer.

If I were to check spec->name first, what error should I report to the caller? 
Is something like the following OK?

if (spec->name == NULL) {
PyErr_SetString(PyExc_SystemError,
   "Type spec does not define the name field.");
goto fail;
}

--
components: Interpreter Core
messages: 324073
nosy: berker.peksag, izbyshev, serhiy.storchaka
priority: normal
severity: normal
status: open
title: PyType_FromSpecWithBases: spec->name is dereferenced before checking for 
NULL
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34501>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34502] Docs of sys.exit() mention utf8_mode for an unclear reason

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8400
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34502>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34502] Docs of sys.exit() mention utf8_mode for an unclear reason

2018-08-25 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

Docs for sys.exit() contain the following note added in 
https://github.com/python/cpython/blame/1871a4a35123b278443b960255cb14314ce196f6/Doc/library/sys.rst#L336

Changed in version 3.7: Added utf8_mode attribute for the new -X utf8 flag.

Does it belong to the adjacent docs for sys.flags (though they mention 
utf8_mode too)?

--
assignee: docs@python
components: Documentation
messages: 324086
nosy: docs@python, izbyshev, vstinner
priority: normal
severity: normal
status: open
title: Docs of sys.exit() mention utf8_mode for an unclear reason
versions: Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34502>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34503] Reference leak in PyErr_SetObject()

2018-08-25 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

'exc_value' reference is leaked at 
https://github.com/python/cpython/blob/5f79b50763d687aeeed8edcb4efcc7ac9f8fa186/Python/errors.c#L113

--
components: Interpreter Core
messages: 324099
nosy: benjamin.peterson, berker.peksag, izbyshev, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Reference leak in PyErr_SetObject()
type: resource usage
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34503>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34504] PySequence_Check: argument is dereferenced and then checked for NULL

2018-08-25 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The argument of PySequence_Check() is dereferenced at 
https://github.com/python/cpython/blob/5f79b50763d687aeeed8edcb4efcc7ac9f8fa186/Objects/abstract.c#L1511
 and then checked for NULL.

Reported by Svace static analyzer.

Since other PySomething_Check() functions seem to expect a non-NULL argument, I 
suggest to remove the NULL check instead of moving it.

--
components: Interpreter Core
messages: 324101
nosy: benjamin.peterson, berker.peksag, izbyshev, serhiy.storchaka
priority: normal
severity: normal
status: open
title: PySequence_Check: argument is dereferenced and then checked for NULL
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34504>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34501] PyType_FromSpecWithBases: spec->name is dereferenced before checking for NULL

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8402
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34501>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34504] PySequence_Check: argument is dereferenced and then checked for NULL

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8407
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34504>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34501] PyType_FromSpecWithBases: spec->name is dereferenced before checking for NULL

2018-08-25 Thread Alexey Izbyshev


Alexey Izbyshev  added the comment:

@enedil It seems that all later check either set a message or simply propagate 
the error set by a called function. Did you mean any specific later check?

--

___
Python tracker 
<https://bugs.python.org/issue34501>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34503] Reference leak in PyErr_SetObject()

2018-08-25 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8406
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34503>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34492] Python/coreconfig.c: Missing NULL check in _Py_wstrlist_copy()

2018-08-24 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The return value of PyMem_RawMalloc() is not checked for NULL at 
https://github.com/python/cpython/blob/ef8861c112ed1dac9351958c121bc24ca4ecdb08/Python/coreconfig.c#L71.

Reported by Svace static analyzer.

--
components: Interpreter Core
messages: 324021
nosy: berker.peksag, izbyshev, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Python/coreconfig.c: Missing NULL check in _Py_wstrlist_copy()
type: behavior
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34492>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34492] Python/coreconfig.c: Missing NULL check in _Py_wstrlist_copy()

2018-08-24 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8382
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34492>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34512] Document platform-specific strftime() behavior for non-ASCII format strings

2018-08-26 Thread Alexey Izbyshev

New submission from Alexey Izbyshev :

If a format string contains code points outside of ASCII range, time.strftime() 
can behave in four different ways depending on the platform, the current locale 
and the code points:

* raise a UnicodeEncodeError
* return an empty string
* for surrogates in \uDC80-\uDCFF range, replace them with different code 
points in the output (potentially mangling nearby parts of the output as well)
* round-trip them correctly

Some examples:

* Linux (glibc 2.27):
Python 3.6.4 (default, Jan 03 2018, 13:52:55) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import time, locale
>>> locale.getlocale()
('en_US', 'UTF-8')
>>> time.strftime('\x80')
'\x80'
>>> time.strftime('\u044f')
'я' # '\u044f'
>>> time.strftime('\ud800')
'\ud800'
>>> time.strftime('\udcff')
'\udcff'
>>> locale.setlocale(locale.LC_CTYPE, 'C')
'C'
>>> time.strftime('\x80')
'\x80'
>>> time.strftime('\u044f')
'я' # '\u044f'
>>> time.strftime('\ud800')
'\ud800'
>>> time.strftime('\udcff')
'\udcff'

* macOS 10.13.6 and FreeBSD 11.1:
Python 3.7.0 (default, Jul 23 2018, 20:22:55)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import time, locale
>>> locale.getlocale()
('en_US', 'UTF-8')
>>> time.strftime('\x80')
'\x80'
>>> time.strftime('\u044f')
'я' # '\u044f'
>>> time.strftime('\ud800')
''
>>> time.strftime('\udcff')
''
>>> locale.setlocale(locale.LC_CTYPE, 'C')
'C'
>>> time.strftime('\x80')
'\x80'
>>> time.strftime('\u044f')
''
>>> time.strftime('\ud800')
''
>>> time.strftime('\udcff')
''

* Windows 8.1:
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit 
(AMD64)] on win32
>>> import time, locale
>>> locale.getlocale()
(None, None)
>>> time.strftime('\x80')
'\x80'
>>> time.strftime('\u044f')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'locale' codec can't encode character '\u044f' in position 
0: encoding error
>>> time.strftime('\ud800')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'locale' codec can't encode character '\ud800' in position 
0: encoding error
>>> time.strftime('\udcff')
'y' # '\xff'
>>> locale.setlocale(locale.LC_CTYPE, '')
'Russian_Russia.1251'
>>> time.strftime('\x80')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'locale' codec can't encode character '\x80' in position 0: 
encoding error
>>> time.strftime('\u044f')
'я' # '\u044f'
>>> time.strftime('\ud800')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'locale' codec can't encode character '\ud800' in position 
0: encoding error
>>> time.strftime('\udcff')
'я' # '\u044f'

The reasons of such differences are the following:
* Reliance on either wcsftime() or strftime() from the C library depending on 
the platform.
* For strftime(), the input is encoded into the charset of the current locale 
with 'surrogateescape' error handler, and the output is decoded back in the 
same way.
* Different handling of code points which are unrepresentable in the charset of 
the current locale by glibc and macOS/FreeBSD.

I suggest to at least document that the format string, despite being an 'str', 
requires special care if it contains non-ASCII code points.

The 'datetime' module docs warn about the locale-dependent output, but only 
with regard to particular format specifiers [1].

I'll submit a draft PR. Suggestions are welcome.

[1] 
https://docs.python.org/3.7/library/datetime.html#strftime-and-strptime-behavior

--
assignee: docs@python
components: Documentation
messages: 324136
nosy: belopolsky, docs@python, izbyshev, p-ganssle, taleinat
priority: normal
severity: normal
status: open
title: Document platform-specific strftime() behavior for non-ASCII format 
strings
type: enhancement
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34512>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34482] datetime: Tests for potential crashes due to non-UTF-8-encodable strings

2018-08-28 Thread Alexey Izbyshev


Alexey Izbyshev  added the comment:

Yes, I've referenced the relevant message from #6697 in #34454.

The specific test you've referenced should be changed after #34481 is fixed -- 
it highlights inconsistency between C and Python implementations.

--

___
Python tracker 
<https://bugs.python.org/issue34482>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34435] Missing NULL check in unicode_encode_ucs1()

2018-08-19 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

A possible NULL return on out-of-memory condition at 
Objects/unicodeobject.c:6835 is not handled.

Reported by Svace static analyzer.

--
components: Interpreter Core
messages: 323762
nosy: inada.naoki, izbyshev, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Missing NULL check in unicode_encode_ucs1()
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34435>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34436] Overallocation is never disabled in _PyBytes_FormatEx()

2018-08-19 Thread Alexey Izbyshev


Change by Alexey Izbyshev :


--
keywords: +patch
pull_requests: +8305
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue34436>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue34436] Overallocation is never disabled in _PyBytes_FormatEx()

2018-08-19 Thread Alexey Izbyshev


New submission from Alexey Izbyshev :

The condition for disabling overallocation at 225b055/Objects/bytesobject.c:822 
is always false.

Reported by Svace static analyzer.

--
components: Interpreter Core
messages: 323763
nosy: inada.naoki, izbyshev, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Overallocation is never disabled in _PyBytes_FormatEx()
type: performance
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34436>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



  1   2   3   4   >