[issue29829] Documentation lacks clear warning of subprocess issue with pythonw

[Eryk Sun]

Eryk Sun added the comment:

An exception (not a crash) is possible if a standard handle is invalid and a 
Popen call tries to replace one or two of the other standard handles (e.g. 
stdin is invalid and you pass the argument stdout=PIPE). Note that subprocess 
uses the standard handles directly, so this problem is unrelated to Python's 
sys.std* streams. IMO, the presence of an invalid handle in the standard 
handles should not cause Popen to fail. This is an old problem, and it should 
have been addressed a long time ago by substituting a handle for \\.\NUL.

In Windows 8+ this isn't particular to running pythonw.exe. In older versions 
such as Windows 7, if you start pythonw.exe from a console application, then 
the parent's standard handle values are copied to the pythonw.exe child. But 
these handles are invalid because pythonw.exe doesn't automatically attach to 
its parent's console. (You have to manually call AttachConsole(-1) or 
AllocConsole(), but that's complicated because you may need to reset the CRT's 
standard file descriptors and FILE streams and Pythons sys.std* streams.)

Note that pythonw.exe doesn't delete or close the standard streams. They're 
normally set to None because, in the default case, the Windows standard handles 
aren't valid in pythonw.exe, and thus the CRT maps its standard file 
descriptors (0, 1, and 2) to INVALID_HANDLE_VALUE (i.e. -1 cast as a pointer). 
For example:

C:\Temp>pyw -2 -c "import msvcrt; open('test.txt', 
'w').write(str(msvcrt.get_osfhandle(1)))"

C:\Temp>type test.txt
18446744073709551614

You can override this by starting pythonw.exe with explicit standard handles. 
For example, the following redirects stderr (2) to stdout (1) and redirects 
stdout to a pipe:

C:\>set "script=import sys; print(sys.executable); print(sys.stdout); 
print(sys.stderr)"

C:\>2>&1 pyw -2 -c "%script%" | more
C:\Program Files\Python27\pythonw.exe
', mode 'w' at 0x021FB0C0>
', mode 'w' at 0x021FB150>

C:\>2>&1 pyw -3 -c "%script%" | more
C:\Program Files\Python36\pythonw.exe
<_io.TextIOWrapper name='' mode='w' encoding='cp1252'>
<_io.TextIOWrapper name='' mode='w' encoding='cp1252'>

--
components: +Windows
nosy: +eryksun, paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28415] PyUnicode_FromFormat integer format handling different from printf about zeropad

[Xiang Zhang]

Changes by Xiang Zhang :


--
keywords: +easy
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29820] Broken link to "GUI Programming with Python: QT Edition" book

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

Thanks Marco :) I merged your PR, and backported to 3.5 and 3.6.

There are some merge conflict with the 2.7 branch, so I decided not to bother 
with it.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed
versions:  -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29820] Broken link to "GUI Programming with Python: QT Edition" book

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
pull_requests: +562

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29820] Broken link to "GUI Programming with Python: QT Edition" book

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
pull_requests: +563

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29831] os.path.exists seems can not recgnize "~"

[Martin Panter]

Martin Panter added the comment:

I agree with Josh. This is how it is supposed to work.

os.system calls the shell (e.g. Bash) rather than running the "ls" program 
directly. Unix shells translate "~" to the home directory (as well as 
translating a lot of other stuff, e.g. spaces to separate CLI arguments).

But if you pass "~" to os.path.exists (or most other functions), that goes 
directly to the operating system and looks for a directory literally called 
"~". If you created a directory with that name, it should return True then.

--
nosy: +martin.panter
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29831] os.path.exists seems can not recgnize "~"

[Josh Rosenberg]

Josh Rosenberg added the comment:

That's because os.system is executing the command in a shell (which expands ~). 
Without shell support, ~ doesn't mean anything unless used with the APIs that 
specifically support it (e.g. os.path.expanduser).

You wanted os.path.exists(os.path.expanduser('~/.zshrc'))

This is not a bug.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29831] os.path.exists seems can not recgnize "~"

[quanyechavshuo]

New submission from quanyechavshuo:

os.system is ok to recgnize "~",but os.path.exists can not recgnize "~".

eg:

#1.py:
import os
os.system("ls -al ~/.zshrc")

python3 1.py

output:
-rw-r--r--  1 root  wheel  5391  3 14 18:12 /var/root/.zshrc


#2.py:
import os
a=os.path.exists("~/.zshrc")
print(a)

python3 2.py

output:
False

--
messages: 289740
nosy: quanyechavshuo
priority: normal
severity: normal
status: open
title: os.path.exists seems can not recgnize "~"
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29826] " don't work on Mac under IDLE

[Ned Deily]

Ned Deily added the comment:

> Is this restriction correct?

Yes

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29826] " don't work on Mac under IDLE

[Terry J. Reedy]

Terry J. Reedy added the comment:

Marco, did you see this message?
 (r"WARNING: The version of Tcl/Tk ({0}) in use may"
  r" be unstable.\n"
  r"Visit http://www.python.org/download/mac/tcltk/;
  r" for current information.".format(patchlevel))
Or is it somehow lost?

Ned, the version check for this message is only done for Cocoa.  Is this 
restriction correct?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29830] pyexpat.errors doesn't have __spec__ and __loader__ set

[Manuel Jacob]

New submission from Manuel Jacob:

The same applies to pyexpat.model.

It seems like pyexpat is the only builtin module which has submodules (errors, 
model).

Normally, as I understand it, the module gets imported given a spec and the 
import machinery ensures that this spec ends up in the __spec__ attribute of 
the module.  But in this case only pyexpat gets imported by importlib.  The 
submodules are added when initializing the module.

Also, importlib's BuiltinImporter assumes that a builtin module is never a 
package.  Is this reasonable in this case?

--
messages: 289737
nosy: mjacob
priority: normal
severity: normal
status: open
title: pyexpat.errors doesn't have __spec__ and __loader__ set

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29826] " don't work on Mac under IDLE

[Ned Deily]

Ned Deily added the comment:

Yes, this is a symptom of using the default Apple-supplied macOS Tcl/Tk.  See 
https://www.python.org/download/mac/tcltk/ for more information as the warning 
in IDLE suggests.  If you are using Python 3.6.0 downloaded from python.org, 
the simple solution is to install the latest Tcl/Tk 8.5.x (not 8.6.x) from 
ActiveState assuming your use is compatible with the ActiveTcl license.

--
nosy: +ned.deily
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Yury Selivanov]

Yury Selivanov added the comment:

BTW, why can't you use `pthread_atfork` in numpy?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29777] argparse arguments in main parser hide an argument in subparser

[Alan Evangelista]

Alan Evangelista added the comment:

s/Python 2.6/Python 2/ in last comment

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Antoine Pitrou]

Antoine Pitrou added the comment:

The use case is quite clear here.  The specific need to re-seed the Numpy PRNG 
has already come up in two different projects I work on: Numba and Dask.  I 
wouldn't be surprised if other libraries have similar needs.

If you want a reproducible RNG sequence, you should actually use a specific, 
explicit seed (and possibly instantiate a dedicated random state instead of 
using the default one).  When not using an explicit seed, people expect 
different random numbers regardless of whether a function is executed in one or 
several processes.

Note that multiprocessing *already* re-seeds the stdlib PRNG after fork, so 
re-seeding the Numpy PRNG is consistent with current behaviour.

About it being rarely used: the aim is not use by application developers but by 
library authors; e.g. Numpy itself could register the re-seeding callback, 
which would free users from doing it themselves.  It doesn't have to be used a 
lot to be useful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29777] argparse arguments in main parser hide an argument in subparser

[Alan Evangelista]

Alan Evangelista added the comment:

PA> In http://bugs.python.org/issue14910#msg204678 I suggest a subclassing 
patch that might work with Py2.

This solves my particular case. I do not use any argument with action='count', 
so the regression introduced by the new option does not affect me. Thanks!

PA> The patch is big enough that I hesitate to add it to Py2.  

I see 2 solutions:
- keep this issue opened until the bug with short options is fixed and we are 
more confident in fixing this in Py2
- close the issue with "won't fix" resolution and add the known issue to 
argparse documentation in Python 2.6 doc

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Nathaniel Smith]

Nathaniel Smith added the comment:

I think ideally on numpy's end we would reseed iff the RNG was unseeded. Now 
that I think about it I'm a little surprised that we haven't had more 
complaints about this, so I guess it's not a super urgent issue, but that would 
be an improvement over the status quo, I think.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29820] Broken link to "GUI Programming with Python: QT Edition" book

[Marco Buttu]

Changes by Marco Buttu :


--
pull_requests: +561

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Davin Potts]

Davin Potts added the comment:

Having a read through issue16500 and issue6721, I worry that this could again 
become bogged down with similar concerns.

With the specific example of NumPy, I am not sure I would want its random 
number generator to be reseeded with each forked process.  There are many 
situations where I very much need to preserve the original seed and/or current 
PRNG state.

I do not yet see a clear, motivating use case even after reading those two 
older issues.  I worry that if it were added it would (almost?) never get used 
either because the need is rare or because developers will more often think of 
how this can be solved in their own target functions when they first start up.  
The suggestion of a top-level function and Context method make good sense to me 
as a place to offer such a thing but is there a clearer use case?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29829] Documentation lacks clear warning of subprocess issue with pythonw

[R. David Murray]

R. David Murray added the comment:

A warning is not appropriate (we reserve those for things that are security 
related, pretty much). A sentence might be, though.  For example, we could 
change the initial discussion of the run function to say:

   This does not capture stdout or stderr by default. To do so, pass PIPE for 
the stdout and/or stderr arguments.  You *must* do this if you run your python 
program with pythonw on windows, since pythonw closes the standard streams.

Here I'm adding the windows sentence to the existing "This does not capture" 
sentence.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Antoine Pitrou]

Antoine Pitrou added the comment:

That issue seems to have stalled as it seems to have focussed on low-level 
APIs, and also because it is proposing a new module with the API question that 
entails.

Another possible stance is that os.fork() should be left as-is, as a low-level 
primitive, and this functionality should be provided by the higher-level 
multiprocessing module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28261] wrong error messages when using PyArg_ParseTuple to parse normal tuples

[Oren Milman]

Oren Milman added the comment:

as Serhiy pointed out in PR 668, here are some more functions that produce the
same kind of wrong error messages:
- Modules/timemodule.c - gettmarg()
- Modules/socketmodule.c:
* getsockaddrarg()
* socket_getnameinfo()

ISTM that searching for 'PyTuple_Check(' might be a good way to find more such
functions, as they sometimes raise an error in case a type other than tuple was
received (and after that, PyArg_ParseTuple is called).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Yury Selivanov]

Yury Selivanov added the comment:

Maybe a better way would be to proceed with http://bugs.python.org/issue16500?

--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29636] Specifying indent in the json.tool command

[Bob Ippolito]

Bob Ippolito added the comment:

Probably the best thing we could do here is to mirror the options available in 
similar tools, such as jq: https://stedolan.github.io/jq/manual/#Invokingjq

The relevant options here would be:

--indent
--tab
--compact-output
--sort-keys

The default indent in jq is 2, which I tend to prefer these days, but maybe 4 
is still appropriate given PEP 8:

$ echo '[{}, {"a": "b"}, 2, 3, 4]' | jq
[
  {},
  {
"a": "b"
  },
  2,
  3,
  4
]


This is how jq interprets --compact-output:

$ echo '[{}, {"a": "b"}, 2, 3, 4]' | jq --compact-output
[{},{"a":"b"},2,3,4]


I do not think that it's worth having the command-line tool cater to people 
that want to indent in other ways (e.g. using a string that isn't all spaces or 
a single tab).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29820] Broken link to "GUI Programming with Python: QT Edition" book

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

Thanks, Marco. That sounds good. Let's just remove the reference to the books 
from both PyQt and wxPython.

--
keywords: +easy
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

[Charalampos Stratakis]

Charalampos Stratakis added the comment:

In order to reproduce:

Apply the python.patch from bz1268226_reproducer2.tar.gz

Compile python

Run the reproduce4.py from bz1268226_reproducer2.tar.gz

As indicated by the reproducer, the status returned by os.wait() for the child 
is 139.

I will refine a bit the patch and work on a PR.

--
Added file: http://bugs.python.org/file46730/python.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29829] Documentation lacks clear warning of subprocess issue with pythonw

[Steve Barnes]

New submission from Steve Barnes:

When running under pythonw, or pyinstaller with the -w flag, modules that use 
subprocess calls such as popen, run, etc. will crash if the default 
`stdout=None, stderr=None` behaviour is used rather than PIPE. This is an 
obscure problem which is very hard to debug yet there is no warning in the 
documentation on this.

I would like to suggest adding a :warning:`stdout=None, stderr=None` must not 
be used in any of the calls in this module when running under pythonw due to 
the lack of sys.stdout & sys.stderr in that case. Please use `stdout=PIPE, 
stderr=PIPE` instead.

A patch against the default branch would be:
diff -r 4243df51fe43 Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rstFri Feb 10 14:19:36 2017 +0100
+++ b/Doc/library/subprocess.rstThu Mar 16 16:56:24 2017 +
@@ -33,6 +33,13 @@
 function for all use cases it can handle. For more advanced use cases, the
 underlying :class:`Popen` interface can be used directly.
 
+.. warning:: Do not use default parameters on Windows with pythonw.
+
+   As pythonw deletes `sys.stdout` & `sys.stderr` the use of the default 
+   parameters, `stdout=None, stderr=None,`, which defaults to being
+   `stdout=sys.stdout, stderr=sys.stderr,` may cause unexpected crashes
+   it is recommended to use `stdout=PIPE, stderr=PIPE,` instead.
+
 The :func:`run` function was added in Python 3.5; if you need to retain
 compatibility with older versions, see the :ref:`call-function-trio` section.

--
assignee: docs@python
components: Documentation
messages: 289722
nosy: Steve Barnes, docs@python
priority: normal
severity: normal
status: open
title: Documentation lacks clear warning of subprocess issue with pythonw
type: behavior
versions: Python 2.7, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +njs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29828] Allow registering after-fork initializers in multiprocessing

[Antoine Pitrou]

New submission from Antoine Pitrou:

Currently, multiprocessing has hard-coded logic to re-seed the Python random 
generator (in the random module) whenever a process is forked.  This is present 
in two places: `Popen._launch` in `popen_fork.py` and `serve_one` in 
`forkserver.py` (for the "fork" and "forkserver" spawn methods, respectively).

However, other libraries would like to benefit from this mechanism.  For 
example, Numpy has its own random number generator that would also benefit from 
re-seeding after fork().  Currently, this is solvable using 
multiprocessing.Pool which has an `initializer` argument.  However, 
concurrent.futures' ProcessPool does not offer such facility; nor do other ways 
of launching child processes, such as (simply) instantiating a new Process 
object.

Therefore, I'd like to propose adding a new top-level function in 
multiprocessing (and also a new Context method) to register a new initializer 
function for use after fork().  That way, each library can add its own 
initializers if desired, freeing users from the burden of doing so in their 
applications.

--
components: Library (Lib)
messages: 289721
nosy: davin, pitrou, sbt
priority: normal
severity: normal
stage: needs patch
status: open
title: Allow registering after-fork initializers in multiprocessing
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29820] Broken link to "GUI Programming with Python: QT Edition" book

[Marco Buttu]

Marco Buttu added the comment:

Hi Mariatta, all the other seealso entries (PyGObject, PySide, ...) have 
references to some specific books and tutorials.  No one has a reference to the 
wiki, maybe because there is already a reference to it (at the end of the 
page).  IMHO we can keep the PyQt description consistent with the other GUIs 
descriptions, and we can fix the broken URL just removing the reference to the 
outdated book.  What do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29826] " don't work on Mac under IDLE

[Brett Cannon]

Changes by Brett Cannon :


--
assignee:  -> terry.reedy
components: +IDLE
nosy: +terry.reedy
title: " don't work on Mac -> " don't work on Mac under IDLE

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29827] os.path.exists() returns False for certain file name

[Eryk Sun]

Eryk Sun added the comment:

In a string literal, '\t' represents a tab character (ordinal 9). Windows 
filenames cannot contain control characters [1], i.e. ordinals 1-31. For path 
literals I recommend using forward slashes and normalizing via 
os.path.normpath, e.g. filepath = 
os.path.normpath('C:/Users/mavri/Desktop/proba/testni.wav'). 

FYI, the way exists() is written handles any OSError as False:

def exists(path):
try:
os.stat(path)
except OSError:
return False
return True

In your case, stat() fails with ERROR_INVALID_NAME (123):

>>> os.stat('\testni.wav')
Traceback (most recent call last):
  File "", line 1, in 
OSError: [WinError 123] The filename, directory name, or volume label
syntax is incorrect: '\testni.wav'

[1]: https://msdn.microsoft.com/en-us/library/aa365247#naming_conventions

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29827] os.path.exists() returns False for certain file name

[Marko Mavrinac]

Marko Mavrinac added the comment:

I am sorry, I didn't realize \t affected how the path was recognized.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29827] os.path.exists() returns False for certain file name

[Marko Mavrinac]

New submission from Marko Mavrinac:

I have two files in two different folders, both on desktop.
If I try using os.path.exists() on both of them, it returns True for one file 
and False for the other file.
After renaming file that I got False from, I get returned True and when I 
rename it back, it returns False again.

File name causing the problem is "testni.wav", I assume it's anything starting 
with "test", but I'm sure you guys will know better. Thank you

Detailed description with screenshots can be seen here: 
http://stackoverflow.com/questions/42834408/os-path-exists-returning-false-for-one-and-true-for-another-file-both-files-e

--
components: Windows
messages: 289717
nosy: Marko Mavrinac, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: os.path.exists() returns False for certain file name
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

[Daniel Birnstiel]

Daniel Birnstiel added the comment:

Currently using
Python 3.6.0 (default, Mar  4 2017, 12:32:34) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin

> So, somehow the print() statement is blocking, which I have /no/ idea how to 
> go about debugging. I assume there's a lock /in/ the print statement function 
> call, and I'm probably going to look into wrapping both the print() call and 
> the multiprocessing.Process() call  execution in a single, shared 
> multiprocessing lock, but that
> seems like a very patchwork solution to something that should just work.

I am currently having a similar issue where I get a deadlock to a stdout.flush 
call (which I assume is called when printing). Flush internally appears to 
acquire a lock which is getting stuck in the subprocess. 

This is the backtrace I was able to obtain through lldb, showing the waiting 
for a lock after the flush-call:

* thread #1: tid = 0x77c27d, 0x7fffe33c9c86 
libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
frame #0: 0x7fffe33c9c86 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x7fffe34b396a libsystem_pthread.dylib`_pthread_cond_wait + 
712
frame #2: 0x0001021ecad8 Python`PyThread_acquire_lock_timed + 256
frame #3: 0x00010221cc2f Python`_enter_buffered_busy + 169
frame #4: 0x00010221ed36 Python`_io_BufferedWriter_write + 203
frame #5: 0x00010215448b Python`_PyCFunction_FastCallDict + 529
frame #6: 0x00010211b3f0 Python`_PyObject_FastCallDict + 237
frame #7: 0x00010211be9e Python`PyObject_CallMethodObjArgs + 240
frame #8: 0x00010222171a Python`_textiowrapper_writeflush + 150
  * frame #9: 0x000104a8 Python`_io_TextIOWrapper_flush + 239


Is there any update on this issue or any solution to avoid deadlocking without 
wrapping every fork/print/logging call with a multiprocessing (or billiard in 
my case) lock?

--
nosy: +Birne94

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29636] Specifying indent in the json.tool command

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

It's not just the support burden. It is also a burden of learning and 
remembering new API.

The support burden itself is not tiny too. It includes careful designing, 
writing the implementation and test (the more options you have the more 
combinations you need to test), increasing running time of tests (CLI tests are 
slow!), and you need to return to that after adding every new feature for 
testing compatibility with them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29636] Specifying indent in the json.tool command

[Daniel Himmelstein]

Daniel Himmelstein added the comment:

@serhiy.storchaka I totally understand the desire to keep json.tool simple. 
However, given the description of json.tool in the documentation (below), I 
think an indentation option is within scope:

> The json.tool module provides a simple command line interface to validate and 
> pretty-print JSON objects.

Indentation/newlines are a fundamental aspect of "pretty-printing". Right now I 
rarely use json.tool, since indent=4 is too extreme from a visual and file size 
perspective. Instead I prefer `indent=2` (or even `indent=1`) and I now have to:

1. create a python script to set my desired input
2. make sure every environment has access to this python script (the real 
annoyance)

Currently, json.tool has a --sort-keys argument, which I think is great. 
--sort-keys is also an essential feature from my perspective (bpo-21650). So in 
short, I think json.tool is close to being a really useful utility but falls a 
bit short without an indentation option.

Given that json.tool exists, I think it makes sense to take steps to make sure 
it's actually relevant as a json reformatter. Given this motivation, I'm not 
opposed to adding --compact, if we're confident it will be forward compatible 
with the json.dump API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29636] Specifying indent in the json.tool command

[R. David Murray]

R. David Murray added the comment:

Easier, but if we do it in the tool, then it is done for everyone and they 
don't *each* have to spend that "less time" writing their own script.  And 
--indent and --compact are both useful for debugging/hand testing, since it 
allows you to generate the input your code is expecting, and input your code 
might not be expecting.

On the other hand, I'm not contributing much these days, so I'm not in a good 
position to be suggesting additions to the support burden :(.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29636] Specifying indent in the json.tool command

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I'm not particularly interested in this feature. Adding two or three options 
looks excessive. Python is a programming language and it is easy to write a 
simple script for your needs. Much easier than implement the general command 
line interface that supports all options of programming API.

--
nosy: +bob.ippolito, ezio.melotti, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19521] Parallel build race condition on AIX since python-2.7

[Michael Haubenwallner]

Changes by Michael Haubenwallner :


--
pull_requests: +560

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29826] " don't work on Mac

[Marco Viscito]

New submission from Marco Viscito:

When typing the ' key or the " key on the IDLE Python application for macOS, 
the application. I think it might have something to do with that beta version 
of Tcl/Tk (8.5.9) as Python says it is 'unstable'.

--
files: Screen Shot 2017-03-16 at 09.34.26.png
messages: 289711
nosy: Marco Viscito
priority: normal
severity: normal
status: open
title: " don't work on Mac
type: crash
versions: Python 3.6
Added file: http://bugs.python.org/file46729/Screen Shot 2017-03-16 at 
09.34.26.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29825] PyFunction_New() not validate code object

[LCatro]

New submission from LCatro:

PyFunction_New() not validate code object ,so we can make a string object to 
fake code object

This is Python ByteCode :

  LOAD_CONST '\x41\x41\x41\x41'
  MAKE_FUNCTION 0

in source code ,we can see that string object trace to variant v

TARGET(MAKE_FUNCTION)
{
v = POP(); /* code object */  <=  now it is a string object
x = PyFunction_New(v, f->f_globals);  <=  using in there

and than ,we making a string object will taking into PyFunction_New()

PyFunction_New(PyObject *code, PyObject *globals)
{
PyFunctionObject *op = PyObject_GC_New(PyFunctionObject,
_Type);
static PyObject *__name__ = 0;
if (op != NULL) {  <=  there just check new alloc object point but not 
checking the argument code's python type (actually it is TYPE_CODE) ..
PyObject *doc;
PyObject *consts;
PyObject *module;
op->func_weakreflist = NULL;
Py_INCREF(code);
op->func_code = code;
Py_INCREF(globals);
op->func_globals = globals;
op->func_name = ((PyCodeObject *)code)->co_name;
Py_INCREF(op->func_name);  <=  it will make an arbitrary address inc by 
one ..

Opcode MAKE_CLOSURE similar too ..

TARGET(MAKE_CLOSURE)
{
v = POP(); /* code object */
x = PyFunction_New(v, f->f_globals);

poc and crash detail in update file

--
components: Interpreter Core
files: inc_by_one.rar
messages: 289710
nosy: imso666
priority: normal
severity: normal
status: open
title: PyFunction_New() not validate code object
type: security
versions: Python 2.7
Added file: http://bugs.python.org/file46728/inc_by_one.rar

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29824] Hostname validation in SSL match_hostname()

[Christian Heimes]

Changes by Christian Heimes :


--
nosy: +alex, dstufft, janssen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29824] Hostname validation in SSL match_hostname()

[Christian Heimes]

Christian Heimes added the comment:

I don't see 1) as a problem. You won't be able to resolve these names in DNS, 
would you?

Regarding 2). Yes, it would be beneficial to have more elaborate checks to 
protect against wildcard attacks like *.com. However Python is not a browser. 
It's really hard to do it right and even harder to keep the rule set up to 
date. Some TLDs like .uk have sublevel namespaces, e.g. co.uk. *.co.uk is also 
invalid.

The problem is going to shift anyway. For Python 3.7 I'm going to deprecate 
support for OpenSSL < 1.0.2 and use OpenSSL's hostname verification code 
instead of ssl.match_hostname().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29824] Hostname validation in SSL match_hostname()

[Suphannee]

New submission from Suphannee:

1. Allowing attempting to match invalid hostname
According to domain name specification in RFC 1035, only alphanumeric, dot and
hyphen are valid characters in domain name. We observe that
the function match_hostname() in Lib/ssl.py allows other special characters 
(e.g., '=', '&') in hostname when attempting to match with certificate 
commonName (CN)/subjectAltName DNS. An example would be matching hostname
"example.a=.com" with certificate CN/DNS "example.a=.com" or CN/DNS 
"*.a=.example.com". Ensuring that CN/DNS with invalid characters are rejected, 
will make the library more robust against attacks that utilize such characters.

2. Matching wildcard in public suffix
As noted in section 7.2 of RFC 6125, some wildcard location specifications are
not clear. We found that the function allows wildcard over public suffix in
certificate as well as allows attempting to match in hostname verification,
e.g., matches hostname "google.com" and "example.com" with
certificate CN/DNS "*.com". This is not an RFC violation, but we might benefit 
from implementing the check, for example "*.one_label" is restricted. A better 
option will be having a list of all TLD's and check against it.

Thanks.

--
assignee: christian.heimes
components: SSL
messages: 289708
nosy: christian.heimes, ssivakorn
priority: normal
severity: normal
status: open
title: Hostname validation in SSL match_hostname()
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com