Re: [Python-Dev] pathlib - current status of discussions

Ethan Furman stoneleaf.us> writes: > Do we allow bytes to be returned from os.fspath()? If yes, then do we > allow bytes from __fspath__()? De-lurking. Especially since the ultimate goal is better interoperability, I feel like an implementation that people can play with would help guide the

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On Tue, Apr 12, 2016 at 8:57 AM, Sven R. Kunze wrote: > As an example: time.sleep takes a number of seconds (notice the primitive > datatype just like a string) and does not take timedelta. > > Why don't we add datetime.timedelta support to time.sleep? Very same thing. yup --

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On Mon, 11 Apr 2016 14:15:02 -0700 Ethan Furman wrote: > We've pretty decided that we have two options: > > 1. remove pathlib > 2. make the stdlib work with pathlib > > So we're trying to make option 2 work before falling back to option 1. I have been doing a lot of

Re: [Python-Dev] pathlib - current status of discussions

On Tue, Apr 12, 2016 at 9:32 AM, Koos Zevenhoven wrote: > > 1) A "proper" path object -- i.e. pathlib.Path or anything else that > > supports the path protocol. > > > > 2) the bytes that the OS actually needs. > > > > You do have a point there. But since bytes pathnames are

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

On Wed, Apr 13, 2016 at 2:15 AM, Ethan Furman wrote: > On 04/11/2016 04:43 PM, Victor Stinner wrote: >> >> Le 11 avr. 2016 11:11 PM, "Ethan Furman" a écrit : > > >>> So my concern in such a case is what happens if we pass this SE >>> string somewhere else: a UTF-8 file, or

Re: [Python-Dev] pathlib - current status of discussions

On Mon, Apr 11, 2016 at 10:40 PM, Greg Ewing wrote: > > So the ONLY thing >> you should do with it is pass it along to another low level system >> call. >> > > Not quite -- you can separate it into components and > work with them. Essentially the same set of

Re: [Python-Dev] pathlib - current status of discussions

On 04/12/2016 09:26 AM, Koos Zevenhoven wrote: So I'm, once again, posing this question (that I don't think got any reactions previously): Is there a significant audience for this new function, or is it enough to keep it a private function for the stdlib to use? Quite frankly, I expect the

Re: [Python-Dev] pathlib - current status of discussions

On Tue, Apr 12, 2016 at 7:19 PM, Chris Barker wrote: > > One more though came up just now: there are different level sof abstractions > and representations for paths. We don't want to make Path a subclass of > string, because Path is supposed to be a higher level

Re: [Python-Dev] pathlib - current status of discussions

On Tue, Apr 12, 2016 at 11:56 AM, Nick Coghlan wrote: > One possible way to address this concern would be to have the > underlying protocol be bytes/str (since boundary code frequently needs > to handle the paths-are-bytes assumption in POSIX), but offer an > "os.fspathname"

Re: [Python-Dev] pathlib - current status of discussions

On Tue, Apr 12, 2016 at 6:52 PM, Stephen J. Turnbull wrote: > > (A) Why does anybody need bytes out of a pathlib.Path (or other > __fspath__-toting, higher-level API) *inside* the boundary? Note > that the APIs in os (etc) *don't need* bytes because they are >

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On 4/12/2016 12:14, Sven R. Kunze wrote: I cannot remember us using another datetime library. So, I don't value this "advantage" as much as you do. They exist, and there are many cases where you would use a datetime library other than datetime for various reasons (integration in third party

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

On 04/12/2016 09:20 AM, Chris Angelico wrote: On Wed, Apr 13, 2016 at 2:15 AM, Ethan Furman latin1? I thought latin1 had a code point for 0-255, so how could using it raise an encoding error? Latin-1 / ISO-8859-1 defines a character for every byte, so any byte string will *decode*. It only

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

On Tue, Apr 12, 2016 at 9:20 AM, Chris Angelico wrote: > > latin1? I thought latin1 had a code point for 0-255, so how could using > it > > raise an encoding error? > > Latin-1 / ISO-8859-1 defines a character for every byte, so any byte > string will *decode*. It only defines

[Python-Dev] ping on issue 18378: locale.getdefaultlocale() fails on recent Mac OS X

Hi folks, There have been multiple reports of folks having failures on startup of matplotlib, which appears to be due to the most recent OS-X version setting the locale weirdly. This was identified last summer in this issue: http://bugs.python.org/issue18378 It looks like the issue was figured

[Python-Dev] List posting custom [was: current status of discussions]

The following is my opinion, as will become obvious, but it's based on over a decade of observing these lists, and other open source development lists. In a context where some core developers have unsubscribed from these lists, and others regularly report muting threads with a certain air of

Re: [Python-Dev] Not receiving bug tracker emails

On 4/4/2016 5:05 PM, Terry Reedy wrote: Since a few days, I am getting bug tracker emails again, in my Inbox. I just got a Rietveld review in the Inbox and I believe it went there directly instead of first to Junk. Thank you to whoever made the improvements. -- Terry Jan Reedy

Re: [Python-Dev] pathlib - current status of discussions

On 04/11/2016 10:14 PM, Chris Barker - NOAA Federal wrote: Consider os.path.join: Why in the world do the os.path functions need to work with Path objects? ( and other conforming objects) Because library XYZ that takes a path and wants to open it shouldn't have to care whether that path

[Python-Dev] Maybe, just maybe, pathlib doesn't belong.

Alexander Walters writes: > If there is headway being made, I do not see it. Filter out everything but the posts by Brett, and see if you still feel that way. (Other people have contributed[1], but that filter has about 20dB better S/N than the whole thread does.) Footnotes: [1] Brett may

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

INADA Naoki writes: > > Why not print(obj)? print(obj) will give mojibake by default if sys.getfilenameencoding() != sys.getdefaultencoding(). > > str() is normal high-level API, and __fspath__ and os.fspath() should be > > low level API. > > Normal users shouldn't use __fspath__ and

Re: [Python-Dev] pathlib - current status of discussions

On 12 April 2016 at 15:28, Stephen J. Turnbull wrote: > Donald Stufft writes: > > > I think yes and yes [__fspath__ and fspath should be allowed to > > handle bytes, otherwise] it seems like making it needlessly harder > > to deal with a bytes path > > It's not needless.

Re: [Python-Dev] pathlib - current status of discussions

On 12 April 2016 at 06:28, Stephen J. Turnbull wrote: > Donald Stufft writes: > > > I think yes and yes [__fspath__ and fspath should be allowed to > > handle bytes, otherwise] it seems like making it needlessly harder > > to deal with a bytes path > > It's not needless.

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 06:57:37PM +1000, Chris Angelico wrote: > And yes, you win if you get another module. Interestingly, you're > allowed to import urllib.parse, but not urllib itself; but "import > urllib.parse" makes urllib available - and, since modules inside > modules are blacklisted,

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On 11 April 2016 at 17:53, Jon Ribbens wrote: >> You're limiting the subset of Python that people can use, >> understood. And you're trying to ensure that people can't do "bad >> things". Again, understood. But what subset are you actually allowing, >> and what

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 8:06 PM, Jon Ribbens wrote: > On Tue, Apr 12, 2016 at 06:57:37PM +1000, Chris Angelico wrote: >> The sandbox code assumes that an attacker cannot create files in the >> current directory. > > If the attacker can create such files then the

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Sat, Apr 9, 2016 at 12:18 AM, Jon Ribbens wrote: > Anyway the code is at https://github.com/jribbens/unsafe > It requires Python 3.4 or later (it could probably be made to work on > Python 2.7 as well, but it would need some changes). Rather annoying point:

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On 11 April 2016 at 22:21, Sven R. Kunze wrote: > On 11.04.2016 23:08, Random832 wrote: >> >> On Mon, Apr 11, 2016, at 17:04, Sven R. Kunze wrote: >>> >>> PS: The only way out that I can imagine is to fix pathlib. I am not in >>> favor of fixing functions of "os" and "os.path" to

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 6:17 PM, Paul Moore wrote: > Well, I'll ask the obvious question, then. In allowing "import" did > you allow "import ctypes"? If so, then I win :-) Or did you explicitly > whitelist certain modules? And if so, which ones are they, and did I > succeed

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 06:28:34PM +1000, Chris Angelico wrote: > On Sat, Apr 9, 2016 at 12:18 AM, Jon Ribbens > wrote: > > Anyway the code is at https://github.com/jribbens/unsafe > > It requires Python 3.4 or later (it could probably be made to work on > >

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 1:14 PM, Jon Ribbens wrote: > On Tue, Apr 12, 2016 at 06:21:04AM -0400, Isaac Morland wrote: >> On Tue, 12 Apr 2016, Jon Ribbens wrote: >> >>This is still a massive game of whack-a-mole. >> > >> >No, it still isn't. If the names blacklist

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 01:38:09PM +0200, Maciej Fijalkowski wrote: > Jon, let me reiterate. You asked people to break it (that's the title > of the thread) and they did so almost immediately. Then you patched > the thing and asked them to break it again and they did. Now the > faulty assumption

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 14:18 GMT+02:00 Jon Ribbens : > The question is: with a minimal (or empty) set of builtins, and a > restriction on ast.Name and ast.Attribute nodes, can exec/eval be > made 'safe' so they cannot execute code outside the sandbox. According to multiple

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 02:16:57PM +0200, Victor Stinner wrote: > I read your code and the code of CPython. I found many issues. Thanks for your efforts. > Your "safe import" hides real functions with a proxy. Ok. But the code > of modules is still run in the real namespace, Yes, that was the

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 10:45:06PM +1000, Chris Angelico wrote: > On Tue, Apr 12, 2016 at 10:42 PM, Jon Ribbens > wrote: > > That's not a vulnerability, and it's something I already explicitly > > mentioned - if you can get a function to return an object's

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

I haven't been following this thread in detail, so perhaps I have missed something, but I have a question... On Tue, Apr 12, 2016 at 02:05:06PM +0200, Victor Stinner wrote: > You don't understand that even if the visible "Python scope", "Python > namespace", or call it as you want (the code

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 08:27:14PM +1000, Chris Angelico wrote: > On Tue, Apr 12, 2016 at 8:06 PM, Jon Ribbens > wrote: > > No, it still isn't. If the names blacklist had to keep being extended > > then you would be right, but that hasn't happened so far.

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 06:21:04AM -0400, Isaac Morland wrote: > On Tue, 12 Apr 2016, Jon Ribbens wrote: > >>This is still a massive game of whack-a-mole. > > > >No, it still isn't. If the names blacklist had to keep being extended > >then you would be right, but that hasn't happened so far.

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-08 16:18 GMT+02:00 Jon Ribbens : > I've made another attempt at Python sandboxing, which does something > which I've not seen tried before - using the 'ast' module to do static > analysis of the untrusted code before it's executed, to prevent most > of the

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 13:38 GMT+02:00 Maciej Fijalkowski : > (...) you end up with either a > completely unusable python (the python that can't run anything is > trivially secure) Yeah, that's the obvious question: what's the purpose of such very limited Python subset, for example

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 02:05:06PM +0200, Victor Stinner wrote: > 2016-04-12 13:10 GMT+02:00 Jon Ribbens : > > No, it's a matter of reducing the whitelist. I must admit that > > I don't understand in what way this is not already clear. Look: > > > > >>>

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 02:31:19PM +0200, Victor Stinner wrote: > Oh, I forgot to mention another vulnerability: you block access to > attributes by replacing getattr and by analyzing the AST. Ok, but one > more time, it's not enough. If you get access to obj.__dict__, you > will likely get access

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 9:10 PM, Jon Ribbens wrote: > On Tue, Apr 12, 2016 at 08:27:14PM +1000, Chris Angelico wrote: >> decimal.getcontext is a simple and obvious example of a way that >> global mutable objects can be accessed across the boundary. There is >> no

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 10:42 PM, Jon Ribbens wrote: > On Tue, Apr 12, 2016 at 02:31:19PM +0200, Victor Stinner wrote: >> Oh, I forgot to mention another vulnerability: you block access to >> attributes by replacing getattr and by analyzing the AST. Ok, but one

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 13:10 GMT+02:00 Jon Ribbens : > No, it's a matter of reducing the whitelist. I must admit that > I don't understand in what way this is not already clear. Look: > > >>> len(unsafe._SAFE_MODULES) > 23 You don't understand that even if the visible

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

2016-04-12 14:16 GMT+02:00 Victor Stinner : > I read your code and the code of CPython. I found many issues. > (...) > The exploit is based on two things: > > * update_wrapper() is used to get the secret attribute using the real > getattr() function > * update_wrapper() +

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 10:49 PM, Jon Ribbens wrote: > On Tue, Apr 12, 2016 at 10:45:06PM +1000, Chris Angelico wrote: >> On Tue, Apr 12, 2016 at 10:42 PM, Jon Ribbens >> wrote: >> > That's not a vulnerability, and it's

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 11:12 PM, Steven D'Aprano wrote: > To give a concrete example... suppose that the C locale library is > unsafe. Further, let's suppose that the str.isdigit method calls code > from the C locale library, to determine whether or not the string is > made

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, 12 Apr 2016, Jon Ribbens wrote: This is still a massive game of whack-a-mole. No, it still isn't. If the names blacklist had to keep being extended then you would be right, but that hasn't happened so far. Whitelists by definition contain only a small, limited number of potential

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 11:03:11PM +1000, Chris Angelico wrote: > One of the key differences with scripts in web browsers is that there > *is* no "outer environment" to access. If you think that then I think you considerably misunderstand how modern browsers work. > Remember what I said about

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 11:12:27PM +1000, Steven D'Aprano wrote: > I can think of one possible threat. Suppose that the locale library > has a bug, so that calling "aardvark".isdigit seg faults, potentially > executing arbitrary C code, but at the very least crashing the > application. Is that

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

On Tue, Apr 12, 2016 at 01:40:57PM +, David Wilson wrote: > On Tue, Apr 12, 2016 at 11:12:27PM +1000, Steven D'Aprano wrote: > > I can think of one possible threat. Suppose that the locale library > > has a bug, so that calling "aardvark".isdigit seg faults, potentially > > executing arbitrary

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On 12.04.2016 00:56, Random832 wrote: Fully general re-dispatch from argument types on any call to a function that raises TypeError or NotImplemented? [e.g. call Path.__missing_func__(os.open, path, mode)] Have pathlib monkey-patch things at import? Implicit conversion. No, thanks. On Mon,

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On 12.04.2016 12:41, Paul Moore wrote: As your thoughts appear to have been triggered by my comments, I feel I should clarify. 1. I like pathlib even as it is right now, and I'm strongly -1 on removing it. 2. The "external dependency" aspect of 3rd party solutions makes them far less useful to

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On Tue, Apr 12, 2016, at 10:52, Sven R. Kunze wrote: > On 12.04.2016 00:56, Random832 wrote: > > Fully general re-dispatch from argument types on any call to a function > > that raises TypeError or NotImplemented? [e.g. call > > Path.__missing_func__(os.open, path, mode)] > > > > Have pathlib

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

> On Apr 12, 2016, at 10:52 AM, Sven R. Kunze wrote: > > Path(...).open() is your friend then. I don't see why you need os.open. > > Refusing to upgrade it like saying, everything was better in the old days. So > let's use os.open instead of Path(...).open(). I think it was

Re: [Python-Dev] pathlib - current status of discussions

Sorry for disturbing this thread's harmony. On 12.04.2016 08:00, Ethan Furman wrote: On 04/11/2016 10:14 PM, Chris Barker - NOAA Federal wrote: Consider os.path.join: Why in the world do the os.path functions need to work with Path objects? ( and other conforming objects) Because

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On 12.04.2016 16:59, Random832 wrote: Strings are strings. Paths are paths. That's were the difference is. Yes but why aren't these both "things that you may want to use to open a file"? Because "things that you may want to use to open a file" is a bit vague and thus conceal the fact that

Re: [Python-Dev] pathlib - current status of discussions

On Tue, Apr 12, 2016 at 7:58 AM, Ethan Furman wrote: > Sticking points: > --- > > Do we allow bytes to be returned from os.fspath()? If yes, then do we allow > bytes from __fspath__()? > I would say No and No, on the basis that it's *far* easier to widen their

Re: [Python-Dev] pathlib - current status of discussions

Nick Coghlan writes: > One possible way to address this concern would be to have the > underlying protocol be bytes/str (since boundary code frequently > needs to handle the paths-are-bytes assumption in POSIX), What "needs"? As has been pointed out several times, with PEP 383 you can deal

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

one little note: On Tue, Apr 12, 2016 at 3:41 AM, Paul Moore wrote: > 4. There are further improvements that could be made to pathlib, > certainly, but again they are optional, and pathlib is fine without > them. > Exactly -- "improvements to pathlib" and "make the stdlib

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On Tue, Apr 12, 2016 at 7:54 AM, Sven R. Kunze wrote: > > My conclusion is that these changes are not optional and tweaking os, io > and shutil is just yet another workaround for a clean solution. :) > Is the clean solution to re-implement EVERYTHING in the stdlib that involves

Re: [Python-Dev] pathlib - current status of discussions

On 04/11/2016 02:58 PM, Ethan Furman wrote: Sticking points: --- Do we allow bytes to be returned from os.fspath()? If yes, then do we allow bytes from __fspath__()? On 04/11/2016 10:28 PM, Stephen J. Turnbull wrote: > In text applications, "bytes as carcinogen" is an apt

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

On 12.04.2016 18:04, Chris Barker wrote: On Tue, Apr 12, 2016 at 7:54 AM, Sven R. Kunze > wrote: My conclusion is that these changes are not optional and tweaking os, io and shutil is just yet another workaround for a clean solution. :)

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

On 04/11/2016 04:43 PM, Victor Stinner wrote: Le 11 avr. 2016 11:11 PM, "Ethan Furman" a écrit : So my concern in such a case is what happens if we pass this SE string somewhere else: a UTF-8 file, or over a socket, or into a database? Does this have issues that we wouldn't face if we just