Re: [Python-ideas] Was `os.errno` undocumented?
On 29/05/2018 23:58, Guido van Rossum wrote: > Have we received complaints about other cases? Whenever this breaks > code for which I am responsible I just blush and fix it, I don't > complain. Repeating the same warning typically just causes warning > fatigue rather than helping anyone. > Maybe what we need is to add a, possibly optional, or suppressible, warning whenever the import system encounters an implicit/indirect import? If an import that is working because the package we are importing it from has imported it from elsewhere, but it is not included in the __init__ for that package, then code authors will get some warning before the breakage and any blushes will really be due. If I was getting something along the lines of: Warning: Implicit Import in MyCode.py line 16 - import of os.errno is unsupported and may stop working on any update consider direct import. Then I would try to address it at the current version. I would be happy to try to implement this and think that the performance impact should be low. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
On Wed, May 30, 2018 at 9:11 AM, Greg Ewing wrote: > BTW, I wouldn't argue that Python shouldn't provide things > that are only useful to root. While writing setuid utilities > in Python is a bad idea for lots of reasons, I don't think > there's anything wrong with becoming root by another means > and then running a Python program that you know well enough > to trust. I'd go further. Once a shell script gets longer than about a page or two of code, it often needs to be rewritten in a different language, and Python is well situated to be that language. That doesn't change when the script is to be run as root. I've written many Python scripts to do sysadminning jobs for me - usually one-shot scripts, but also some that stick around. Since I wrote the scripts myself, the trust issue doesn't come up; I trust the Python interpreter the same way that I trust /bin/bash. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
On 5/29/2018 6:06 AM, Petr Viktorin wrote: Python 3.7 removes the undocumented internal import `os.errno`. Among a couple of hundred others, including some from idlelib. We consider that a an implementation detail, which can be changed *without notice* even in a bugfix release. We core developers occasionally run a linter on our code, or someone does it for us. Those of us who like clean code attend to warnings such as 'unused import'. I removed a couple of idlelib inports because of someone else running flake8. Projects that depend on it are incorrect and should be fixed. I think we can agree that better warning of 'use of stdlib internal import' would be great. On bpo-33666, there's a debate on whether the removal should be mentioned in release notes, on the grounds that it broke some projects, is used in quire a few tutorials/books/examples, and it's been working since Python 2.5 or so. Since I agree that *this* removal may possibly have the greatest impact, and that the purpose of What's New is to help users, I am glad Guido spoke up to say 'do it'. See my reponse to his post for a suggested entry. But here's the thing: the more I think about this, the less I consider `os.errno` as "undocumented". Here's what I consider a reasonable path a programmer might go through: # Where do I find errno values? # Maybe it's in `os`, like all other basic platform bindings? >>> import os >>> os.err os.errno os.error( >>> help(os.errno) Help on built-in module errno: ... # Yup, There it is! Suppose a naive beginner who confuses 'IDLE' with 'python' (they exist!) thinks "Where can I find re functions? How about 'pyshell' >>> import idlelib.pyshell as ps >>> ps.r >>> help(ps.re) Help on module re: ! Is that reasoning sound? Do you claim that os.errno is more documented (by the 'official' docs and help) than idlelib.pyshell.re? -- Terry Jan Reedy ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
BTW, I wouldn't argue that Python shouldn't provide things that are only useful to root. While writing setuid utilities in Python is a bad idea for lots of reasons, I don't think there's anything wrong with becoming root by another means and then running a Python program that you know well enough to trust. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
Steven D'Aprano wrote: Look closely at my example: the file ownership recursively changed from steve.steve to steve.users. You're quite right. I hadn't realised that chown can be used to change the group as well as the user. It's permitted to change the group to one that the user is a member of. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
Have we received complaints about other cases? Whenever this breaks code for which I am responsible I just blush and fix it, I don't complain. Repeating the same warning typically just causes warning fatigue rather than helping anyone. On Tue, May 29, 2018 at 3:53 PM, Terry Reedy wrote: > On 5/29/2018 12:09 PM, Guido van Rossum wrote: > >> Yes. What Steve says at the end. Let's be kind and mention this one on >> the release notes. But not all the others. >> > > How about adding a general reminder. In Porting to 3.7: > > "Imports into a module are a private implementation detail unless > otherwise noted. Private imports can be removed when not needed. (See PEP > 8.) For example, the os module no longer needs the errno module, so > 'import errno' was removed for 3.7. If you accessed 'errno' as 'os.errno', > add 'import errno' to your code and remove the 'os.' prefix. The same > applies to the 200 other unneeded imports removed in 3.7." > > Since this is a repeated problem, and a repeated drain on us core > developers, a version of this should be repeated in every What's New. > > -- > Terry Jan Reedy > > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
On 5/29/2018 12:09 PM, Guido van Rossum wrote: Yes. What Steve says at the end. Let's be kind and mention this one on the release notes. But not all the others. How about adding a general reminder. In Porting to 3.7: "Imports into a module are a private implementation detail unless otherwise noted. Private imports can be removed when not needed. (See PEP 8.) For example, the os module no longer needs the errno module, so 'import errno' was removed for 3.7. If you accessed 'errno' as 'os.errno', add 'import errno' to your code and remove the 'os.' prefix. The same applies to the 200 other unneeded imports removed in 3.7." Since this is a repeated problem, and a repeated drain on us core developers, a version of this should be repeated in every What's New. -- Terry Jan Reedy ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Mashup the existing statement grammars to capture predicates
Ah. Nice one. I'll look through that. -- Carl Smith carl.in...@gmail.com On 29 May 2018 at 19:52, Chris Angelico wrote: > On Wed, May 30, 2018 at 4:48 AM, Carl Smith wrote: > > Nah?? > > Nah. It's already been discussed at interminable length as part of PEP > 572. Feel free to browse that document. > > ChrisA > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] shutil zero-copy and exotic filesystems
On Tue, May 29, 2018 at 10:17 AM Barry Scott wrote: > On Tuesday, 29 May 2018 16:42:13 BST Eric Fahlgren wrote: > > Maybe you are suffering from TCP windows scaling not work well enough? > Thanks for the tip, I'll have to mention that to our IT infrastructure guys and see if they can check it out. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Mashup the existing statement grammars to capture predicates
On Wed, May 30, 2018 at 4:48 AM, Carl Smith wrote: > Nah?? Nah. It's already been discussed at interminable length as part of PEP 572. Feel free to browse that document. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Mashup the existing statement grammars to capture predicates
Nah??
-- Carl Smith
carl.in...@gmail.com
On 24 May 2018 at 19:24, Carl Smith wrote:
> This is another suggestion for new syntax for assigning a name to the value
> of the predicate in an if, elif or while statement. It still uses `as` for
> its keyword, but with (more flexible) params instead of a direct
> assignment.
>
> It mashes up the if/while, def/class and for-in grammars, so it still looks
> like Python, and boils down to this:
>
> if|elif|while as ():
>
> If the params contain one simple name (the required minimum), the value of
> the predicate is assigned to that name. In any other case, the value must
> be a sequence, which gets unpacked:
>
> while input('$ ').split() as (command, *args):
>
> if run(command, parse(args)) as (result): render(result)
> else: sys.exit()
>
> -- Carl Smith
> carl.in...@gmail.com
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] shutil zero-copy and exotic filesystems
On Tuesday, 29 May 2018 16:42:13 BST Eric Fahlgren wrote: snip... > On our WAN, which has a VPN endpoint 3000 miles from our office, routing > back to a test server another 2000 miles inside the network (tracert shows > 12-15 hops, 200 ms latency, arrrg), copying is slow no matter what: we are > lucky to see 40 Mbps on a connection that has the slowest link section at > 100 Mbps. > This is a guess on my part, I may be way off base. Maybe you are suffering from TCP windows scaling not work well enough? This page claims there are bugs in the windows implementations. Its also claimed, elsewhere, that some middle boxes mess up TCP windows scaling. http://web.archive.org/web/20120217135039/http://fasterdata.es.net:80/ fasterdata/host-tuning/ms-windows Wikipedia has a good description of TCP windows scalling. Barry ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
Honestly, despite the occasional use case(1), I'm not sure that this is a
battery we need in the stdlib. Nobody seems too excited about writing the
code, and when the operation is needed, shelling out to the system chown is
not too onerous. (Ditto for chmod.)
(1) Not even sure that a use case was shown -- it was just shown that the
operation is not necessarily useless.
On Tue, May 29, 2018 at 5:57 AM, Nick Coghlan wrote:
> On 29 May 2018 at 06:23, Giampaolo Rodola' wrote:
>
>> ...as in (not tested):
>>
>> def _rchown(dir, user, group):
>> for root, dirs, files in os.walk(dir, topdown=False):
>> for name in files:
>> chown(os.path.join(root, name), user, group)
>>
>> def chown(path, user=None, group=None, recursive=False):
>> if recursive and os.path.isdir(path):
>> _rchown(dir, user, group)
>> ...
>>
>> It appears like a common enough use case to me ("chown -R path").
>> Thoughts?
>>
>
> https://bugs.python.org/issue13033 is a long-open RFE for this, proposing
> to add it as "shutil.chowntree" (naming inspired by "shutil.rmtree" and
> "shutil.copytree").
>
> The "walkdir" project I mention on that PR has been on hiatus for a few
> years now (aside from a bit of activity to get a new release out in 2016
> with several contributed fixes), but the main point of the comment where I
> mentioned it still stands: the hard part of designing recursive state
> modification APIs is deciding what to do when an operation fails after
> you've already made changes to the state of the disk.
>
> shutil.rmtree fortunately provides some good precedent there, but it does
> mean this feature would need to be implemented as its own API, rather than
> as an option on shutil.chown.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
--
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
Yes. What Steve says at the end. Let's be kind and mention this one on the release notes. But not all the others. On Tue, May 29, 2018, 07:57 Steven D'Aprano wrote: > On Tue, May 29, 2018 at 05:37:06PM +0300, Serhiy Storchaka wrote: > > > We have removed over 100 module attributes in 3.6 [2] and over 200 > > module attributes in 3.7 [3]. > > > > Some of these removals broke third-part projects [4], even removals of > > underscored names. > > Yes, people will rely on undocumented features. Even when you tell them > not to. > > > If we will document the removal of os.errno, we > > should to document the removal of at least other names for which there > > are known examples of breaking third-party projects, e.g. > > re._pattern_type, uuid._uuid_generate_time and several typing attributes. > > I disagree. I think that _leading underscore names can be removed > without mentioning the removal. If it breaks code, too bad. Anyone using > an explicitly named _private name has only themselves to blame. > > But non-public names *without* a leading underscore are in a grey area, > since they *look* like public names. > > I think we can afford to be kind by documenting such removals -- > especially since it may help to educate people that imported modules > without a leading underscore are still considered implementation > details. > > > -- > Steve > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] shutil zero-copy and exotic filesystems
I have been doing speed testing on our network frequently enough over the last five year that I have a good feel for these changes. I have been benchmarking various means for copying files of 1.0-500 MB between workstations on our 1 Gbps LAN to local servers, and on our WAN which was just upgraded two months ago from a 7x T1 (11.5 Mbps) to 100 Mbps fiber. This is an all Windows network, workstations are Win7/10 and servers are Win Server 2003 (!!!), 2008, 2012 and 2016. The difference between open(dst).write(open(src).read()) and CopyFileW can be pretty signficant, especially for older Win server versions. I see maybe 4x speedup using CopyFileW to those machines with larger files, so it's a big win. On newer Windows Server versions it become sort of a wash, with both ways working better than anything else (much faster than command line utilities like DOS "copy" and Cygwin "cp", or Windows file explorer drag and drop, and very much faster than a C-code write(read()) loop). Here's an example I just ran for a Win Server 2003 copy. 2018-05-29T07:45:09 PDT ---Using CopyFileW Copy 'xmas.mpg' to 'p:/build/' (28,872,704 bytes) 1/3 - 0.27 s ( 905.2 Mbps) 2/3 - 0.25 s ( 983.9 Mbps) 3/3 - 0.26 s ( 968.6 Mbps) Theoretical: 0.25 s at 1000.0 Mbps Best time : 0.25 s at 983.9 Mbps ---Using write(read()) Copy 'xmas.mpg' to 'p:/build/' (28,872,704 bytes) 1/3 - 1.47 s ( 169.9 Mbps) 2/3 - 1.21 s ( 205.0 Mbps) 3/3 - 1.22 s ( 203.4 Mbps) Theoretical: 0.25 s at 1000.0 Mbps Best time : 1.21 s at 205.0 Mbps On our WAN, which has a VPN endpoint 3000 miles from our office, routing back to a test server another 2000 miles inside the network (tracert shows 12-15 hops, 200 ms latency, arrrg), copying is slow no matter what: we are lucky to see 40 Mbps on a connection that has the slowest link section at 100 Mbps. The biggest improvement we saw on WAN copies was when Ben Hoyt built scandir on top of Windows primitives. We were doing a copytree using os.walk that would take almost 20 (twenty) minutes just to build the tree before it copied the first file. When scandir was first released, I rewrote my code with fingers crossed that it would help a bit. On my first test, it took something like 15-20 seconds to build the tree and start copying. I was sure I had screwed up and broken it, but was quite shocked when I saw that it was in fact working correctly. Bottom line is +1 on switching to OS-specific copy mechanisms. Python is already the best* language for this sort of work (I've tried C/C++ and C# as alternatives), and this will make it even better. *Best for me in this particular case is "resulting code is fastest, ease of implementation is secondary." On Tue, May 29, 2018 at 1:59 AM Giampaolo Rodola' wrote: > Whops, I hit "send" too soon. Sorry about the messed up message. > > On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' > wrote: > >> Hello, >> I've been working on a patch which speeds up shutil.copy* operations for >> all 3 major platforms (Linux, Windows, OSX): >> https://bugs.python.org/issue33671 >> Since the speedup is quite consistent I'd love to see this merged in, but >> considering shutil.copy* is quite crucial I wanted to hear other folk's >> opinion first. Attached patch attempts to use platform-specific zero-copy >> syscalls [1] by default and fallbacks on using plain read() / write() >> variant in case of immediate failure. In theory this should work fine, in >> practice I haven't tested it on exotic (e.g. network) filesystems. In order >> to diminish risks of breakage I think if it would make sense to: >> - add a global shutil.NO_ZEROCOPY variable defaulting to False >> - add a "no_zerocopy" argument to all functions involving a copy >> (copyfile(). copy(), copy2(), copytree(), move()) >> >> Thoughts? >> >> [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows) >> >> >> >> >> since the matter is a bit sensitive in terms of potential breakage on >> exotic / untested (e.g. network) filesystems I want to raise some attention >> about: >> https://bugs.python.org/issue33671 >> Attached patch attempts to use platform-specific zero-copy syscalls by >> default and fallbacks on using plain read() / write() copy in case of >> immediate failure. In order to diminish risks I think it would make sense >> to: >> - add a global shutil.NO_ZEROCOPY variable defaulting to False >> - add a "no_zerocopy" argument to all functions involving a copy >> (copyfile(). copy(), copy2(), copytree(), move()) >> >> Thoughts? >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> >> > > > -- > Giampaolo - http://grodola.blogspot.com > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org
Re: [Python-ideas] Was `os.errno` undocumented?
On Tue, May 29, 2018 at 05:37:06PM +0300, Serhiy Storchaka wrote: > We have removed over 100 module attributes in 3.6 [2] and over 200 > module attributes in 3.7 [3]. > > Some of these removals broke third-part projects [4], even removals of > underscored names. Yes, people will rely on undocumented features. Even when you tell them not to. > If we will document the removal of os.errno, we > should to document the removal of at least other names for which there > are known examples of breaking third-party projects, e.g. > re._pattern_type, uuid._uuid_generate_time and several typing attributes. I disagree. I think that _leading underscore names can be removed without mentioning the removal. If it breaks code, too bad. Anyone using an explicitly named _private name has only themselves to blame. But non-public names *without* a leading underscore are in a grey area, since they *look* like public names. I think we can afford to be kind by documenting such removals -- especially since it may help to educate people that imported modules without a leading underscore are still considered implementation details. -- Steve ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
29.05.18 13:06, Petr Viktorin пише: As you can see, the built-in documentation does not contain *any* warnings against using `os.errno`. You might think the fact that it's called "errno" and not "os.errno" is a red flag, but it's not, really -- "os.path" is (on my system) named "posixpath", yet "os.path" is the name to use. Unlike to os.path, which is explicitly documented and included in __all__, os.abc, os.errno, os.sys and os.st are not documented and not included in __all__. They were always an implementations detail. os.errno was added as a side effect of issue1755179. [1] While most people might prefer searching docs.python org to `help()`, editors are getting better and better to presenting introspection and the built-in docs, so more and more people are preferring `pydoc`-ish docs to going online. I don't think we can reasonably expect people who used built-in help as above to go back and check that Python's official docs *do not* contain `os.errno`. Effectively, while `os.errno` is not very *discoverable* using official docs alone, I don't think calling it *undocumented* is fair. So, removing it without notice is not very friendly to our users. We have removed over 100 module attributes in 3.6 [2] and over 200 module attributes in 3.7 [3]. Some of these removals broke third-part projects [4], even removals of underscored names. If we will document the removal of os.errno, we should to document the removal of at least other names for which there are known examples of breaking third-party projects, e.g. re._pattern_type, uuid._uuid_generate_time and several typing attributes. [1] https://bugs.python.org/issue1755179 [2] https://bugs.python.org/msg317881 [3] https://bugs.python.org/msg317876 [4] https://bugs.python.org/msg317998 ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
On 29 May 2018 at 20:06, Petr Viktorin wrote: > Is that reasoning sound? > Should our policy on removing internal imports take that into account? > As Steven noted, the normative answer to this is in PEP 8: https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces Since `os.errno` is a transitive import, is not included in `os.__all__`, and isn't documented in the library reference, it's considered an implementation detail that can be removed without a deprecation period. That said, it should still be mentioned in the "Porting to Python 3.7" section of the What's New guide, with the fix being to switch to "import errno" in any affected code. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
On 29 May 2018 at 06:23, Giampaolo Rodola' wrote:
> ...as in (not tested):
>
> def _rchown(dir, user, group):
> for root, dirs, files in os.walk(dir, topdown=False):
> for name in files:
> chown(os.path.join(root, name), user, group)
>
> def chown(path, user=None, group=None, recursive=False):
> if recursive and os.path.isdir(path):
> _rchown(dir, user, group)
> ...
>
> It appears like a common enough use case to me ("chown -R path").
> Thoughts?
>
https://bugs.python.org/issue13033 is a long-open RFE for this, proposing
to add it as "shutil.chowntree" (naming inspired by "shutil.rmtree" and
"shutil.copytree").
The "walkdir" project I mention on that PR has been on hiatus for a few
years now (aside from a bit of activity to get a new release out in 2016
with several contributed fixes), but the main point of the comment where I
mentioned it still stands: the hard part of designing recursive state
modification APIs is deciding what to do when an operation fails after
you've already made changes to the state of the disk.
shutil.rmtree fortunately provides some good precedent there, but it does
mean this feature would need to be implemented as its own API, rather than
as an option on shutil.chown.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
> I agree that it's technically well within our rights to remove it > without notice. > But ... PEP8? A style guide defines what is a CPython implementation > detail? That's not a place to point to while saying "told you so!" -- > perhaps "sorry for the inconvenience" would be more appropriate :) https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces "Any backwards compatibility guarantees apply only to public interfaces." "All undocumented interfaces should be assumed to be internal." "Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules." > > And maybe this will encourage linters to flag this risky usage, if they > > aren't already doing so. > How do linters find out what's an internal import, and what's correct > usage (like os.path)? It is `correct usage` only when it is in __all__. >>> "path" in os.__all__ True >>> "errno" in os.__all__ False Regards, -- INADA Naoki ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
(Apologies if you received an empty/unfinished e-mail here earlier; I hit Send by mistake.) On 05/29/18 13:57, Steven D'Aprano wrote: On Tue, May 29, 2018 at 12:06:42PM +0200, Petr Viktorin wrote: Hello, Python 3.7 removes the undocumented internal import `os.errno`. We consider that a an implementation detail, which can be changed *without notice* even in a bugfix release. Projects that depend on it are incorrect and should be fixed. PEP 8 makes it clear that imports are implementation details unless explicitly documented otherwise: Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules. This decision dates back to 2013: https://mail.python.org/pipermail/python-dev/2013-July/127284.html so it has been around for a while, long enough that linters ought to be enforcing it, and people ought to know about it. If not, that's a failure of the linter and/or the coder. On bpo-33666, there's a debate on whether the removal should be mentioned in release notes, on the grounds that it broke some projects, is used in quire a few tutorials/books/examples, and it's been working since Python 2.5 or so. I don't see why there should be a debate about mentioning it in the release notes. There's no harm in adding a few lines: "os.errno is a non-public implementation detail, as described in PEP 8, and has been removed. Import the errno module instead." Being an implementation detail, we aren't *required* to do so, but given that (so you say) a few tutorials etc use it, I think it is the kind thing to do. But here's the thing: the more I think about this, the less I consider `os.errno` as "undocumented". Here's what I consider a reasonable path a programmer might go through: [...] All of this is reasonable, and I'm sympathetic, *except* that it is officially documented policy that imports are implementation details unless otherwise stated. I agree that it's technically well within our rights to remove it without notice. But ... PEP8? A style guide defines what is a CPython implementation detail? That's not a place to point to while saying "told you so!" -- perhaps "sorry for the inconvenience" would be more appropriate :) If os.errno was gone and there was no easy fix, I think we could be merciful and rethink the decision. But there is an easy fix: import errno directly instead. And maybe this will encourage linters to flag this risky usage, if they aren't already doing so. How do linters find out what's an internal import, and what's correct usage (like os.path)? As you can see, the built-in documentation does not contain *any* warnings against using `os.errno`. It doesn't need to. And of course, help(os.errno) *cannot* warn about os.errno, since it only receives the errno module as argument, not the expression you used to pass it. Indeed. That's unfortunate, but it is a reason for us to be careful, or perhaps find/document a better policy for handling these. I'm not looking for evidence to justify the changes; I'm looking for ways to be more friendly to our users -- most of whom have not read all of the docs. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
On 05/29/18 13:57, Steven D'Aprano wrote: On Tue, May 29, 2018 at 12:06:42PM +0200, Petr Viktorin wrote: Hello, Python 3.7 removes the undocumented internal import `os.errno`. We consider that a an implementation detail, which can be changed *without notice* even in a bugfix release. Projects that depend on it are incorrect and should be fixed. PEP 8 makes it clear that imports are implementation details unless explicitly documented otherwise: Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules. This decision dates back to 2013: https://mail.python.org/pipermail/python-dev/2013-July/127284.html so it has been around for a while, long enough that linters ought to be enforcing it, and people ought to know about it. If not, that's a failure of the linter and/or the coder. On bpo-33666, there's a debate on whether the removal should be mentioned in release notes, on the grounds that it broke some projects, is used in quire a few tutorials/books/examples, and it's been working since Python 2.5 or so. I don't see why there should be a debate about mentioning it in the release notes. There's no harm in adding a few lines: "os.errno is a non-public implementation detail, as described in PEP 8, and has been removed. Import the errno module instead." Being an implementation detail, we aren't *required* to do so, but given that (so you say) a few tutorials etc use it, I think it is the kind thing to do. But here's the thing: the more I think about this, the less I consider `os.errno` as "undocumented". Here's what I consider a reasonable path a programmer might go through: [...] All of this is reasonable, and I'm sympathetic, *except* that it is officially documented policy that imports are implementation details unless otherwise stated. If os.errno was gone and there was no easy fix, I think we could be merciful and rethink the decision. But there is an easy fix: import errno directly instead. And maybe this will encourage linters to flag this risky usage, if they aren't already doing so. As you can see, the built-in documentation does not contain *any* warnings against using `os.errno`. It doesn't need to. And of course, help(os.errno) *cannot* warn about os.errno, since it only receives the errno module as argument, not the expression you used to pass it. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
On Tue, May 29, 2018 at 12:06:42PM +0200, Petr Viktorin wrote: > Hello, > Python 3.7 removes the undocumented internal import `os.errno`. > We consider that a an implementation detail, which can be changed > *without notice* even in a bugfix release. Projects that depend on it > are incorrect and should be fixed. PEP 8 makes it clear that imports are implementation details unless explicitly documented otherwise: Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules. This decision dates back to 2013: https://mail.python.org/pipermail/python-dev/2013-July/127284.html so it has been around for a while, long enough that linters ought to be enforcing it, and people ought to know about it. If not, that's a failure of the linter and/or the coder. > On bpo-33666, there's a debate on whether the removal should be > mentioned in release notes, on the grounds that it broke some projects, > is used in quire a few tutorials/books/examples, and it's been working > since Python 2.5 or so. I don't see why there should be a debate about mentioning it in the release notes. There's no harm in adding a few lines: "os.errno is a non-public implementation detail, as described in PEP 8, and has been removed. Import the errno module instead." Being an implementation detail, we aren't *required* to do so, but given that (so you say) a few tutorials etc use it, I think it is the kind thing to do. > But here's the thing: the more I think about this, the less I consider > `os.errno` as "undocumented". Here's what I consider a reasonable path a > programmer might go through: [...] All of this is reasonable, and I'm sympathetic, *except* that it is officially documented policy that imports are implementation details unless otherwise stated. If os.errno was gone and there was no easy fix, I think we could be merciful and rethink the decision. But there is an easy fix: import errno directly instead. And maybe this will encourage linters to flag this risky usage, if they aren't already doing so. > As you can see, the built-in documentation does not contain *any* > warnings against using `os.errno`. It doesn't need to. And of course, help(os.errno) *cannot* warn about os.errno, since it only receives the errno module as argument, not the expression you used to pass it. -- Steve ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Was `os.errno` undocumented?
On Tue, May 29, 2018 at 8:06 PM, Petr Viktorin wrote: > I don't think we can reasonably expect people who used built-in help as > above to go back and check that Python's official docs *do not* contain > `os.errno`. Effectively, while `os.errno` is not very *discoverable* using > official docs alone, I don't think calling it *undocumented* is fair. > So, removing it without notice is not very friendly to our users. > At no point did you show that it's *documented*. What you've shown is that it's an unadorned name. There are a lot of those around: >>> antigravity.webbrowser >>> base64.struct >>> dis.types If people are attempting "from os import errno", they need to learn to type "import errno" instead. While it's nice to hide those modules with "import errno as _errno", failure to do so is a bug, not an API feature. The change might break code, but ANY change might break code: https://xkcd.com/1172/ ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] shutil zero-copy and exotic filesystems
Whops, I hit "send" too soon. Sorry about the messed up message. On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' wrote: > Hello, > I've been working on a patch which speeds up shutil.copy* operations for > all 3 major platforms (Linux, Windows, OSX): > https://bugs.python.org/issue33671 > Since the speedup is quite consistent I'd love to see this merged in, but > considering shutil.copy* is quite crucial I wanted to hear other folk's > opinion first. Attached patch attempts to use platform-specific zero-copy > syscalls [1] by default and fallbacks on using plain read() / write() > variant in case of immediate failure. In theory this should work fine, in > practice I haven't tested it on exotic (e.g. network) filesystems. In order > to diminish risks of breakage I think if it would make sense to: > - add a global shutil.NO_ZEROCOPY variable defaulting to False > - add a "no_zerocopy" argument to all functions involving a copy > (copyfile(). copy(), copy2(), copytree(), move()) > > Thoughts? > > [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows) > > > > > since the matter is a bit sensitive in terms of potential breakage on > exotic / untested (e.g. network) filesystems I want to raise some attention > about: > https://bugs.python.org/issue33671 > Attached patch attempts to use platform-specific zero-copy syscalls by > default and fallbacks on using plain read() / write() copy in case of > immediate failure. In order to diminish risks I think it would make sense > to: > - add a global shutil.NO_ZEROCOPY variable defaulting to False > - add a "no_zerocopy" argument to all functions involving a copy > (copyfile(). copy(), copy2(), copytree(), move()) > > Thoughts? > > -- > Giampaolo - http://grodola.blogspot.com > > > -- Giampaolo - http://grodola.blogspot.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] shutil zero-copy and exotic filesystems
Hello, I've been working on a patch which speeds up shutil.copy* operations for all 3 major platforms (Linux, Windows, OSX): https://bugs.python.org/issue33671 Since the speedup is quite consistent I'd love to see this merged in, but considering shutil.copy* is quite crucial I wanted to hear other folk's opinion first. Attached patch attempts to use platform-specific zero-copy syscalls [1] by default and fallbacks on using plain read() / write() variant in case of immediate failure. In theory this should work fine, in practice I haven't tested it on exotic (e.g. network) filesystems. In order to diminish risks of breakage I think if it would make sense to: - add a global shutil.NO_ZEROCOPY variable defaulting to False - add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move()) Thoughts? [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows) since the matter is a bit sensitive in terms of potential breakage on exotic / untested (e.g. network) filesystems I want to raise some attention about: https://bugs.python.org/issue33671 Attached patch attempts to use platform-specific zero-copy syscalls by default and fallbacks on using plain read() / write() copy in case of immediate failure. In order to diminish risks I think it would make sense to: - add a global shutil.NO_ZEROCOPY variable defaulting to False - add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move()) Thoughts? -- Giampaolo - http://grodola.blogspot.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
On Tue, May 29, 2018 at 06:29:50PM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >But it doesn't matter: regular users can call chown -R: > > Only if you're not actually telling it to change anything. That's not correct. Look closely at my example: the file ownership recursively changed from steve.steve to steve.users. > % ls -l foo.txt > -rw-r--r-- 1 greg users 1 29 May 18:19 foo.txt > % chown greg foo.txt > % chown fred foo.txt > chown: foo.txt: Operation not permitted And yet Python provides a chown function despite this alleged uselessness. Maybe it's not quite so useless as you think? Here's a thought... maybe sometimes people actually do run Python scripts as root? And as the comments to this Stackoverflow post explain: https://unix.stackexchange.com/questions/119229/can-not-chown-a-file-from-my-user-to-another-user the ability for unprivileged users to change ownership to another user is configurable under POSIX. > So you have to be root in order to do anything *useful* > with it. Fortunately that is not correct, but even if it were, that's no reason to not allow chown to apply recursively. -- Steve ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add shutil.chown(..., recursive=False)
Steven D'Aprano wrote: But it doesn't matter: regular users can call chown -R: Only if you're not actually telling it to change anything. % ls -l foo.txt -rw-r--r-- 1 greg users 1 29 May 18:19 foo.txt % chown greg foo.txt % chown fred foo.txt chown: foo.txt: Operation not permitted So you have to be root in order to do anything *useful* with it. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
