[issue25596] regular files handled as directories in the glob module

Changes by Xavier de Gaye : Added file: http://bugs.python.org/file41571/glob_scandir_3.patch ___ Python tracker ___

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: FWIW I have followed the idea of having _iterdir() yielding the DirEntry entry instead of the name in Serhiy's patch. There is a slight performance gain. Now _glob0() and _glob1() do return a list of directories when dironly is true but there is now another

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: Actually the microbenchmarks comparison between glob_scandir_2.patch and glob_scandir_3.patch must be made by removing the call to entry.is_symlink() also in glob_scandir_2.patch. In that case the microbenchmarks give about the same results for each patch.

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: Adding a doc patch. -- Added file: http://bugs.python.org/file41533/glob_scandir_doc.patch ___ Python tracker ___

[issue25596] regular files handled as directories in the glob module

Guido van Rossum added the comment: (IOW once this patch has been applied maybe you can do the same for globbing in pathlib as requested in issue #26032.) -- nosy: +gvanrossum ___ Python tracker

[issue25596] regular files handled as directories in the glob module

STINNER Victor added the comment: Related issue: #26032 "Use scandir() to speed up pathlib globbing". -- ___ Python tracker ___

[issue25596] regular files handled as directories in the glob module

Changes by R. David Murray : -- nosy: -r.david.murray ___ Python tracker ___ ___

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: > and since _iterdir() results are always accumulated in a list Right, I failed to note that point. And so, since the file descriptor opened by os.scandir() is closed within each call to recursive _rlistdir(), then my other comment about EMFILE does not stand

[issue25596] regular files handled as directories in the glob module

Serhiy Storchaka added the comment: I think so too. I just wanted someone to confirmed that it is not overcautiousness. For now glob1() is used only in one place in the stdlib (issue16620). But there was other problem with previous patch, the same as with current implementation of os.walk()

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: I may be missing something, anyway here are few comments on the patch: Is the call to entry.is_symlink() in _iterdir() necessary since is_dir() follows symlinks ? If _iterdir() would yield the DirEntry instance instead of DirEntry.name, then _rlistdir() could

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: os.scandir() is called recursively in the last patch and the file descriptors are not closed until returning from the recursion. The glob functions should fail explicitly when scandir() raises OSERROR with posix errno set to EMFILE (The process has too many

[issue25596] regular files handled as directories in the glob module

Serhiy Storchaka added the comment: > os.scandir() is called recursively in the last patch and the file descriptors > are not closed until returning from the recursion. No, os.scandir() is called non-recursively in _iterdir(), and since _iterdir() results are always accumulated in a list, a

[issue25596] regular files handled as directories in the glob module

Changes by Serhiy Storchaka : -- dependencies: +Avoid using private function glob.glob1() in msi module and tools ___ Python tracker

[issue25596] regular files handled as directories in the glob module

Serhiy Storchaka added the comment: In general the patch LGTM. It allows to speed up glob by 5-10% in warn test. But much more gain we can achieve by using os.scandir(). Here are results of microbenchmarks: $ ./python -m timeit -s "from glob import glob" -- "glob('**/*', recursive=True)"

[issue25596] regular files handled as directories in the glob module

Xavier de Gaye added the comment: This second patch rewrites the conditionals that decide whether to call _iglob() recursively, in a more natural way and without changing the behavior from the first patch. Note that when 'dirname == pathname', then basename is empty and glob2() or glob1() are

[issue25596] regular files handled as directories in the glob module

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka nosy: +serhiy.storchaka priority: normal -> low stage: -> patch review ___ Python tracker

[issue25596] regular files handled as directories in the glob module

New submission from Xavier de Gaye: The glob module happily joins names of regular files together with os.path.join() or attempts to list the files contained into a regular file (sic). The same 'except os.error' statement is used to handle both these cases and the case of a non readable