Re: [Python-ideas] A comprehension scope issue in PEP 572

[Juancarlo Añez]

> Implementation details - even just partial sketches - are always
> "busy".  Think of it this way instead:  it's _currently_ the case that
> listcomps & genexps run in a scope S that's the same as the scope C
> that contains them, _except_ that names appearing as `for` targets are
> local to S.  All other names in S resolve to exactly the same scopes
> they resolved to in C (local in C, global in C, nonlocal in C -
> doesn't matter).
>
> What changes now?  Nothing in that high-level description, except that
> a name appearing as a binding expression target in S that's otherwise
> unknown in C establishes that the name is local to C.  That's nothing
> essentially new, though - bindings _always_ establish scopes for
> otherwise-unknown names in Python.


That's a very nice (and short) explanation!

Maybe my distrust is just don't like the new syntax, or that I'am biased
towards using "as".

-- 
Juancarlo *Añez*
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

My choice of words might not be the best, yes.
I do see to different meanings and/or context for the historical assert and the 
one I propose:

1/
 the first one would be something saying that, as a developer, when writing 
>>> assert (expr)
 in my python code, I mean that all my unit tests and real life tests I could 
think of should succeed the test. I do mean "don't go further, I might not know 
where you come from or where you intend to go or why you are behaving as such, 
but you failed to meet this and/or this criteria/condition".

2/
 the second one is there to activate some other checks, not while developing, 
just at runtime when the user uses my extension and want to get some 
diagnostics/enforcing checks to happen, because he/she is using something I 
couldn't think of in the first place, something that would not have been 
checked before.

Yes, those checks might be considered as identical in a language sense, but 
then : as an extension/interpreter writer, why should I only rely on the debug 
assert available today? Why would it not make sense to offer another assert, 
semantically different, aiming at runtime checks issues and this time where 
control is indeed by the consumer/the extension?





-Original Message-
From: Python-ideas  On 
Behalf Of Chris Angelico
Sent: Tuesday, May 8, 2018 7:38 PM
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active

On Wed, May 9, 2018 at 1:51 AM, Eloi Gaudry  wrote:
> I think that is a difference between:
> - the current 'assert' which usage seems (to me) to focus on 
> development correctness (I think of it as the C-assert enabled in any 
> C program in debug build )
> - the runtime_assert that I submitted on the list, which would be 
> focusing on usage correctness (hence runtime), and easily disabled at 
> runtime (when the python command line options parsing is not an 
> option, for instance when the python interpreter is not python itself 
> and/or when the consumer/extension wants to behave differently).

What's the difference between "development correctness" and "usage 
correctness"? Does the latter depend on user input at run time? I still don't 
understand the distinction you're trying to make here.

ChrisA

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Chris Angelico]

On Wed, May 9, 2018 at 5:27 AM, Eloi Gaudry  wrote:
> My choice of words might not be the best, yes.
> I do see to different meanings and/or context for the historical assert and 
> the one I propose:
>
> 1/
>  the first one would be something saying that, as a developer, when writing
 assert (expr)
>  in my python code, I mean that all my unit tests and real life tests I could 
> think of should succeed the test. I do mean "don't go further, I might not 
> know where you come from or where you intend to go or why you are behaving as 
> such, but you failed to meet this and/or this criteria/condition".
>
> 2/
>  the second one is there to activate some other checks, not while developing, 
> just at runtime when the user uses my extension and want to get some 
> diagnostics/enforcing checks to happen, because he/she is using something I 
> couldn't think of in the first place, something that would not have been 
> checked before.
>
> Yes, those checks might be considered as identical in a language sense, but 
> then : as an extension/interpreter writer, why should I only rely on the 
> debug assert available today? Why would it not make sense to offer another 
> assert, semantically different, aiming at runtime checks issues and this time 
> where control is indeed by the consumer/the extension?
>

No, they're not identical. The first one is an assertion; the second
is simply an 'if' and a 'raise'. It doesn't need any special syntax -
all you need is standard exception creation.

def average(values):
if not values:
raise ValueError("Cannot calculate average of empty collection")

This should not be an assertion, "run-time" or otherwise. You never
want to disable it.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Guido]
>> So the way I envision it is that *in the absence of a nonlocal or global
>> declaration in the containing scope*, := inside a comprehension or genexpr
>> causes the compiler to assign to a local in the containing scope, which is
>> elevated to a cell (if it isn't already). If there is an explicit nonlocal
>> or global declaration in the containing scope, that is honored.

[Juancarlo Añez ]
> This seems to be getting awfully complicated. Proof? Try to write the docs
> for the proposed semantics.

Implementation details - even just partial sketches - are always
"busy".  Think of it this way instead:  it's _currently_ the case that
listcomps & genexps run in a scope S that's the same as the scope C
that contains them, _except_ that names appearing as `for` targets are
local to S.  All other names in S resolve to exactly the same scopes
they resolved to in C (local in C, global in C, nonlocal in C -
doesn't matter).

What changes now?  Nothing in that high-level description, except that
a name appearing as a binding expression target in S that's otherwise
unknown in C establishes that the name is local to C.  That's nothing
essentially new, though - bindings _always_ establish scopes for
otherwise-unknown names in Python.


> I don't understand why we went so astray from the original requirements,
> which could all be met by having `if` and `while` accept `as` to bind an
> expression to a variable that would be local to the structured statement.

"Original" depends on when you first jumped into this ;-)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Steven D'Aprano]

On Tue, May 08, 2018 at 01:28:59PM -0400, Juancarlo Añez wrote:
> So the way I envision it is that *in the absence of a nonlocal or global
> > declaration in the containing scope*, := inside a comprehension or genexpr
> > causes the compiler to assign to a local in the containing scope, which is
> > elevated to a cell (if it isn't already). If there is an explicit nonlocal
> > or global declaration in the containing scope, that is honored.
> >
> 
> This seems to be getting awfully complicated. Proof? Try to write the docs
> for the proposed semantics.

Okay, I'll bite. I don't know why you think its complicated: it is 
precisely the same as ordinary ``=`` assignment scoping rules. It is 
comprehensions that are the special case.

* * * 

The binding expression `` := `` evaluates the right hand 
side , binds it to , and then returns that value.

Unless explicitly declared nonlocal or global (in which case that 
declaration is honoured),  will belong to the current scope, the 
same as other assignments such ``name = value``, with one difference.

Inside comprehensions and generator expressions, variables created with 
``for name in ...`` exist in a separate scope distinct from the usual 
local/nonlocal/global/builtin scopes, and are inaccessible from outside 
the comprehension. (They do not "leak".) That is not the case for those 
created with ``:=``, which belong to the scope containing the 
comprehension. To give an example:

a = 0
x = [b := 10*a for a in (1, 2, 3)]
assert x == [10, 20, 30]
assert a = 0
assert b = 30



 
> I don't understand why we went so astray from the original requirements,
> which could all be met by having `if` and `while` accept `as` to bind an
> expression to a variable that would be local to the structured statement.

That is not the original motivation for binding expressions. The 
original requirements were specifically for comprehensions.

https://mail.python.org/pipermail/python-ideas/2018-February/048971.html

This is hardly the only time that something similar has been raised.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

These all match my expectations.  Some glosses:

[Guido]
> So the way I envision it is that *in the absence of a nonlocal or global
> declaration in the containing scope*, := inside a comprehension or genexpr
> causes the compiler to assign to a local in the containing scope, which is
> elevated to a cell (if it isn't already). If there is an explicit nonlocal
> or global declaration in the containing scope, that is honored.

If the genexp/listcomp is at module level, then "assign to a local in
the containing scope" still makes sense ("locals" and "globals" mean
the same thing at module level), but "elevated to a cell" doesn't then
- it's just a plain global.

In absolutely all cases, what I expect is that

NAME := EXPR

in a genexp/listcomp do the binding _as if_

NAME = object_EXPR_evaluates_to

were executed in the immediately containing scope.  Describing the
goal instead of part of the implementation may be easier to grasp ;-)


> Examples:
>
>   # Simplest case, neither nonlocal nor global declaration
>   def foo():
>   [p := q for q in range(10)]  # Creates foo-local variable p
>   print(p)  # Prints 9
>
>   # There's a nonlocal declaration
>   def bar():
>   p = 42  # Needed to determine its scope
>   def inner():
>   nonlocal p
>   [p := q for q in range(10)]  # Assigns to p in bar's scope
>   inner()
>   print(p)  # Prints 9
>
>   # There's a global declaration
>   def baz():
>   global p
>   [p := q for q in range(10)]
>   baz()
>   print(p)  # Prints 9
>
> All these would work the same way if you wrote list(p := q for q in
> range(10)) instead of the comprehension.

100% agreed.  Add at module scope:

[p := q for q in range(10)]
print(p) # Prints 9

But uou're on your own for class scope, because I never did anything
fancy enough at class scope to need to learn how it works ;-)


> We should probably define what happens when you write [p := p for p in
> range(10)]. I propose that this overwrites the loop control variable rather
> than creating a second p in the containing scope -- either way it's probably
> a typo anyway.

A compile-time error would be fine by me too.  Creating two meanings
for `p` is nuts - pick one in case of conflict.  I suggested before
that the first person with a real use case for this silliness should
get the meaning their use case needs, but nobody bit, so "it's local
then" is fine.


> := outside a comprehension/genexpr is treated just like any other assignment
> (other than in-place assignment), i.e. it creates a local unless a nonlocal
> or global declaration exists.

Also agreed.  People have total control over scopes in explicitly
given functions now, and if the compiler magically made anything
nonlocal they would have no way to stop it.  Well, I suppose we could
add a "non_nonlocal" declaration, but I'd rather not ;-)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Chris Angelico]

On Wed, May 9, 2018 at 1:51 AM, Eloi Gaudry  wrote:
> I think that is a difference between:
> - the current 'assert' which usage seems (to me) to focus on
> development correctness (I think of it as the C-assert enabled in any C
> program in debug build )
> - the runtime_assert that I submitted on the list, which would be
> focusing on usage correctness (hence runtime), and easily disabled at
> runtime (when the python command line options parsing is not an option,
> for instance when the python interpreter is not python itself and/or
> when the consumer/extension wants to behave differently).

What's the difference between "development correctness" and "usage
correctness"? Does the latter depend on user input at run time? I
still don't understand the distinction you're trying to make here.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Juancarlo Añez]

So the way I envision it is that *in the absence of a nonlocal or global
> declaration in the containing scope*, := inside a comprehension or genexpr
> causes the compiler to assign to a local in the containing scope, which is
> elevated to a cell (if it isn't already). If there is an explicit nonlocal
> or global declaration in the containing scope, that is honored.
>

This seems to be getting awfully complicated. Proof? Try to write the docs
for the proposed semantics.

I don't understand why we went so astray from the original requirements,
which could all be met by having `if` and `while` accept `as` to bind an
expression to a variable that would be local to the structured statement.

Cheers,

-- 
Juancarlo *Añez*
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Guido van Rossum]

So the way I envision it is that *in the absence of a nonlocal or global
declaration in the containing scope*, := inside a comprehension or genexpr
causes the compiler to assign to a local in the containing scope, which is
elevated to a cell (if it isn't already). If there is an explicit nonlocal
or global declaration in the containing scope, that is honored.

Examples:

  # Simplest case, neither nonlocal nor global declaration
  def foo():
  [p := q for q in range(10)]  # Creates foo-local variable p
  print(p)  # Prints 9

  # There's a nonlocal declaration
  def bar():
  p = 42  # Needed to determine its scope
  def inner():
  nonlocal p
  [p := q for q in range(10)]  # Assigns to p in bar's scope
  inner()
  print(p)  # Prints 9

  # There's a global declaration
  def baz():
  global p
  [p := q for q in range(10)]
  baz()
  print(p)  # Prints 9

All these would work the same way if you wrote list(p := q for q in
range(10)) instead of the comprehension.

We should probably define what happens when you write [p := p for p in
range(10)]. I propose that this overwrites the loop control variable rather
than creating a second p in the containing scope -- either way it's
probably a typo anyway.

:= outside a comprehension/genexpr is treated just like any other
assignment (other than in-place assignment), i.e. it creates a local unless
a nonlocal or global declaration exists.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Wed, 2018-05-09 at 01:34 +1000, Steven D'Aprano wrote:
> On Tue, May 08, 2018 at 07:37:55AM +, Eloi Gaudry wrote:
> 
> > * debug assert, for helping developing a new features, used with
> > debug-
> > builds
> > * runtime assert, for ensuring correctness and/or diagnosing issue
> > with
> > production/release-builds on-site.
> 
> I don't think that is a useful distinction to make.
> 
> I use `assert` all the time, and I have never once cared whether it
> is a 
> "debug build" or a "production build". I use assert in my code, and
> then 
> people (including me) can use it with whatever build of Python they 
> like.
I do understand your point but I don't share your opinion.
I think that is a difference between:
- the current 'assert' which usage seems (to me) to focus on
development correctness (I think of it as the C-assert enabled in any C
program in debug build )
- the runtime_assert that I submitted on the list, which would be
focusing on usage correctness (hence runtime), and easily disabled at
runtime (when the python command line options parsing is not an option,
for instance when the python interpreter is not python itself and/or
when the consumer/extension wants to behave differently).

> I don't even know which builds of Python I'm running. My *guess* is
> that 
> the OS-provided Python is probably a non-debug build, and the ones
> I've 
> compiled from source will be whatever the default build settings
> are, 
> but I don't know what that is or how to find out.
> 
> And if there was some sort of runtime_assert that could be enabled
> and 
> disabled individually, why wouldn't I use it with debug builds as
> well 
> as non-debug builds?
There would be no reason why, but you would benefit from being able to
easily activate/deactivate the runtime assert.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Tim]
>> (inside a synthetic function created to
>> implement a listcomp/genexp, names bound by "=" are local; names bound
>> by ":=" are nonlocal; names bound by both are "who cares?"-
>> compiler-time error would be fine by me, or the first person to show a
>> real use case wins).

[Jacco van Dorp ]
> Wait, you can't use = in a listcomp, right ? Or are you talking about
> the implementation hidden to the casual user ?

Sorry, I was too obscure there - I intended "=" to mean "name binding
by any means other than :=".  Off the top of my head, I believe that -
today - the only "any means other than :=" possible in a
listcomp/genexp is appearing as a target in a `for` clause (like the
`i` in "[i+1 for i in iterable]`).  If there's some other way I
missed, I meant to cover that too.

But, yes, you're right, `names bound by "="` makes no literal sense at
all there.


> I thought letting := bind to the surrounding scope was fine basically
> because it's currently not possible, so therefore there would be no
> syntactic ambiguity, and it'd actually do what people would expect.

It's not really about the semantics of `:=` so much as about how
synthetic functions are defined.  In most cases, it amounts to saying
"in the nested function synthesized for a listcomp/genexp, if a name
`x` appears as the target of a binding expression in the body, a
`nonlocal x` declaration is generated near the top of the synthetic
function".

For example, if this block appears inside a function:

it = (i for i in range(10))
total = 0
for psum in (total := total + value for value in it):
print(psum}

under the current PEP meaning it blows up in the same way this code
blows up today:

it = (i for i in range(10))
total = 0
def _func(it):
for value in it:
  total = total + value  # blows up here
  yield total
for psum in _func(it):
  print(psum)

with

UnboundLocalError: local variable 'total' referenced before assignment

But add

nonlocal total

at the top of `_func()` and it works fine (displays 0, 1, 3, 6, 10, 15, ...).

So it's not really about what ":=" does, but about how ":=" affects
scope in synthesized nested functions.

But if you wrote a nested function yourself?  There's no suggestion
here that ":=" have any effect on scope decisions in explicitly given
nested functions (same as for "=", it would imply "the target is
local"), just on those generated "by magic" for listcomps/genexps.

Maybe there should be, though.  My initial thought was "no, because
the user has total control over scope decisions in explicitly given
functions today, but if something was magically made nonlocal they
would have no way to override that".
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Serhiy Storchaka]

08.05.18 18:34, Steven D'Aprano пише:

I don't even know which builds of Python I'm running. My *guess* is that
the OS-provided Python is probably a non-debug build, and the ones I've
compiled from source will be whatever the default build settings are,
but I don't know what that is or how to find out.


There are different means of "debug build".

It may mean that binaries are build with enabling runtime checks in C 
code. When you build from sources you will get a non-debug build by 
default. You need to pass a special option --with-pydebug to ./configure 
for getting a debug build. The OS-provided Python is a non-debug build too.


It may means that the "assert" statement in Python code is not a no-op 
and the building __debug__ constant is True. Python is ran in debug mode 
by default. You have to pass the -O option to the python command for 
running Python in non-debug mode. ISTM the OP uses the term "debug 
build" in this meaning.


Finally, a special "development mode" mode was introduced in 3.7. It is 
enabled by the command line option -X dev. It switches on several 
expensive runtime checks in C code (if they are not switched off by 
compiling binaries in "release build" in the first meaning).


https://docs.python.org/3.8/whatsnew/3.7.html#new-development-mode-x-dev

All these things are virtually orthogonal and can be combined arbitrary.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Tue, 2018-05-08 at 09:58 -0400, Eric V. Smith wrote:
> I think what's confusing to me (and maybe others) is that we haven't 
> seen your vision on how this would look in Python code.
> 
> An example that would throw runtime assertions and the same example 
> where it wouldn't (after a global switch is set?) would be helpful.
> 
> Eric

Eric, I can only agree to your comment.

As a matter of fact, I have only used the case where the runtime assert
is activated through an extension. In this case, adding a set method is
simple : in our framework, we would set the variable
Py_RuntimeAssertFlag to 1, depending on some command line parameters,
or settings found in a configuration file).

In pure python, if something as having a method registered in
__builtins__ make sense, it could be used to trigger the assertive
behavior.

In this example, let's assume that you may want to perform some basic
check on a given file in diagnostics mode:

>>> def check_mounts_size():
...   return len( open( '/proc/self/mounts', 'r' ).readlines() )>1024
...
>>> runtime_assert( len( open( '/proc/self/mounts', 'r'
).readlines()>1024 ) )
>>> runtime_assert( check_mounts_size() )

>>> __builtins__.set_runtime_assert( True )

>>> runtime_assert( len( open( '/proc/self/mounts', 'r'
).readlines()>1024 ) )
Traceback (most recent call last):
  File "", line 1, in 
AssertionError

>>> runtime_assert( check_mounts_size() )
Traceback (most recent call last):
  File "", line 1, in 
AssertionError
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Serhiy Storchaka]

07.05.18 20:52, Guido van Rossum пише:

He basically wants a macro so that

   runtime_assert()

expands to

   if  and ():
   raise AssertionError

In Lisp this would be easy. :-)


Python is not Lisp (still). But there is the MacroPy project. And at end 
you always can use an external tool for code generation. For example the 
old good cpp.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Random832]

On Mon, May 7, 2018, at 02:05, Steve Barnes wrote:
> In a lot of uses of os.walk it is desirable to skip version control 
> directories, (which are usually hidden directories), to the point that 
> almost all of the examples given look like:

CVS isn't a hidden directory on Linux. Maybe it can be on windows, but it 
probably won't be if it's manually created, which you mentioned issues with 
below. There's probably a discussion we should be having about exposing these 
system-specific attributes, but they really can't be a general solution for the 
problem you have.

MacOS, incidentally, has two distinct attributes for hiding files [chflags 
hidden and setfile -a V], along with a ".private" file that can be in a 
directory containing a list of filenames to hide.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Steven D'Aprano]

On Tue, May 08, 2018 at 09:58:51AM -0400, Eric V. Smith wrote:

> I think what's confusing to me (and maybe others) is that we haven't 
> seen your vision on how this would look in Python code.
> 
> An example that would throw runtime assertions and the same example 
> where it wouldn't (after a global switch is set?) would be helpful.

In Eloi's first post in this thread, he gave the example:

runtime_assert( expr )


Although it is written as a function call, he refers to it as a 
statement, so I guess he means:

runtime_assert expr


He says that would compile to the equivalent of:

if runtime_assert_active and expr:
print(RuntimeAssertionError())


but he gives no idea of what runtime_assert_active is (is it a 
per-module global variable? a single application-wide super-global? a 
local variable? something else?) or how we are supposed to set it.

Nor does he explain why failed assertions merelt *print* an error 
message, rather than raising an exception.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Steven D'Aprano]

On Tue, May 08, 2018 at 07:37:55AM +, Eloi Gaudry wrote:

> * debug assert, for helping developing a new features, used with debug-
> builds
> * runtime assert, for ensuring correctness and/or diagnosing issue with
> production/release-builds on-site.

I don't think that is a useful distinction to make.

I use `assert` all the time, and I have never once cared whether it is a 
"debug build" or a "production build". I use assert in my code, and then 
people (including me) can use it with whatever build of Python they 
like.

I don't even know which builds of Python I'm running. My *guess* is that 
the OS-provided Python is probably a non-debug build, and the ones I've 
compiled from source will be whatever the default build settings are, 
but I don't know what that is or how to find out.

And if there was some sort of runtime_assert that could be enabled and 
disabled individually, why wouldn't I use it with debug builds as well 
as non-debug builds?



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Giampaolo Rodola']

On Tue, May 8, 2018 at 2:00 PM, David Mertz  wrote:

> I like the idea. I think an argument to os.walk() is the simplest option
> for most users. But per some comments, "hidden" is actually more subtle
> than the filesystem bit sometimes. I.e. dot-files, ~ suffix, maybe .bak,
> etc.
>
> I'd suggest meeting the ideas slightly and making the new argument
> 'filter' or 'skip' that takes a callable. Default to None, but provide an
> os.is_hidden that users don't need to figure out how to implement. E.g.
>
> os.walk(PATH, skip=os.is_hidden)
>
> os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak',
> '.tmp')))
>

I think this would be a good addition because it gives direct access to the
underlying os.scandir() objects which are currently inaccessible and
discarded (if os.walk() were to be written today it'd probably yield (root,
os.DirEntry) instead of (root, dirs, files)). As such one can implement
advanced filtering logic without having to call os.stat() for each path
string yielded by os.walk() (faster).

IMO the callback should accept a (root, os.DirEntry) pair though, because the
"root" path can also be part of the filtering logic.

-- 

Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Tue, 2018-05-08 at 07:37 -0700, Guido van Rossum wrote:
> > > In Lisp this would be easy. :-)
> > 
> > and he already has a diff ready for review if needed (basically
> > very
> > similar to the current 'assert' implementation :)
> 
> That seems premature. There is not even a hint of agreement that such
> a feature would be useful *in general* (I'm not doubting your
> situation) and worth our limited volunteer developer resources to
> maintain, document etc. for decades to come.

I didn't mean to push too fast, just to show how things could be
implemented and maybe clarify my comments/statements.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Guido van Rossum]

On Tue, May 8, 2018 at 12:29 AM, Eloi Gaudry  wrote:

> On Mon, 2018-05-07 at 10:52 -0700, Guido van Rossum wrote:
> > On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka  > > wrote:
> > > I just don't understand why you need a new keyword for writing
> > > runtime checks.
> >
> > Oh, that's pretty clear. The OP wants to be able to turn these checks
> > off with some flag he can set/clear at runtime, and when it's off he
> > doesn't want to incur the overhead of evaluating the check. The
> > assert statement has the latter property, but you have to use -O to
> > turn it off. He basically wants a macro so that
> >
> >   runtime_assert()
> >
> > expands to
> >
> >   if  and ():
> >   raise AssertionError
> >
> > In Lisp this would be easy. :-)
>
> and he already has a diff ready for review if needed (basically very
> similar to the current 'assert' implementation :)


That seems premature. There is not even a hint of agreement that such a
feature would be useful *in general* (I'm not doubting your situation) and
worth our limited volunteer developer resources to maintain, document etc.
for decades to come.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Eric Fahlgren]

On Tue, May 8, 2018 at 2:31 AM, Oleg Broytman  wrote:

>So anyone who wants to filter os.walk() must reimplement os.walk()
> themselves instead of passing something like filter_dir and filter_file
> (or accept_dir/accept_file) to os.walk()? Kind of painful, no?
>

​Not really.  It's pretty simple code so you put it in your 'usual
suspects' module and just forget about it.  Here's our version, maybe 10
years old (reworked last whenever scandir came out)​:

def _compiled_patterns(patterns, globs=True, flags=0):
""" $uuid:95a9b8e2-fb6a-59be-b9c2-da0e6e12f8d3$
Compile a list of patterns into regex patterns.  If ``globs``
is true, use ``fnmatch`` to convert the patterns into regular
expressions prior to compilation.  ``flags`` is any of the ``re``
module's regular expression flags.
"""
if globs:
patterns = list(_fnmatch.translate(glob) for glob in patterns)
return list(_re.compile(regex, flags=flags) for regex in patterns)

def walk(root_path, ignore_directories=[], show_directories=False):
""" $uuid:f77197cd-239b-5d93-9253-c3eb7439d720$
Walk the directory tree and return all the file entries, trimming
directories as we go.  ``ignore_directories`` is a list of Unix
file globs.
"""
ignore_directories = _compiled_patterns(ignore_directories)

def do_walk(top):
""" $uuid:e6a4f789-5b5f-56a2-8551-297c142c3e17$ """
for entry in _scandir.scandir(top):
if not entry.is_dir():
yield entry
elif entry.is_symlink():
pass # Ignore links.
elif not any(ignore.match(entry.name) for ignore in
ignore_directories):
if show_directories:
yield entry
for entry in do_walk(entry.path):
yield entry

return do_walk(root_path)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Tue, 2018-05-08 at 09:35 -0400, Eric V. Smith wrote:
> On 5/8/2018 3:37 AM, Eloi Gaudry wrote:
> > On Mon, 2018-05-07 at 16:56 +, Brett Cannon wrote:
> > > 
> > > My question is how is this different to running with -O which
> > > leaves
> > > the assert statement out of the bytecode and so you avoid any
> > > run-
> > > time cost of the statement entirely?
> > 
> > Not so much different, except that:
> > - the switch won't need to be on the command line
> 
> So, is the switch set in code? If so, what would that look like?
> 
> Eric
So far, I only have the "extension" method, where a global variable is
set directly from within the C-interface, as currently done with the
debug assert:

Include/pydebug.h : int Py_RuntimeAssertFlag = 1;
Your_extension/main.c : SetFlag(Py_RuntimeAssertFlag);
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eric V. Smith]

On 5/8/2018 9:50 AM, Eloi Gaudry wrote:

On Tue, 2018-05-08 at 09:35 -0400, Eric V. Smith wrote:

On 5/8/2018 3:37 AM, Eloi Gaudry wrote:

On Mon, 2018-05-07 at 16:56 +, Brett Cannon wrote:


My question is how is this different to running with -O which
leaves
the assert statement out of the bytecode and so you avoid any
run-
time cost of the statement entirely?


Not so much different, except that:
- the switch won't need to be on the command line


So, is the switch set in code? If so, what would that look like?

Eric

So far, I only have the "extension" method, where a global variable is
set directly from within the C-interface, as currently done with the
debug assert:

Include/pydebug.h : int Py_RuntimeAssertFlag = 1;
Your_extension/main.c : SetFlag(Py_RuntimeAssertFlag);


I think what's confusing to me (and maybe others) is that we haven't 
seen your vision on how this would look in Python code.


An example that would throw runtime assertions and the same example 
where it wouldn't (after a global switch is set?) would be helpful.


Eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eric V. Smith]

On 5/8/2018 3:37 AM, Eloi Gaudry wrote:

On Mon, 2018-05-07 at 16:56 +, Brett Cannon wrote:


My question is how is this different to running with -O which leaves
the assert statement out of the bytecode and so you avoid any run-
time cost of the statement entirely?


Not so much different, except that:
- the switch won't need to be on the command line


So, is the switch set in code? If so, what would that look like?

Eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Mon, 2018-05-07 at 16:56 +, Brett Cannon wrote:
> 
> My question is how is this different to running with -O which leaves
> the assert statement out of the bytecode and so you avoid any run-
> time cost of the statement entirely? 

Not so much different, except that:
- the switch won't need to be on the command line
- the 2 different asserts would be considered differently by
developers/users : 
* debug assert, for helping developing a new features, used with debug-
builds
* runtime assert, for ensuring correctness and/or diagnosing issue with
production/release-builds on-site.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[David Mertz]

I like the idea. I think an argument to os.walk() is the simplest option
for most users. But per some comments, "hidden" is actually more subtle
than the filesystem bit sometimes. I.e. dot-files, ~ suffix, maybe .bak,
etc.

I'd suggest meeting the ideas slightly and making the new argument 'filter'
or 'skip' that takes a callable. Default to None, but provide an
os.is_hidden that users don't need to figure out how to implement. E.g.

os.walk(PATH, skip=os.is_hidden)

os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak', '.tmp')))

On Tue, May 8, 2018, 5:47 AM Oleg Broytman  wrote:

> Hi!
>
> On Tue, May 08, 2018 at 07:12:35AM +, Yuval Greenfield <
> ubershme...@gmail.com> wrote:
> > If you
> > want to avoid duplicate `stat` calls, you'll probably write:
> >
> > import os
> > import stat
> > def is_hidden(st):
> > return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
> > def visible_walk(path):
> > for entry in os.scandir(path):
> > if entry.is_dir():
> > if not is_hidden(entry.stat()):
> > yield from visible_walk(entry.path)
> > else:
> > if not is_hidden(entry.stat()):
> > yield entry.path
>
>So anyone who wants to filter os.walk() must reimplement os.walk()
> themselves instead of passing something like filter_dir and filter_file
> (or accept_dir/accept_file) to os.walk()? Kind of painful, no?
>
> > Cheers,
> > Yuval
>
> Oleg.
> --
>  Oleg Broytmanhttp://phdru.name/p...@phdru.name
>Programmers don't die, they just GOSUB without RETURN.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter

[Steven D'Aprano]

On Tue, May 08, 2018 at 10:13:43AM +0200, Vincent Maillol wrote:

> Perhaps something like:
> >
> 
> > >>> itemgetter(-1, 0, -2,
> > ...default0="first argument default",
> > ...default1=["second", "argument", "default"],
> > ...default={"global": "default"},
> > ... )

Sorry Vincent, but I think that's precisely the sort of overly 
complicated API that Raymond was worried about when he voted against 
this proposal. A single default value that applies when the item (or 
key) isn't present is easy to comprehend. But this extention is, I 
think, a case of an over-engineered, excessively intricate solution, 
and I doubt it solves any real problem.

And I'm not even sure I know what it means. You have keyword parameters 
default0, default1, and then *default* with no suffix. Is that a typo 
for default2, or is it supposed to be a default default, the default to 
use when no default is specified?


> The keywords suffixed by indice are uncommon. Maybe we can use dedicated
> object.
> 
> >>> itemgetter(
> ... itemgetter.WithDefault(-1, "first argument default"),
> ... itemgetter.WithDefault(0, ["second", "argument", "default"])
> ... -2  # has no default value
> ... )

I'm afraid there's no elegance to this design either.

I've had many occasions where I want to get an item, falling back on a 
default if it is not present. With dicts this is common enough that we 
have dict.get(). It is less convenient with lists, and I'd like to see 
itemgetter support that case.

But I cannot think of any case where I have needed or wanted to extract

item 5 with "spam" as the default
item 2 with no default
item 3 with "eggs" as the default

(for example), let alone that this is *so common* that I'd rather read 
the complicated documenation to work out how to use the function, rather 
than just write a short helper:

def extract(sequence):
return ("spam" if len(sequence) < 5 else sequence[5],
sequence[2],
"eggs" if len(sequence) < 5 else sequence[3],
)

Raymond and Serhey are right: beyond a certain point, we should just 
write a custom function (whether lambda or not). We just disagree on 
which side of that point the single-default value case is, but I think 
we will agree that your example is so far past the point that we cannot 
even see it from here :-)


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Oleg Broytman]

Hi!

On Tue, May 08, 2018 at 07:12:35AM +, Yuval Greenfield 
 wrote:
> If you
> want to avoid duplicate `stat` calls, you'll probably write:
> 
> import os
> import stat
> def is_hidden(st):
> return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
> def visible_walk(path):
> for entry in os.scandir(path):
> if entry.is_dir():
> if not is_hidden(entry.stat()):
> yield from visible_walk(entry.path)
> else:
> if not is_hidden(entry.stat()):
> yield entry.path

   So anyone who wants to filter os.walk() must reimplement os.walk()
themselves instead of passing something like filter_dir and filter_file
(or accept_dir/accept_file) to os.walk()? Kind of painful, no?

> Cheers,
> Yuval

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
> 
> One reason is sub-interpreter support: you can have multiple 
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
> 
> With heap types, each sub-interpreter can have its own copy of the
> type 
> object. But with builtins, changes done in one interpreter would be 
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I guess) are only running
one interpreter.

In case several intepreters are used, it would make sense to have a
non-heap type that would be seen as a singleton across all of them, no
?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter

[Vincent Maillol]

>
> As attrgetter/itemgetter might get heterogeneus data, I would expect a
> per-get default, not [just] a global default.
>
+1

Perhaps something like:
>

> >>> itemgetter(-1, 0, -2,
> ...default0="first argument default",
> ...default1=["second", "argument", "default"],
> ...default={"global": "default"},
> ... )
>

The keywords suffixed by indice are uncommon. Maybe we can use dedicated
object.

>>> itemgetter(
... itemgetter.WithDefault(-1, "first argument default"),
... itemgetter.WithDefault(0, ["second", "argument", "default"])
... -2  # has no default value
... )

Another possibility is that itemgetter becomes an object with `add_default`
method.

>>> getter = itemgetter(-1, 0, 2)
>>> getter.add_default(0, "first argument default")
>>> getter.add_default(2, "third argument default")

or simply default parameter should be a list

>>> itemgetter(-1, 0, 2,
...default=["first", itemgetter.NoDefault, "third"])



2018-05-08 1:37 GMT+02:00 Danilo J. S. Bellini :

> (1)
>
> On 7 May 2018 at 01:07, Raymond Hettinger 
> wrote:
>
>> He could've written:
>>
>> get = rpartial(getattr, 'foo', None)
>> return get(args) or get(config) or get(env)
>>
>
> That's somewhat hybrid, it would behave like:
>
> lambda obj: getattr(obj, "foo", None),
>
> but one might expect a "reversed partial" to have its arguments reversed
> as well, like:
>
> lambda obj: getattr(obj, None, "foo").
>
> (2)
>
> Why getattr can't accept keyword arguments?
>
> >>> getattr_zero = partial(getattr, default=0)
> >>> getattr_zero({}, "blah")
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: getattr() takes no keyword arguments
>
> Since partial/partialmethod can apply keyword arguments, allowing the
> "default=" keyword argument in getattr would be an alternative to the
> "rpartial".
>
> (3)
>
> At one time, lambda was the one obvious way. [...]
>>
>
> I really like lambdas, but there are some "closure gotchas" like:
>
> >>> for idx in indices:
> ... do_something(lambda seq: seq[idx])
>
> If the lambda is stored somewhere to be called after the loop ends (or
> after its iteration), the seq[idx] would load the item with the last
> iterated index. This:
>
> >>> for idx in indices:
> ... do_something(itemgetter(idx))
>
> would behave more like this instead:
>
> >>> for idx in indices:
> ... do_something((lambda i: (lambda seq: seq[i]))(idx))
>
> Which I wouldn't call "the one obvious way".
>
> (4)
>
> If itemgetter and attrgetter only did a single lookup, a default might
>> make sense.  However, that doesn't fit well with multiple and/or chained
>> lookups where are number of options are possible. (See
>> https://bugs.python.org/issue14384#msg316222 for examples and
>> alternatives).
>
> As attrgetter/itemgetter might get heterogeneus data, I would expect a
> per-get default, not [just] a global default.
> Perhaps something like:
>
> >>> itemgetter(-1, 0, -2,
> ...default0="first argument default",
> ...default1=["second", "argument", "default"],
> ...default={"global": "default"},
> ... )
>
> --
> Danilo J. S. Bellini
> ---
> "*It is not our business to set up prohibitions, but to arrive at
> conventions.*" (R. Carnap)
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Mon, 2018-05-07 at 10:52 -0700, Guido van Rossum wrote:
> On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka  > wrote:
> > I just don't understand why you need a new keyword for writing
> > runtime checks.
> 
> Oh, that's pretty clear. The OP wants to be able to turn these checks
> off with some flag he can set/clear at runtime, and when it's off he
> doesn't want to incur the overhead of evaluating the check. The
> assert statement has the latter property, but you have to use -O to
> turn it off. He basically wants a macro so that
> 
>   runtime_assert()
> 
> expands to
> 
>   if  and ():
>   raise AssertionError
> 
> In Lisp this would be easy. :-)

and he already has a diff ready for review if needed (basically very
similar to the current 'assert' implementation :)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Jacco van Dorp]

> [Tim]
> (inside a synthetic function created to
> implement a listcomp/genexp, names bound by "=" are local; names bound
> by ":=" are nonlocal; names bound by both are "who cares?"-
> compiler-time error would be fine by me, or the first person to show a
> real use case wins).

Wait, you can't use = in a listcomp, right ? Or are you talking about
the implementation hidden to the casual user ?

I thought letting := bind to the surrounding scope was fine basically
because it's currently not possible, so therefore there would be no
syntactic ambiguity, and it'd actually do what people would expect.

Jacco
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Yuval Greenfield]

On Mon, May 7, 2018 at 9:44 PM Steve Barnes  wrote:

> Since the implementation of os.walk has changed to use os.scandir which
> exposes the returned file statuses in the os.DirEntry.stat() the
> overhead should be minimal.
>
> An alternative would be to add another new function, say os.vwalk(), to
> only walk visible entries.
>

On Tue, May 8, 2018 at 12:06 AM Steven D'Aprano  wrote:

> I would write something like:
> for root, dirs, files in filter(ignorable, os.walk(some_dir)):


I agree with Steven with regards to `filter` needing to be flexible. If you
want to avoid duplicate `stat` calls, you'll probably write:

import os
import stat
def is_hidden(st):
return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
def visible_walk(path):
for entry in os.scandir(path):
if entry.is_dir():
if not is_hidden(entry.stat()):
yield from visible_walk(entry.path)
else:
if not is_hidden(entry.stat()):
yield entry.path


Then you can decide whether you want to ignore hidden files or just hidden
directories. The variations for such a need are many. So it makes sense to
leave any specific filtering need outside of the standard library. A PyPI
package with a few standard filtered walks could be a nice exploration for
this idea.

Cheers,

Yuval
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Steven D'Aprano]

On Mon, May 07, 2018 at 06:05:15AM +, Steve Barnes wrote:
> In a lot of uses of os.walk it is desirable to skip version control 
> directories, (which are usually hidden directories), to the point that 
> almost all of the examples given look like:
> 
> import os
> for root, dirs, files in os.walk(some_dir):
>  if 'CVS' in dirs:
>  dirs.remove('CVS')  # or .svn or .hg etc.
>  # do something...

I would write something like:

for root, dirs, files in filter(ignorable, os.walk(some_dir)):
...

where ignorable() is a function that returns False for whatever you want 
to ignore.

I don't think we can possibly agree on a single definition of 
"ignorable". This could include any combination of:

- dot files;
- files with the invisible bit set, for file systems which support that;
- files within certain directories;
- files ending in ~ (backup files);
- files with certain extensions;

or more.

Possibly this is a good use-case for composible functions, so we could 
have a set of pre-built filters:

ignorable = invisible + dotfiles + directories('.git', '.hg') + extensions('~', 
'.pdf')

but that sounds like it ought to be a separate module, not built in.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Delivery Status Notification (Failure)

[Jacco van Dorp]

2018-05-07 21:56 GMT+02:00 Neil Girdhar :
> Regular expressions are not just "an order of magnitude better"—they're
> asymptotically faster.  See
> https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm
> for a non-regular-expression algorithm.

Hence my

>> [Jacco wrote, capitalized important words]
>> regular expressions would probably be AT LEAST an order of magnitude
>> better in speed, if it's a bottleneck to you. But pure python
>> implementation for this is a lot easier than it would be for the
>> current string.count().
>>

But I think my point stands that that's what you need to do if speed
is an issue, and python code is fine when it isn't.

Also, intersting read. Thanks.


Jacco
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/