Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

Hi Brett,
Sorry about that, I did not mean to be rude.

What I wanted to says is:

1.  That I relied on such a feature

2.  Other people on this mailing-list already asked something similar at 
several occasions

3.  HEAPTYPE would not always be a solution

4.  Then I thought this was something that would indeed need more 
discussion and might get acceptance if discussed once again.

Eloi

From: Brett Cannon 
Sent: Tuesday, June 26, 2018 8:53 PM
To: Eloi Gaudry 
Cc: encu...@gmail.com; python-ideas@python.org
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)


On Thu, Jun 21, 2018, 12:27 Eloi Gaudry, 
mailto:eloi.gau...@fft.be>> wrote:

This request didn't have a lot of traction, but I still consider this is 
something that would need to be supported

Please be careful about using the word "need" as it comes off as demanding 
instead of as a suggestion.


-Brett


(2 lines of code to be changed; no regression so far with python 2 and python 
3).



My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type 
attributes




Petr, what do you think ?

Eloi


From: Python-ideas 
mailto:fft...@python.org>> 
on behalf of Eloi Gaudry mailto:eloi.gau...@fft.be>>
Sent: Tuesday, May 8, 2018 9:26:47 AM
To: encu...@gmail.com<mailto:encu...@gmail.com>; 
python-ideas@python.org<mailto:python-ideas@python.org>
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
>
> One reason is sub-interpreter support: you can have multiple
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
>
> With heap types, each sub-interpreter can have its own copy of the
> type
> object. But with builtins, changes done in one interpreter would be
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I guess) are only running
one interpreter.

In case several intepreters are used, it would make sense to have a
non-heap type that would be seen as a singleton across all of them, no
?
___
Python-ideas mailing list
Python-ideas@python.org<mailto:Python-ideas@python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org<mailto:Python-ideas@python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

Hi Guido,

I would like to be sure that the lack of support would not be the result of my 
inability to sum-up my use case.

This is why I gave some links to illustrate

-the reason why the behavior was changed a decade ago

-that such a possibility was actually needed by other extension 
developers (where their built-in types would benefit from being able to 
redefine some method dynamically)

-that python core developers and python extension developers can have 
different needs and objectives (which was the main reason why I was submitting 
this to the mailing-list again)

I feel sorry if that only resulted in looking like I was repeating myself.

Have a good day,
Eloi

From: Guido van Rossum 
Sent: Tuesday, June 26, 2018 7:20 PM
To: Eloi Gaudry 
Cc: Python-Ideas ; Serhiy Storchaka 

Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)

Hey Eloi,

I think you need to just give up on this. Nobody here seems to support or 
understand your use case. At this point you are repeating yourself (again 
claiming there is no good reason for the prohibition and that it's only a few 
lines of code to change) and you can be assured that the response will also be 
the same.

--Guido

On Tue, Jun 26, 2018 at 8:00 AM Eloi Gaudry 
mailto:eloi.gau...@fft.be>> wrote:

the origin of this feature disappearing for built-in types:

http://bugs.jython.org/issue1058



'''
object.__set/delattr__ allow modification of built in types, this is
known as the Carlo Verre hack:

Jython 2.3a0+ (trunk:4630:4631M, Jun 14 2008, 20:07:38)
[Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13
Type "help", "copyright", "credits" or "license" for more information.
>>> object.__setattr__(str, 'lower', str.upper)
>>> 'dammit Carlo!'.lower()
'DAMMIT CARLO!'
'''

but I do not see any reason why having an explicit flag for python extensions 
written in C to declare their types as static struct, and still be able to 
change their __setattr__, __getattr__, etc. slots would not make sense.

extensions and core types have not the same constraints and purposes, this 
should be reflected on the capabilities the first would have somewhere then.







From: Eloi Gaudry
Sent: Tuesday, June 26, 2018 4:27:18 PM
To: python-ideas@python.org<mailto:python-ideas@python.org>
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)


some literature:

https://mail.python.org/pipermail/python-dev/2008-February/077180.html

https://mail.python.org/pipermail/python-dev/2008-February/077169.html



where it is stated that python C struct type should not be able to have their 
attributes changed.

but the extension needs is clearly not taken into account.

____
From: Python-ideas 
mailto:fft...@python.org>> 
on behalf of Eloi Gaudry mailto:eloi.gau...@fft.be>>
Sent: Thursday, June 21, 2018 5:26:37 PM
To: python-ideas@python.org<mailto:python-ideas@python.org>; 
encu...@gmail.com<mailto:encu...@gmail.com>
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)


This request didn't have a lot of traction, but I still consider this is 
something that would need to be supported (2 lines of code to be changed; no 
regression so far with python 2 and python 3).



My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type 
attributes



Petr, what do you think ?

Eloi

____
From: Python-ideas 
mailto:fft...@python.org>> 
on behalf of Eloi Gaudry mailto:eloi.gau...@fft.be>>
Sent: Tuesday, May 8, 2018 9:26:47 AM
To: encu...@gmail.com<mailto:encu...@gmail.com>; 
python-ideas@python.org<mailto:python-ideas@python.org>
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
>
> One reason is sub-interpreter support: you can have multiple
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
>
> With heap types, each sub-interpreter can have its own copy of the
> type
> object. But with builtins, changes done in one interpreter would be
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I g

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

some literature:

https://mail.python.org/pipermail/python-dev/2008-February/077180.html

<https://mail.python.org/pipermail/python-dev/2008-February/077180.html>https://mail.python.org/pipermail/python-dev/2008-February/077169.html


where it is stated that python C struct type should not be able to have their 
attributes changed.

but the extension needs is clearly not taken into account.



From: Python-ideas  on 
behalf of Eloi Gaudry 
Sent: Thursday, June 21, 2018 5:26:37 PM
To: python-ideas@python.org; encu...@gmail.com
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)


This request didn't have a lot of traction, but I still consider this is 
something that would need to be supported (2 lines of code to be changed; no 
regression so far with python 2 and python 3).


My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type 
attributes


Petr, what do you think ?

Eloi


From: Python-ideas  on 
behalf of Eloi Gaudry 
Sent: Tuesday, May 8, 2018 9:26:47 AM
To: encu...@gmail.com; python-ideas@python.org
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
>
> One reason is sub-interpreter support: you can have multiple
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
>
> With heap types, each sub-interpreter can have its own copy of the
> type
> object. But with builtins, changes done in one interpreter would be
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I guess) are only running
one interpreter.

In case several intepreters are used, it would make sense to have a
non-heap type that would be seen as a singleton across all of them, no
?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

the origin of this feature disappearing for built-in types:

http://bugs.jython.org/issue1058


'''

object.__set/delattr__ allow modification of built in types, this is
known as the Carlo Verre hack:

Jython 2.3a0+ (trunk:4630:4631M, Jun 14 2008, 20:07:38)
[Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13
Type "help", "copyright", "credits" or "license" for more information.
>>> object.__setattr__(str, 'lower', str.upper)
>>> 'dammit Carlo!'.lower()
'DAMMIT CARLO!'
'''

but I do not see any reason why having an explicit flag for python extensions 
written in C to declare their types as static struct, and still be able to 
change their __setattr__, __getattr__, etc. slots would not make sense.

extensions and core types have not the same constraints and purposes, this 
should be reflected on the capabilities the first would have somewhere then.






From: Eloi Gaudry
Sent: Tuesday, June 26, 2018 4:27:18 PM
To: python-ideas@python.org
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)


some literature:

https://mail.python.org/pipermail/python-dev/2008-February/077180.html

<https://mail.python.org/pipermail/python-dev/2008-February/077180.html>https://mail.python.org/pipermail/python-dev/2008-February/077169.html


where it is stated that python C struct type should not be able to have their 
attributes changed.

but the extension needs is clearly not taken into account.


________
From: Python-ideas  on 
behalf of Eloi Gaudry 
Sent: Thursday, June 21, 2018 5:26:37 PM
To: python-ideas@python.org; encu...@gmail.com
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)


This request didn't have a lot of traction, but I still consider this is 
something that would need to be supported (2 lines of code to be changed; no 
regression so far with python 2 and python 3).


My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type 
attributes


Petr, what do you think ?

Eloi

________
From: Python-ideas  on 
behalf of Eloi Gaudry 
Sent: Tuesday, May 8, 2018 9:26:47 AM
To: encu...@gmail.com; python-ideas@python.org
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
>
> One reason is sub-interpreter support: you can have multiple
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
>
> With heap types, each sub-interpreter can have its own copy of the
> type
> object. But with builtins, changes done in one interpreter would be
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I guess) are only running
one interpreter.

In case several intepreters are used, it would make sense to have a
non-heap type that would be seen as a singleton across all of them, no
?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

This request didn't have a lot of traction, but I still consider this is 
something that would need to be supported (2 lines of code to be changed; no 
regression so far with python 2 and python 3).


My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type 
attributes


Petr, what do you think ?

Eloi


From: Python-ideas  on 
behalf of Eloi Gaudry 
Sent: Tuesday, May 8, 2018 9:26:47 AM
To: encu...@gmail.com; python-ideas@python.org
Subject: Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
>
> One reason is sub-interpreter support: you can have multiple
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
>
> With heap types, each sub-interpreter can have its own copy of the
> type
> object. But with builtins, changes done in one interpreter would be
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I guess) are only running
one interpreter.

In case several intepreters are used, it would make sense to have a
non-heap type that would be seen as a singleton across all of them, no
?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

The check is made against a boolean value in the C extension, I don't think 
that it offers a significant speed-up against the pure python code.

but it offers a simpler (reduced, global) assertion syntax though.


From: Python-ideas  on 
behalf of Kyle Lahnakoski 
Sent: Tuesday, June 5, 2018 7:25:45 PM
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active



I currently use the form

 and log_function(  )

where  is some module variable, usually "DEBUG".  I do this 
because it is one line, and it ensures the log_function parameters are not 
evaluated.

*IF* runtime assertions had a switch so they have no overhead when not active, 
how much faster can it get?  How expensive is the  check?


On 2018-05-10 03:55, Barry Scott wrote:

My logging example would be

log( control_flag, msg_expr )

expanding to:

if :
log_function(  )

Barry

This idea requires the same sort of machinery in python that I was hoping for 
to implement the short circuit logging.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

They are basically the same thing, with one difference being that 
runtime_assert would be used for extension mainly, and switchable on/off 
without using -o flag on the command line.

Eloi


From: Python-ideas  on 
behalf of Michel Desmoulin 
Sent: Tuesday, June 5, 2018 3:33:30 PM
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active

Maybe somebody already answered this, but what's the difference between
this and the keyword "assert", which basically can be stripped of using
"python -o" ?

Le 05/05/2018 à 10:04, Eloi Gaudry a écrit :
>
> Hi folks,
>
>
> I intend to add a runtime assertion feature in python. Before
> submitting a PEP, I am sending a draft to this mailing list to
> discuss whether it would make sense (above this message). I have
> actually been using it for the last 2 years and it has proven to be
> robust and has achieved its primary goal.
>
>
> Briefly, the idea is to add a new assert that can be switch on/off
> depending on some variable/mechanism at runtime. The whole point of
> this assert is that it should not bring any overhead when off, i.e.
> by avoiding evaluating the expression enclosed in the runtime
> assert. It thus relies on Python grammar.
>
>
> Thanks for your feedback.
>
>
> Eloi
>
>
>
>
>
>
>
>
>
>
>
> Abstract
>
> This PEP aims at offering a runtime assert functionnality, extending the
> compiletime assert already available.
>
>
> Rationale
>
> There is no runtime assert relying on python grammar available. For
> diagnostics and/or debugging reasons, it would be valuable to add such a
> feature.
>
> A runtime assert makes sense when extra checks would be needed in a
> production environment (where non-debug builds are used).
>
> By extending the current python grammar, it would be possible to limit
> the overhead induces by those runtime asserts when running in a non
> "assertive" mode (-ed). The idea here is to avoid evaluating the
> expression when runtime assertion is not active.
>
> A brief example why avoiding evaluating the expression is needed to
> avoid any overhead in case the runtime assert should be ignored.
>
> ::
>
> runtime_assert( 999 in { i:None for i in range( 1000 ) } )
>
>
> Usage
>
> ::
>
> runtime_assert( expr )
>
> #would result in if expr and runtime_assert_active:
>
> print RuntimeAssertionError()
>
>
> Implementation details
>
> There is already an implementation available, robust enough for
> production. The implementation is mainly grammar based, and thus the
> parser and the grammar needs to be updated:
>
>   * Python/graminit.c
>   * Python/ast.c
>   * Python/symtable.c
>   * Python/compile.c
>   * Python/Python-ast.c
>   * Python/sysmodule.c
>   * Modules/parsermodule.c
>   * Include/Python-ast.h
>   * Include/graminit.h
>   * Include/pydebug.h
>   * Grammar/Grammar
>   * Parser/Python.asdl
>   * Lib/lib2to3/Grammar.txt
>   * Lib/compiler/ast.py
>   * Lib/compiler/pycodegen.py
>   * Lib/compiler/transformer.py
>   * Lib/symbol.py
>   * Modules/parsermodule.c
>
>
> References
>
> [1]
>
>
>
> PEP 306, How to Change Python's Grammar
> (http://www.python.org/dev/peps/pep-0306)
>
>
> Copyright
>
> This document has been placed in the public domain.
>
>
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Thu, 2018-05-17 at 03:09 +1000, Steven D'Aprano wrote:
> On Wed, May 16, 2018 at 01:27:50PM +0000, Eloi Gaudry wrote:
> > On Wed, 2018-05-16 at 21:15 +1000, Steven D'Aprano wrote:
> > > On Wed, May 16, 2018 at 08:29:00AM +, Eloi Gaudry wrote:
> > > > Is there some interest in the proposal or should I finally
> > > > close
> > > > this
> > > > thread ?
> > > 
> > > I'm definitely interested in the concept, not the suggested
> > > syntax
> > > or semantics.
> > 
> > Would you briefly describe a syntax that would better fit this
> > concept
> > ?
> 
> The syntax is the minor point: you give is an ungainly name, 
> "runtime_assert", and your proposed PEP shows it requiring
> parentheses 
> as if it were an ordinary function.
> 
> The bigger problem is the semantics. As I already said in an earlier 
> email, you don't explain what "runtime_assert_active" is (is it a 
> per-module global variable? a single application-wide super-global?
> a 
> local variable? something else?) or how we are supposed to set it. 
I proposed to have several ways to set it (see previous answers) : one
would be extension based and the other would rely on having a builtin
value that one would be able to change using a set function of the
python core.

> That 
> too is an ungainly name, and the fact that there's only *one* such
> flag 
> (whether it is per module or not) makes this proposal useless for my 
> needs.
I agree with you on this : the main issue with my proposal is that
having only one assert (function with deferred evaluation) more is not
enough : we would need to cover several other scenarii such as an
extension wanting to activate its own asserts without activating other
extensions checks.


> Another problem is that your runtime_assert *prints* the error 
> message instead of raising an exception, and there's no way to 
> customise the message.
It actually raise an AssertionError.
I do agree that it should allow to set/display a message though.

> 
> But the most important reason is that I'm not really interested in 
> adding a new keyword for this. I would much prefer to explore ways
> to 
> allow ordinary functions to receive arguments and be able to delay 
> evaluation of those arguments until needed.
What would then be solution for implementing the runtime_assert with
the current python api ? A lambda ?

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Wed, 2018-05-16 at 21:15 +1000, Steven D'Aprano wrote:
> On Wed, May 16, 2018 at 08:29:00AM +0000, Eloi Gaudry wrote:
> > Is there some interest in the proposal or should I finally close
> > this
> > thread ?
> 
> I'm definitely interested in the concept, not the suggested syntax
> or 
> semantics.

Would you briefly describe a syntax that would better fit this concept
?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

Is there some interest in the proposal or should I finally close this
thread ?

Thanks for your feedback,
Eloi


> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

My choice of words might not be the best, yes.
I do see to different meanings and/or context for the historical assert and the 
one I propose:

1/
 the first one would be something saying that, as a developer, when writing 
>>> assert (expr)
 in my python code, I mean that all my unit tests and real life tests I could 
think of should succeed the test. I do mean "don't go further, I might not know 
where you come from or where you intend to go or why you are behaving as such, 
but you failed to meet this and/or this criteria/condition".

2/
 the second one is there to activate some other checks, not while developing, 
just at runtime when the user uses my extension and want to get some 
diagnostics/enforcing checks to happen, because he/she is using something I 
couldn't think of in the first place, something that would not have been 
checked before.

Yes, those checks might be considered as identical in a language sense, but 
then : as an extension/interpreter writer, why should I only rely on the debug 
assert available today? Why would it not make sense to offer another assert, 
semantically different, aiming at runtime checks issues and this time where 
control is indeed by the consumer/the extension?





-Original Message-
From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft...@python.org> On 
Behalf Of Chris Angelico
Sent: Tuesday, May 8, 2018 7:38 PM
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active

On Wed, May 9, 2018 at 1:51 AM, Eloi Gaudry <eloi.gau...@fft.be> wrote:
> I think that is a difference between:
> - the current 'assert' which usage seems (to me) to focus on 
> development correctness (I think of it as the C-assert enabled in any 
> C program in debug build )
> - the runtime_assert that I submitted on the list, which would be 
> focusing on usage correctness (hence runtime), and easily disabled at 
> runtime (when the python command line options parsing is not an 
> option, for instance when the python interpreter is not python itself 
> and/or when the consumer/extension wants to behave differently).

What's the difference between "development correctness" and "usage 
correctness"? Does the latter depend on user input at run time? I still don't 
understand the distinction you're trying to make here.

ChrisA

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Wed, 2018-05-09 at 01:34 +1000, Steven D'Aprano wrote:
> On Tue, May 08, 2018 at 07:37:55AM +0000, Eloi Gaudry wrote:
> 
> > * debug assert, for helping developing a new features, used with
> > debug-
> > builds
> > * runtime assert, for ensuring correctness and/or diagnosing issue
> > with
> > production/release-builds on-site.
> 
> I don't think that is a useful distinction to make.
> 
> I use `assert` all the time, and I have never once cared whether it
> is a 
> "debug build" or a "production build". I use assert in my code, and
> then 
> people (including me) can use it with whatever build of Python they 
> like.
I do understand your point but I don't share your opinion.
I think that is a difference between:
- the current 'assert' which usage seems (to me) to focus on
development correctness (I think of it as the C-assert enabled in any C
program in debug build )
- the runtime_assert that I submitted on the list, which would be
focusing on usage correctness (hence runtime), and easily disabled at
runtime (when the python command line options parsing is not an option,
for instance when the python interpreter is not python itself and/or
when the consumer/extension wants to behave differently).

> I don't even know which builds of Python I'm running. My *guess* is
> that 
> the OS-provided Python is probably a non-debug build, and the ones
> I've 
> compiled from source will be whatever the default build settings
> are, 
> but I don't know what that is or how to find out.
> 
> And if there was some sort of runtime_assert that could be enabled
> and 
> disabled individually, why wouldn't I use it with debug builds as
> well 
> as non-debug builds?
There would be no reason why, but you would benefit from being able to
easily activate/deactivate the runtime assert.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Tue, 2018-05-08 at 09:58 -0400, Eric V. Smith wrote:
> I think what's confusing to me (and maybe others) is that we haven't 
> seen your vision on how this would look in Python code.
> 
> An example that would throw runtime assertions and the same example 
> where it wouldn't (after a global switch is set?) would be helpful.
> 
> Eric

Eric, I can only agree to your comment.

As a matter of fact, I have only used the case where the runtime assert
is activated through an extension. In this case, adding a set method is
simple : in our framework, we would set the variable
Py_RuntimeAssertFlag to 1, depending on some command line parameters,
or settings found in a configuration file).

In pure python, if something as having a method registered in
__builtins__ make sense, it could be used to trigger the assertive
behavior.

In this example, let's assume that you may want to perform some basic
check on a given file in diagnostics mode:

>>> def check_mounts_size():
...   return len( open( '/proc/self/mounts', 'r' ).readlines() )>1024
...
>>> runtime_assert( len( open( '/proc/self/mounts', 'r'
).readlines()>1024 ) )
>>> runtime_assert( check_mounts_size() )

>>> __builtins__.set_runtime_assert( True )

>>> runtime_assert( len( open( '/proc/self/mounts', 'r'
).readlines()>1024 ) )
Traceback (most recent call last):
  File "", line 1, in 
AssertionError

>>> runtime_assert( check_mounts_size() )
Traceback (most recent call last):
  File "", line 1, in 
AssertionError
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Tue, 2018-05-08 at 07:37 -0700, Guido van Rossum wrote:
> > > In Lisp this would be easy. :-)
> > 
> > and he already has a diff ready for review if needed (basically
> > very
> > similar to the current 'assert' implementation :)
> 
> That seems premature. There is not even a hint of agreement that such
> a feature would be useful *in general* (I'm not doubting your
> situation) and worth our limited volunteer developer resources to
> maintain, document etc. for decades to come.

I didn't mean to push too fast, just to show how things could be
implemented and maybe clarify my comments/statements.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Tue, 2018-05-08 at 09:35 -0400, Eric V. Smith wrote:
> On 5/8/2018 3:37 AM, Eloi Gaudry wrote:
> > On Mon, 2018-05-07 at 16:56 +, Brett Cannon wrote:
> > > 
> > > My question is how is this different to running with -O which
> > > leaves
> > > the assert statement out of the bytecode and so you avoid any
> > > run-
> > > time cost of the statement entirely?
> > 
> > Not so much different, except that:
> > - the switch won't need to be on the command line
> 
> So, is the switch set in code? If so, what would that look like?
> 
> Eric
So far, I only have the "extension" method, where a global variable is
set directly from within the C-interface, as currently done with the
debug assert:

Include/pydebug.h : int Py_RuntimeAssertFlag = 1;
Your_extension/main.c : SetFlag(Py_RuntimeAssertFlag);
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Mon, 2018-05-07 at 16:56 +, Brett Cannon wrote:
> 
> My question is how is this different to running with -O which leaves
> the assert statement out of the bytecode and so you avoid any run-
> time cost of the statement entirely? 

Not so much different, except that:
- the switch won't need to be on the command line
- the 2 different asserts would be considered differently by
developers/users : 
* debug assert, for helping developing a new features, used with debug-
builds
* runtime assert, for ensuring correctness and/or diagnosing issue with
production/release-builds on-site.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:
> On 05/07/18 11:37, Eloi Gaudry wrote:
> > I mean, to my knowledge, there is no reason why a type should be
> > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html
> > ) to
> > be able to change its attributes at Python level.
> 
> One reason is sub-interpreter support: you can have multiple 
> interpreters per process, and those shouldn't influence each other.
> (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor
> t)
> 
> With heap types, each sub-interpreter can have its own copy of the
> type 
> object. But with builtins, changes done in one interpreter would be 
> visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature
neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable
across interpreters. And we (as most users I guess) are only running
one interpreter.

In case several intepreters are used, it would make sense to have a
non-heap type that would be seen as a singleton across all of them, no
?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Mon, 2018-05-07 at 10:52 -0700, Guido van Rossum wrote:
> On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka  > wrote:
> > I just don't understand why you need a new keyword for writing
> > runtime checks.
> 
> Oh, that's pretty clear. The OP wants to be able to turn these checks
> off with some flag he can set/clear at runtime, and when it's off he
> doesn't want to incur the overhead of evaluating the check. The
> assert statement has the latter property, but you have to use -O to
> turn it off. He basically wants a macro so that
> 
>   runtime_assert()
> 
> expands to
> 
>   if  and ():
>   raise AssertionError
> 
> In Lisp this would be easy. :-)

and he already has a diff ready for review if needed (basically very
similar to the current 'assert' implementation :)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

Hi,

I'd like to bring back this discussion (from 2005, by Greg):
https://bugs.python.org/issue1229239

Briefly, non-heap types cannot have their
attributes changed by Python code. This makes sense for python builtin
types, but not for the types defined in extension/modules.

As we have been using this patch for the very same reasons and for more
than 10 years, I think it might make sense to reconsider the discussion
that Greg started.

The main question at that time was "why not using a heap type instead
?" (because heap-type do not have this limitation).

But I think that the right question could have been "why imposing such
a restriction on non-heap types as defined in (non Python core)
extensions ?".

I mean, to my knowledge, there is no reason why a type should be
allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to
be able to change its attributes at Python level.

I'm not saying that all non-heap types should be able to do so, just
that it would make sense to allow this behavior, as an option (bit
flag).

At the implementation level, the changes needed are really limited
(about a few lines):
- Include/object.h
- Objects/typeobject.c:

Eloi
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Sun, 2018-05-06 at 01:30 +1000, Steven D'Aprano wrote:
> On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote:
> 
> > Hi folks,
> > I intend to add a runtime assertion feature in python.
> 
> I'm very interested in this idea, but I'm afraid your draft PEP
> isn't 
> very clear about this feature. Comments below.
Thanks, I'll try to clarify this here.

> > Rationale
> > There is no runtime assert relying on python grammar available.
> > For 
> > diagnostics and/or debugging reasons, it would be valuable to add
> > such 
> > a feature.
> 
> Why not this?
> 
> if DEBUG_ENABLED:
> run_check(expression)
> 
> where run_check is an ordinary function. If DEBUG_ENABLED is false,
> the 
> only runtime overhead is the name lookup and bool test, the
> expression 
> is not evaluated.
for the sake of clarity across any Python application/extension for
instance, and because having run_check defined as a statement seems
actually the good thing to do.
> 
> What features does your suggestion offer over this simple technique?
Code readiness I think (kiss) : simpler and shorter.
The simple technique would still be used for other purpose (assert that
are context based for instance).


> 
> [...]
> > A brief example why avoiding evaluating the expression is needed
> > to 
> > avoid any overhead in case the runtime assert should be ignored.
> > ::
> > runtime_assert( 999 in { i:None for i in range( 1000 ) } )
> 
> You say that you have been using this feature in production for two 
> years. You should give real-life examples, not fake examples like
> the 
> above. That example would be better written as:
> 
> runtime_assert(True)
> 
> since no part of the test actually depends on either the enormous
> dict 
> or range objects that you build only to throw away.
Real life examples would be more application-based I think.

In my case, I used this for switching off some heaving computing
expression evaluation at runtime. Those expression would have been
responsible for sorting vectors and communicating them through MPI
using some method from our framework, in order to finally return a
boolean.


> > Usage
> > ::
> > 
> > runtime_assert( expr )
> > 
> > #would result in 
> > if expr and runtime_assert_active:
> > print RuntimeAssertionError()
> 
> Surely the test should be the other way around?
> 
> if runtime_assert_active and expr:
> print(RuntimeAssertionError())
otherwise the expression is evaluated regardless of whether the
runtime 
> assertions are active or not.
You are absolutely right :)

> Please use Python 3 syntax for the PEP, as Python 2 is in
> maintenance 
> mode and will absolutely not get any new language features.
Sure

> 
> Some more observations:
> 
> I like the idea of more control over runtime checking, but your PEP 
> should justify why this is a good idea and why "just write more unit 
> tests" isn't the solution.
My main reasons is the one I gave in the context description of the
PEP. Assert remains an efficient way for controlling your application
behavior when running in production (as opposed to a debug build).

Writing more unit tests is always a good practice, but :
- it does not necessarily cover all possible paths that are runtime
dependent (an application relies on various app-parameters and
architecture-parameters for instance)
- it does not offer a solution for your extension/application consumers
(that might be using your application in a very specific set of
paths/set of decision/etc.). 

> What happens if the caller has defined their own function or
> variable 
> called runtime_assert?
making runtime_assert a statement (as assert is already) would forbid
such a definition, thus it would break backward compatibility (in those
specific cases) for sure.


> Can the caller customise the assertion error message?
At this stage, no, but I don't know if this would be something
difficult to add.

> If you are changing the grammar, why not make runtime_assert a
> statement?
I'm not really changing the grammar, just making 'runtime_assert' a
statement as 'assert' already is (and IMHO could have been named as
debug_assert). There is no significant difference between both assert
in term of implementation.

> Can the caller set an application-wide global across all modules, or
> do 
> they have to set
> 
> runtime_assert_active = True
> 
> in every module they want to use runtime_assert?
I would vote in favor of having two different ways to activate or not
the runtime assertion evaluation:
- the first one would consist of controlling a global variable at the
C-level (the current recipe in cPython) and would be used by python
extension. This is the 

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

I didn't mean to replace the current (debug) assert but I wanted to add another 
one that would allow to be switch on or off on production builds.

The need for a new keyword (not syntax) comes from this difference.


 I cannot think of another example that would convince you of the benefit of 
having a specific keyword for such a runtime assert.  I do believe that having 
such a feature in non-debug build is more than interesting but indeed needed.


At some point, there should be a trade-off between to relying on a simple 
keyword/mechanism (for a runtime assert) versus the development cost of adding 
such a feature and maintaining it. Anyway, thanks for your feedback Serhiy, it 
helps.





From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft...@python.org> on 
behalf of Serhiy Storchaka <storch...@gmail.com>
Sent: Saturday, May 5, 2018 17:44
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active

05.05.18 18:04, Eloi Gaudry пише:
> By 'self-contained', I meant that using the assert keyword and its
> expression is sufficient. An inline assertive expression as the one you
> describe does not fulfill this assert requirement.

Sufficient for what? And why writing with using the existing syntax is
not sufficient?

> My proposal is simply to extend the current assert to non-debug builds
> and allow to switch it off/on at runtime.

The main property of the assert statement is that has a zero overhead in
non-debug run. If you remove this property, it will be not the assert
statement, and you will not need a special syntax support for writing
this runtime check.

> The syntax example I gave illustrated what I meant by syntax aware.

It doesn't illustrate why a new syntax is necessary. Or I can't
understand this illustration.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

By 'self-contained', I meant that using the assert keyword and its expression 
is sufficient. An inline assertive expression as the one you describe does not 
fulfill this assert requirement.

My proposal is simply to extend the current assert to non-debug builds and 
allow to switch it off/on at runtime.

The syntax example I gave illustrated what I meant by syntax aware.



De : Serhiy Storchaka
Envoyé : samedi 5 mai à 15:10
Objet : Re: [Python-ideas] Runtime assertion with no overhead when not active
À : python-ideas@python.org


05.05.18 15:54, Eloi Gaudry пише: > I meant avoiding the overhead of the 
expression evaluation enclosed in the assert itself, not the active/disabled 
state (that comes at virtually no cost). > ex: >>>> runtime_assert( { i:None 
for i in range( 1000 ) } ) > > By using the syntax you describe ('boolean 
and not expr'), you loose all the benefit of the Python grammar: > - 
self-contained Could you please explain what you mean? > - able to catch 
semantic/syntax bugs > > ex: >>>> f=1 >>>> runtime_assert( f==1 ) >>>> 
runtime_assert( f=1 ) > File "", line 1 > runtime_assert( f=1 ) > ^ > 
SyntaxError: invalid syntax Until inline assignment expression is implemented, 
this is an error too: >>> if debug and not f=1: File "", line 1 if debug and 
not f=1: ^ SyntaxError: invalid syntax 
___ Python-ideas mailing list 
Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas 
Code of Conduct: http://python.org/psf/codeofconduct/



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

I meant avoiding the overhead of the expression evaluation enclosed in the 
assert itself, not the active/disabled state (that comes at virtually no cost).
ex:
>>> runtime_assert( { i:None for i in range( 1000 ) } )  

By using the syntax you describe ('boolean and not expr'), you loose all the 
benefit of the Python grammar:
- self-contained
- able to catch semantic/syntax bugs

ex:
>>> f=1
>>> runtime_assert( f==1 )
>>> runtime_assert( f=1 )
  File "", line 1
runtime_assert( f=1 )
 ^
SyntaxError: invalid syntax


-Original Message-
From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft...@python.org> On 
Behalf Of Serhiy Storchaka
Sent: Saturday, May 5, 2018 2:05 PM
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active

05.05.18 11:04, Eloi Gaudry пише:
> Briefly, the idea is to add a new assert that can be switch on/off
> depending on some variable/mechanism at runtime. The whole point of
> this assert is that it should not bring any overhead when off, i.e.
> by avoiding evaluating the expression enclosed in the runtime
> assert. It thus relies on Python grammar.

You should have an overhead for checking if it is switched on/off, isn't? And 
this overhead is virtually the same as testing the value of the global boolean 
variable.

> runtime_assert( expr )
> 
> #would result in if expr and runtime_assert_active:
> 
> print RuntimeAssertionError()

How this will be different from

 if debug and not expr:
 print(RuntimeAssertionError())

?

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

Hi folks,
I intend to add a runtime assertion feature in python. Before submitting a PEP, 
I am sending a draft to this mailing list to discuss whether it would make 
sense (above this message). I have actually been using it for the last 2 years 
and it has proven to be robust and has achieved its primary goal.
Briefly, the idea is to add a new assert that can be switch on/off depending on 
some variable/mechanism at runtime. The whole point of this assert is that it 
should not bring any overhead when off, i.e. by avoiding evaluating the 
expression enclosed in the runtime assert. It thus relies on Python grammar.
Thanks for your feedback.
Eloi



Abstract

This PEP aims at offering a runtime assert functionnality, extending the 
compiletime assert already available.

Rationale

There is no runtime assert relying on python grammar available. For diagnostics 
and/or debugging reasons, it would be valuable to add such a feature.

A runtime assert makes sense when extra checks would be needed in a production 
environment (where non-debug builds are used).

By extending the current python grammar, it would be possible to limit the 
overhead induces by those runtime asserts when running in a non "assertive" 
mode (-ed). The idea here is to avoid evaluating the expression when runtime 
assertion is not active.

A brief example why avoiding evaluating the expression is needed to avoid any 
overhead in case the runtime assert should be ignored.
::
runtime_assert( 999 in { i:None for i in range( 1000 ) } )
Usage
::

runtime_assert( expr )

#would result in if expr and runtime_assert_active:
print RuntimeAssertionError()
Implementation details

There is already an implementation available, robust enough for production. The 
implementation is mainly grammar based, and thus the parser and the grammar 
needs to be updated:

  *   Python/graminit.c
  *   Python/ast.c
  *   Python/symtable.c
  *   Python/compile.c
  *   Python/Python-ast.c
  *   Python/sysmodule.c
  *   Modules/parsermodule.c
  *   Include/Python-ast.h
  *   Include/graminit.h
  *   Include/pydebug.h
  *   Grammar/Grammar
  *   Parser/Python.asdl
  *   Lib/lib2to3/Grammar.txt
  *   Lib/compiler/ast.py
  *   Lib/compiler/pycodegen.py
  *   Lib/compiler/transformer.py
  *   Lib/symbol.py
  *   Modules/parsermodule.c

References
[1]

PEP 306, How to Change Python's Grammar 
(http://www.python.org/dev/peps/pep-0306)

Copyright

This document has been placed in the public domain.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/