Re: [Python-Dev] PEP 451 update

2013-10-31 Thread Eric Snow 
On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan  wrote:
> There's also the option of implementing the constraint directly in the
> finder, which *does* have the necessary info (with the change to pass the
> previous spec to find_spec).

Yeah, I thought of that.  I just prefer the more explicit
supports_reload().  That said...

>
> I still think it makes more sense to leave this out for the moment - it's
> not at all clear we need the extra method, and adding it later would be a
> straightforward protocol update.

...I agree that makes the most sense for now. :)

BTW, thanks for pushing these issues.  I think the API has gotten
pretty solid.  I just need to make sure the PEP covers the cases and
conclusions we're discussing.

-eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-31 Thread Nick Coghlan 
On 31 Oct 2013 18:52, "Eric Snow"  wrote:
>
> On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan  wrote:
> > There's also the option of implementing the constraint directly in the
> > finder, which *does* have the necessary info (with the change to pass
the
> > previous spec to find_spec).
>
> Yeah, I thought of that.  I just prefer the more explicit
> supports_reload().  That said...
>
> >
> > I still think it makes more sense to leave this out for the moment -
it's
> > not at all clear we need the extra method, and adding it later would be
a
> > straightforward protocol update.
>
> ...I agree that makes the most sense for now. :)
>
> BTW, thanks for pushing these issues.  I think the API has gotten
> pretty solid.  I just need to make sure the PEP covers the cases and
> conclusions we're discussing.

Thanks are also due to PJE for making me realise we were handwaving too
much when it came to the expected reload semantics :)

Cheers,
Nick.

>
> -eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear?

2013-10-31 Thread Victor Stinner 
2013/10/29 Victor Stinner :
> 2013/10/29 Kristján Valur Jónsson :
>> I was thinking something similar.  It would be useful to be able to "pause" 
>> and "resume"
>> if one is doing any analysis work in the live environment.  This would 
>> reduce the
>> need to have "Filter" objects.
>
> Internally, tracemalloc uses a thread-local variable (called the
> "reentrant" flag) to disable temporarly tracing allocations in the
> current thread. It only disables tracing new allocations,
> deallocations are still proceed.

If I give access to this flag, it would be possible to disable
temporarily tracing in the current thread, but tracing would still be
enabled in other threads. Would it fit your requirement?

Example:
---
tracemalloc.enable()
# start your application
...
# spawn many threads
...
# oh no, I don't want to trace this ugly function
tracemalloc.disable_local()
ugly_function()
tracemalloc.enable_local()
...
snapshot = take_snapshot()
---

You can imagine a context manager based on these two functions:
---
with disable_tracing_temporarily_in_current_thread():
  ugly_function()
---

I still don't understand why you would need to stop tracing
temporarily. When I use tracemalloc, I never disable it.

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear?

2013-10-31 Thread Victor Stinner 
2013/10/31 Victor Stinner :
> If I give access to this flag, it would be possible to disable
> temporarily tracing in the current thread, but tracing would still be
> enabled in other threads. Would it fit your requirement?

It's probably not what you are looking for :-)

As I wrote in the PEP, the API of tracemalloc was inspired by the
faulthandler module. enable() / disable() makes sense in faulthandler
because faulthandler is passive: it only do something on a trigger
(synchonous signals like SIGFPE or SIGSEGV). I realized that
tracemalloc is different: as written in the documentation, enable()
*starts* tracing. After enable() has been called, tracemalloc becomes
active. So tracemalloc should use names start() / stop() rather than
enable() / disable().

I did another experiment. I replaced enable/disable/is_enabled with
start/stop/is_tracing, and added enable/disable/is_enabled functions
to disable temporarily tracing.

API:

- clear_traces(): clear traces
- start(): start tracing (the old "enable")
- stop(): stop tracing and clear traces (the old "disable")
- disable(): disable temporarily tracing
- enable(): reenable tracing
- is_tracing(): True if tracemalloc is tracing, False otherwise (the
old "is_enabled")
- is_enabled(): True if tracemalloc is enabled, False otherwise

All these functions are process-wide (affect all threads).

tracemalloc is only tracing new allocations if is_tracing() and
is_enabled() are True.

If is_tracing() is True and is_enabled() is False, deallocations still
remove traces (otherwise, the internal dictionary of traces would
become inconsistent).

Example:
---
tracemalloc.start()
# start your application
...
useful = UsefulObject()
huge = HugeObject()
...
snapshot1 = take_snapshot()
...
# oh no, I don't want to trace this ugly object, but please don't
trash old traces
tracemalloc.disable()
ugly = ugly_object()
...
# release memory of the huge object
huge = None
...
# restart tracing (ugly is still alive)
tracemalloc.enable()
...
snapshot2 = take_snapshot()
tracemalloc.stop()
---

snapshot1 contains traces of objects:
- useful
- huge

snapshot2 contains traces of objects:
- useful

huge is missing from snapshot2 even if the module was disabled. ugly
is missing from snapshot2 because tracing was disabled.

Does it look better? I don't see the usecase of disable() / enable()
yet, but it's cheap (it just add a flag).

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear?

2013-10-31 Thread Ethan Furman 

On 10/31/2013 05:20 AM, Victor Stinner wrote:

I did another experiment. I replaced enable/disable/is_enabled with
start/stop/is_tracing, and added enable/disable/is_enabled functions
to disable temporarily tracing.

API:

- clear_traces(): clear traces
- start(): start tracing (the old "enable")
- stop(): stop tracing and clear traces (the old "disable")
- disable(): disable temporarily tracing
- enable(): reenable tracing
- is_tracing(): True if tracemalloc is tracing, False otherwise (the
old "is_enabled")
- is_enabled(): True if tracemalloc is enabled, False otherwise


These names make more sense.  However, `stop` is still misleading as it both stops and destroys data.  An easy fix for 
that is for stop to save the data somewhere so get_traces (or whatever) can still retrieve it.


If `stop` really must destroy the data, perhaps it should be called `close` instead; StringIO has a similar close method 
that when called destroys any stored data, and get_value must be called first if that data is wanted.


--
~Ethan~
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Compiler security

2013-10-31 Thread MRAB 

Has anybody here heard about this, and, if so, is it anything we should
be thinking about:

How your compiler may be compromising application security
http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiler security

2013-10-31 Thread Christian Heimes 
Am 31.10.2013 15:48, schrieb MRAB:
> Has anybody here heard about this, and, if so, is it anything we should
> be thinking about:
> 
> How your compiler may be compromising application security
> http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security

http://bugs.python.org/issue17405 addresses one issue with dead code
elimination.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiler security

2013-10-31 Thread Guido van Rossum 
Interesting read. I'm surprised that the researchers didn't contact us,
since the article mentions they found 5 bugs in Python. Regarding security:
the article seems to use that term mostly to attract eyeballs; there are no
specifics, just the implication that this *could* affect security.

But it's hardly news -- as GCC versions became more aggressive we've had to
fix our share of undefined code in Python. Usually the unittests catch
these early.


On Thu, Oct 31, 2013 at 7:48 AM, MRAB  wrote:

> Has anybody here heard about this, and, if so, is it anything we should
> be thinking about:
>
> How your compiler may be compromising application security
> http://www.itworld.com/**security/380406/how-your-**
> compiler-may-be-compromising-**application-security
> __**_
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiler security

2013-10-31 Thread Benjamin Peterson 
I believe the 5 problems they found in Python were dealt with here
http://bugs.python.org/issue17016

2013/10/31 MRAB :
> Has anybody here heard about this, and, if so, is it anything we should
> be thinking about:
>
> How your compiler may be compromising application security
> http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiler security

2013-10-31 Thread Christian Heimes 
Am 31.10.2013 15:48, schrieb MRAB:
> Has anybody here heard about this, and, if so, is it anything we should
> be thinking about:
> 
> How your compiler may be compromising application security
> http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security
> 

I didnt' see this at first:

  STACK was run against a number of systems written in C/C++  and
  it found 160 new bugs in the systems tested, including ...
  and Python (5).

Has anybody contact us? I neither saw a bug report nor a mail to PSRT.

Christian

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-31 Thread PJ Eby 
On Thu, Oct 31, 2013 at 5:52 AM, Nick Coghlan  wrote:
>
> On 31 Oct 2013 18:52, "Eric Snow"  wrote:
>>
>> On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan  wrote:
>> > There's also the option of implementing the constraint directly in the
>> > finder, which *does* have the necessary info (with the change to pass
>> > the
>> > previous spec to find_spec).
>>
>> Yeah, I thought of that.  I just prefer the more explicit
>> supports_reload().  That said...
>>
>> >
>> > I still think it makes more sense to leave this out for the moment -
>> > it's
>> > not at all clear we need the extra method, and adding it later would be
>> > a
>> > straightforward protocol update.
>>
>> ...I agree that makes the most sense for now. :)
>>
>> BTW, thanks for pushing these issues.  I think the API has gotten
>> pretty solid.  I just need to make sure the PEP covers the cases and
>> conclusions we're discussing.
>
> Thanks are also due to PJE for making me realise we were handwaving too much
> when it came to the expected reload semantics :)

You're welcome.  ;-)  But speaking of handwaving, I also want to be
sure that loader developers know that "reloading" is only really
"reloading" if there's a previous existing spec, or the module type
is...

Hm.   Actually, I think I now know how to state what's bugging me
every time I see this "supports_reload()" or "reload=True" or other
reloading flags in this process.

I think that references to reloading should be replaced with
references to what's *actually* at issue, because "reloading" itself
is vague and carries too many assumptions for a loader author to
understand or get right.  (Look how hard it is for *us*!)

That is, I think we should clarify what use cases there are for
knowing whether a "reload" is happening, and address those use cases
explicitly rather than lumping them under a general heading.

For example, if the reason a loader cares about reloading is because
it's a C extension using a custom module type, and the existing module
isn't of the right type, then we should just spell out how to handle
it.  (e.g. raise an exception)

If the reason a loader cares about reloading is because of some sort
of caching or reuse, then we should just spell out how to handle that,
too.

Lumping these cases together under a "reloading" flag or a check for
"reloading" support is a nasty code smell, because it requires a
loader developer to have the *same* vaguely-defined idea of
"reloading" as the PEP authors.  ;-)

I also suspect, that if properly spelled out, those use cases are
going to boil down to:

1. Throwing errors if you have an existing module object you can't
load into, and
2. Passing in a previous spec object, if available

In other words, loaders should not really have any responsibility for
or concept of "reloading" -- they always load into a module object
(that they may or may not have created), and they may get given a spec
from a previous load.  They should deal only in "module reuse" and
"spec reuse".  While a typical reload() might involve both reuses,
there are cases where one sort of reuse could occur independently, and
not all loaders care about both (or even either) condition.

At any rate, it means a loader author doesn't have to figure out how
to handle "reloading", all they have to figure out is whether they can
load into a particular module object, and whether they can do
something useful with a spec that was previously used to load a module
with the same name -- a spec that may or may not refer to a similar
previous loader.  These are rather more well-defined endeavors than
trying to determine in the abstract whether one "supports reload".
;-)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiler security

2013-10-31 Thread Serhiy Storchaka 

31.10.13 16:56, Benjamin Peterson написав(ла):

I believe the 5 problems they found in Python were dealt with here
http://bugs.python.org/issue17016


Ah, now I have remembered author's name.

http://bugs.python.org/issue18684 contains some other fixes of this kind.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiler security

2013-10-31 Thread Terry Reedy 

On 10/31/2013 10:57 AM, Christian Heimes wrote:


I didnt' see this at first:

   STACK was run against a number of systems written in C/C++  and
   it found 160 new bugs in the systems tested, including ...
   and Python (5).

Has anybody contact us? I neither saw a bug report nor a mail to PSRT.


from http://css.csail.mit.edu/stack/

Our software is hosted on Github at https://github.com/xiw/stack/.
Obtain the latest code of STACK using:
git clone git://github.com/xiw/stack
See README and INSTALL for more information.

--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-31 Thread Nick Coghlan 
On 1 Nov 2013 01:37, "PJ Eby"  wrote:
.  ;-)
>
> I also suspect, that if properly spelled out, those use cases are
> going to boil down to:
>
> 1. Throwing errors if you have an existing module object you can't
> load into, and
> 2. Passing in a previous spec object, if available
>
> In other words, loaders should not really have any responsibility for
> or concept of "reloading" -- they always load into a module object
> (that they may or may not have created), and they may get given a spec
> from a previous load.  They should deal only in "module reuse" and
> "spec reuse".  While a typical reload() might involve both reuses,
> there are cases where one sort of reuse could occur independently, and
> not all loaders care about both (or even either) condition.
>
> At any rate, it means a loader author doesn't have to figure out how
> to handle "reloading", all they have to figure out is whether they can
> load into a particular module object, and whether they can do
> something useful with a spec that was previously used to load a module
> with the same name -- a spec that may or may not refer to a similar
> previous loader.  These are rather more well-defined endeavors than
> trying to determine in the abstract whether one "supports reload".
> ;-)

It may not have been clear from the email exchange, but that's basically
where we ended up :)

The change Eric is currently going to make to the PEP is to add an optional
"previous" parameter to the various find_spec APIs.

At the time when find_spec runs, sys.modules hasn't been touched yet, so
the old module state is still available when reloading. Passing the spec in
lets the loader decide whether or not it can actually load that module
given the information about the previous load operation.

However, perhaps it makes more sense to pass the entire existing module,
rather than just the spec? I'd like to minimise the need for new-style
loader authors to retrieve things directly from the sys module, and you're
right that "can I reload into this particular module?" is a slightly
different question from "can I reload a module previously loaded using this
particular spec?".

A spec-based API could still be used to answer the first question by
looking up sys.modules, but so could the name based API. Passing in the
module reference, on the other hand, should let loaders answer both
questions without needing to touch the sys module.

I also now think this pass-the-module approach when finding the spec
approach could also clean up some of the messiness that is __main__
initialisation, where we repeatedly load things into the same module object.

Let's say we be completely explicit and call the new parameter something
like "load_target". If we do that then I would make the *same* change to
runpy.run_path and runpy.run_module: let you pass in an existing module
object under that name to say "hey, run in this namespace, rather than a
fresh one". (Those APIs currently only support pre-populating a *fresh*
namespace, rather than allowing you to directly specify an existing one)

Most loaders won't care, but those that do will have all the info they need
to throw an exception saying "I could provide a spec for this, but it's not
compatible with that load target".

Cheers,
Nick.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com