Re: [Python-Dev] PEP 451 update
On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan ncogh...@gmail.com wrote: There's also the option of implementing the constraint directly in the finder, which *does* have the necessary info (with the change to pass the previous spec to find_spec). Yeah, I thought of that. I just prefer the more explicit supports_reload(). That said... I still think it makes more sense to leave this out for the moment - it's not at all clear we need the extra method, and adding it later would be a straightforward protocol update. ...I agree that makes the most sense for now. :) BTW, thanks for pushing these issues. I think the API has gotten pretty solid. I just need to make sure the PEP covers the cases and conclusions we're discussing. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On 31 Oct 2013 18:52, Eric Snow ericsnowcurren...@gmail.com wrote: On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan ncogh...@gmail.com wrote: There's also the option of implementing the constraint directly in the finder, which *does* have the necessary info (with the change to pass the previous spec to find_spec). Yeah, I thought of that. I just prefer the more explicit supports_reload(). That said... I still think it makes more sense to leave this out for the moment - it's not at all clear we need the extra method, and adding it later would be a straightforward protocol update. ...I agree that makes the most sense for now. :) BTW, thanks for pushing these issues. I think the API has gotten pretty solid. I just need to make sure the PEP covers the cases and conclusions we're discussing. Thanks are also due to PJE for making me realise we were handwaving too much when it came to the expected reload semantics :) Cheers, Nick. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc) disable == clear?
2013/10/29 Victor Stinner victor.stin...@gmail.com: 2013/10/29 Kristján Valur Jónsson krist...@ccpgames.com: I was thinking something similar. It would be useful to be able to pause and resume if one is doing any analysis work in the live environment. This would reduce the need to have Filter objects. Internally, tracemalloc uses a thread-local variable (called the reentrant flag) to disable temporarly tracing allocations in the current thread. It only disables tracing new allocations, deallocations are still proceed. If I give access to this flag, it would be possible to disable temporarily tracing in the current thread, but tracing would still be enabled in other threads. Would it fit your requirement? Example: --- tracemalloc.enable() # start your application ... # spawn many threads ... # oh no, I don't want to trace this ugly function tracemalloc.disable_local() ugly_function() tracemalloc.enable_local() ... snapshot = take_snapshot() --- You can imagine a context manager based on these two functions: --- with disable_tracing_temporarily_in_current_thread(): ugly_function() --- I still don't understand why you would need to stop tracing temporarily. When I use tracemalloc, I never disable it. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc) disable == clear?
2013/10/31 Victor Stinner victor.stin...@gmail.com: If I give access to this flag, it would be possible to disable temporarily tracing in the current thread, but tracing would still be enabled in other threads. Would it fit your requirement? It's probably not what you are looking for :-) As I wrote in the PEP, the API of tracemalloc was inspired by the faulthandler module. enable() / disable() makes sense in faulthandler because faulthandler is passive: it only do something on a trigger (synchonous signals like SIGFPE or SIGSEGV). I realized that tracemalloc is different: as written in the documentation, enable() *starts* tracing. After enable() has been called, tracemalloc becomes active. So tracemalloc should use names start() / stop() rather than enable() / disable(). I did another experiment. I replaced enable/disable/is_enabled with start/stop/is_tracing, and added enable/disable/is_enabled functions to disable temporarily tracing. API: - clear_traces(): clear traces - start(): start tracing (the old enable) - stop(): stop tracing and clear traces (the old disable) - disable(): disable temporarily tracing - enable(): reenable tracing - is_tracing(): True if tracemalloc is tracing, False otherwise (the old is_enabled) - is_enabled(): True if tracemalloc is enabled, False otherwise All these functions are process-wide (affect all threads). tracemalloc is only tracing new allocations if is_tracing() and is_enabled() are True. If is_tracing() is True and is_enabled() is False, deallocations still remove traces (otherwise, the internal dictionary of traces would become inconsistent). Example: --- tracemalloc.start() # start your application ... useful = UsefulObject() huge = HugeObject() ... snapshot1 = take_snapshot() ... # oh no, I don't want to trace this ugly object, but please don't trash old traces tracemalloc.disable() ugly = ugly_object() ... # release memory of the huge object huge = None ... # restart tracing (ugly is still alive) tracemalloc.enable() ... snapshot2 = take_snapshot() tracemalloc.stop() --- snapshot1 contains traces of objects: - useful - huge snapshot2 contains traces of objects: - useful huge is missing from snapshot2 even if the module was disabled. ugly is missing from snapshot2 because tracing was disabled. Does it look better? I don't see the usecase of disable() / enable() yet, but it's cheap (it just add a flag). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc) disable == clear?
On 10/31/2013 05:20 AM, Victor Stinner wrote: I did another experiment. I replaced enable/disable/is_enabled with start/stop/is_tracing, and added enable/disable/is_enabled functions to disable temporarily tracing. API: - clear_traces(): clear traces - start(): start tracing (the old enable) - stop(): stop tracing and clear traces (the old disable) - disable(): disable temporarily tracing - enable(): reenable tracing - is_tracing(): True if tracemalloc is tracing, False otherwise (the old is_enabled) - is_enabled(): True if tracemalloc is enabled, False otherwise These names make more sense. However, `stop` is still misleading as it both stops and destroys data. An easy fix for that is for stop to save the data somewhere so get_traces (or whatever) can still retrieve it. If `stop` really must destroy the data, perhaps it should be called `close` instead; StringIO has a similar close method that when called destroys any stored data, and get_value must be called first if that data is wanted. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Compiler security
Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler security
Am 31.10.2013 15:48, schrieb MRAB: Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security http://bugs.python.org/issue17405 addresses one issue with dead code elimination. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler security
Interesting read. I'm surprised that the researchers didn't contact us, since the article mentions they found 5 bugs in Python. Regarding security: the article seems to use that term mostly to attract eyeballs; there are no specifics, just the implication that this *could* affect security. But it's hardly news -- as GCC versions became more aggressive we've had to fix our share of undefined code in Python. Usually the unittests catch these early. On Thu, Oct 31, 2013 at 7:48 AM, MRAB pyt...@mrabarnett.plus.com wrote: Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/**security/380406/how-your-** compiler-may-be-compromising-**application-securityhttp://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security __**_ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/**mailman/listinfo/python-devhttps://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** guido%40python.orghttps://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler security
I believe the 5 problems they found in Python were dealt with here http://bugs.python.org/issue17016 2013/10/31 MRAB pyt...@mrabarnett.plus.com: Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler security
Am 31.10.2013 15:48, schrieb MRAB: Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security I didnt' see this at first: STACK was run against a number of systems written in C/C++ and it found 160 new bugs in the systems tested, including ... and Python (5). Has anybody contact us? I neither saw a bug report nor a mail to PSRT. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On Thu, Oct 31, 2013 at 5:52 AM, Nick Coghlan ncogh...@gmail.com wrote: On 31 Oct 2013 18:52, Eric Snow ericsnowcurren...@gmail.com wrote: On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan ncogh...@gmail.com wrote: There's also the option of implementing the constraint directly in the finder, which *does* have the necessary info (with the change to pass the previous spec to find_spec). Yeah, I thought of that. I just prefer the more explicit supports_reload(). That said... I still think it makes more sense to leave this out for the moment - it's not at all clear we need the extra method, and adding it later would be a straightforward protocol update. ...I agree that makes the most sense for now. :) BTW, thanks for pushing these issues. I think the API has gotten pretty solid. I just need to make sure the PEP covers the cases and conclusions we're discussing. Thanks are also due to PJE for making me realise we were handwaving too much when it came to the expected reload semantics :) You're welcome. ;-) But speaking of handwaving, I also want to be sure that loader developers know that reloading is only really reloading if there's a previous existing spec, or the module type is... Hm. Actually, I think I now know how to state what's bugging me every time I see this supports_reload() or reload=True or other reloading flags in this process. I think that references to reloading should be replaced with references to what's *actually* at issue, because reloading itself is vague and carries too many assumptions for a loader author to understand or get right. (Look how hard it is for *us*!) That is, I think we should clarify what use cases there are for knowing whether a reload is happening, and address those use cases explicitly rather than lumping them under a general heading. For example, if the reason a loader cares about reloading is because it's a C extension using a custom module type, and the existing module isn't of the right type, then we should just spell out how to handle it. (e.g. raise an exception) If the reason a loader cares about reloading is because of some sort of caching or reuse, then we should just spell out how to handle that, too. Lumping these cases together under a reloading flag or a check for reloading support is a nasty code smell, because it requires a loader developer to have the *same* vaguely-defined idea of reloading as the PEP authors. ;-) I also suspect, that if properly spelled out, those use cases are going to boil down to: 1. Throwing errors if you have an existing module object you can't load into, and 2. Passing in a previous spec object, if available In other words, loaders should not really have any responsibility for or concept of reloading -- they always load into a module object (that they may or may not have created), and they may get given a spec from a previous load. They should deal only in module reuse and spec reuse. While a typical reload() might involve both reuses, there are cases where one sort of reuse could occur independently, and not all loaders care about both (or even either) condition. At any rate, it means a loader author doesn't have to figure out how to handle reloading, all they have to figure out is whether they can load into a particular module object, and whether they can do something useful with a spec that was previously used to load a module with the same name -- a spec that may or may not refer to a similar previous loader. These are rather more well-defined endeavors than trying to determine in the abstract whether one supports reload. ;-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler security
31.10.13 16:56, Benjamin Peterson написав(ла): I believe the 5 problems they found in Python were dealt with here http://bugs.python.org/issue17016 Ah, now I have remembered author's name. http://bugs.python.org/issue18684 contains some other fixes of this kind. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler security
On 10/31/2013 10:57 AM, Christian Heimes wrote: I didnt' see this at first: STACK was run against a number of systems written in C/C++ and it found 160 new bugs in the systems tested, including ... and Python (5). Has anybody contact us? I neither saw a bug report nor a mail to PSRT. from http://css.csail.mit.edu/stack/ Our software is hosted on Github at https://github.com/xiw/stack/. Obtain the latest code of STACK using: git clone git://github.com/xiw/stack See README and INSTALL for more information. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 451 update
On 1 Nov 2013 01:37, PJ Eby p...@telecommunity.com wrote: . ;-) I also suspect, that if properly spelled out, those use cases are going to boil down to: 1. Throwing errors if you have an existing module object you can't load into, and 2. Passing in a previous spec object, if available In other words, loaders should not really have any responsibility for or concept of reloading -- they always load into a module object (that they may or may not have created), and they may get given a spec from a previous load. They should deal only in module reuse and spec reuse. While a typical reload() might involve both reuses, there are cases where one sort of reuse could occur independently, and not all loaders care about both (or even either) condition. At any rate, it means a loader author doesn't have to figure out how to handle reloading, all they have to figure out is whether they can load into a particular module object, and whether they can do something useful with a spec that was previously used to load a module with the same name -- a spec that may or may not refer to a similar previous loader. These are rather more well-defined endeavors than trying to determine in the abstract whether one supports reload. ;-) It may not have been clear from the email exchange, but that's basically where we ended up :) The change Eric is currently going to make to the PEP is to add an optional previous parameter to the various find_spec APIs. At the time when find_spec runs, sys.modules hasn't been touched yet, so the old module state is still available when reloading. Passing the spec in lets the loader decide whether or not it can actually load that module given the information about the previous load operation. However, perhaps it makes more sense to pass the entire existing module, rather than just the spec? I'd like to minimise the need for new-style loader authors to retrieve things directly from the sys module, and you're right that can I reload into this particular module? is a slightly different question from can I reload a module previously loaded using this particular spec?. A spec-based API could still be used to answer the first question by looking up sys.modules, but so could the name based API. Passing in the module reference, on the other hand, should let loaders answer both questions without needing to touch the sys module. I also now think this pass-the-module approach when finding the spec approach could also clean up some of the messiness that is __main__ initialisation, where we repeatedly load things into the same module object. Let's say we be completely explicit and call the new parameter something like load_target. If we do that then I would make the *same* change to runpy.run_path and runpy.run_module: let you pass in an existing module object under that name to say hey, run in this namespace, rather than a fresh one. (Those APIs currently only support pre-populating a *fresh* namespace, rather than allowing you to directly specify an existing one) Most loaders won't care, but those that do will have all the info they need to throw an exception saying I could provide a spec for this, but it's not compatible with that load target. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
