[issue45546] Unable to pickle enum with nested frozen dataclass?
Eric Cousineau added the comment: > I get an error with 3.8.10, but not on the main branch (3.11) [...] Confirmed, using 3.11.0a1 (using Docker). Seems like at least the pickling is fixed? > I'm confused -- your initial report talks about pickling fields, but your > tests are pickling the entire class... what am I missing? Sorry about that; the original file, `enum_test.py`, was just trying to pickle an enum field, i.e. `DoesNotWork.a`. However, because it uses the type `DoesNotWork.NestedValue2`, `pickle` tries to unpackage that type, but cannot access it via name because `DoesNotWork.NestedValue2` is wrapped as an enum *field*, rather than just a normal field. I've uploaded a new file, `enum_nested_type_test.py`, that perhaps shows the root cause - the `Enum` interprets the nested class as an enum field (wrapping it as an enum value), rather than as "just" a nested class. In contrast, it did not seem to wrap the function. Does that make sense / help at all? -- Added file: https://bugs.python.org/file50384/enum_nested_type_test.py ___ Python tracker <https://bugs.python.org/issue45546> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45546] Unable to pickle enum with nested frozen dataclass?
Eric Cousineau added the comment: Thinking on it some more, I the current `Enum` metaclass doesn't distinguish between nested class and normal field, so it wraps it, thus why `pickle` gets confused. Perhaps it's possible to recognize this case - the class w/ `__qualname__` can be checked to see if it's nested. If so, then options are (a) skip wrapping the field, (b) failing fast, or (c) "no fix"? -- ___ Python tracker <https://bugs.python.org/issue45546> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45546] Unable to pickle enum with nested frozen dataclass?
Eric Cousineau added the comment: Ah, forgot to include error message: ``` E ... _pickle.PicklingError: Can't pickle : it's not the same object as __main__.DoesNotWork.NestedValue2 ``` -- ___ Python tracker <https://bugs.python.org/issue45546> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45546] Unable to pickle enum with nested frozen dataclass?
New submission from Eric Cousineau : I seem cannot pickle enum values from an enum class w/ a nested frozen dataclass. I can pickle a field from a normal class w/ a nested frozen dataclass, and I can pickle a field from an enum with a top-level frozen dataclass - perhaps it's some interplay with `__qualname__` and `enum.py`?. See attached file for sample test. I checked on 3.6.9, 3.9.7, and 3.10.0. Looked around, couldn't easily find existing issue for this. -- components: Library (Lib) files: enum_test.py messages: 404539 nosy: Eric Cousineau priority: normal severity: normal status: open title: Unable to pickle enum with nested frozen dataclass? type: behavior versions: Python 3.10, Python 3.6, Python 3.9 Added file: https://bugs.python.org/file50378/enum_test.py ___ Python tracker <https://bugs.python.org/issue45546> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40240] Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?
Eric Cousineau added the comment:
Aye. Using a workaround for now ("leak" the object by incrementing the refcount
on first resurrection):
https://github.com/RobotLocomotion/pybind11/pull/39
I may try Antoine's suggestion later on, but will definitely reformulate this
to use the public API version of `gc.is_finalized()` when it comes about.
Thanks!
--
___
Python tracker
<https://bugs.python.org/issue40240>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40240] Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?
Eric Cousineau added the comment: Er, to clarify: "PEP 442 had that effect" => "PEP 442 had the effect of making it resurrectable only once, but not more than that." -- ___ Python tracker <https://bugs.python.org/issue40240> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40240] Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?
Eric Cousineau added the comment: > [...] but I am not sure it makes sense to allow to mark an object as > finalized because that could mess with the GC algorithm. Actually, I would like the opposite, to mark it unfinalized to resurrect the object more than once. PEP 442 from a ways back had this effect (per bpo-32377), and Antoine updated the docs after I filed the issue. I didn't chime in early enough to snag the "more-than-once" functionality, so I guess this is what I get for dipping into private API without asking to make it public until 2 years later... d'oh! -- ___ Python tracker <https://bugs.python.org/issue40240> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40240] Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?
Eric Cousineau added the comment: C functions sound great! I am certainly not wed to the macros (nor do I love them), as I do not have intense performance requirements where inlining (and spilling implementation guts) is necessary. -- ___ Python tracker <https://bugs.python.org/issue40240> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32377] Difference in ressurrection behavior with __del__ in py2 vs. py3
Eric Cousineau added the comment: See also bpo-40240: "Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?" -- ___ Python tracker <https://bugs.python.org/issue32377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40240] Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?
New submission from Eric Cousineau : Motivated by this downstream project issue that I am working on: https://github.com/RobotLocomotion/drake/issues/13026 In https://bugs.python.org/issue32377, I encountered PEP 442's updated resurrection behavior when moving from supporting Python 2 to Python 3. There, Antoine Pitrou (pitrou) said that using this API (finalized + set finalized) could work, but that I could also try recreating the wrapper object. I have not yet attempted his suggestion given that (a) wrapping code is nuanced (pybind11, inheritance, etc.) and (b) this API has been working for us for the past 2 years. Related to this, I saw some mentions of breakage of Cython due to its usage of this API: https://bugs.python.org/issue35081#msg330045 The breakage was mitigated by keeping this internal API exposed (so kinda public, but not really?). Is it at all possible to considering making some of this public API? -- components: C API messages: 366059 nosy: Eric Cousineau priority: normal severity: normal status: open title: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED? type: enhancement versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue40240> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32377] Difference in ressurrection behavior with __del__ in py2 vs. py3
Eric Cousineau added the comment: Super late response, but for this part: > Well... perhaps you could create another PyObject (it's just a wrapper, > right?) since the old one doesn't have any outside references to it > remaining. In certain cases, yes, that would be the case. I do have unittests that (hackily) check `id(o)` before and after resurrection by using a `weakref`, so I could relax that contract. However, in other cases, the "wrapper" object in Python is actually a C++ base class that has been derived from in Python. If ownership of the C++ instance (which should more-or-less own the Python portion) goes solely to C++, then the Python portion could be garbage collected, and you get a sort-of slicing effect: https://github.com/pybind/pybind11/issues/1145 Granted, within this web of issues, one alternative is to always extend the lifetime of the Python object by the C++ portion. However, that creates a (opaque?) reference cycle between C++ and CPython, so I am a bit afraid of what that would do. I would prefer to make memory management conservative if possible. Or rather: It's something cool to try out, but I have not yet had time to dig in :( -- ___ Python tracker <https://bugs.python.org/issue32377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32845] Mac: Local modules can shadow builtins (e.g. a local `math.py` can shadow `math`)
Eric Cousineau <eric.cousineau@tri.global> added the comment:
> P.S. This issue points out once again why it is generally a bad idea to
> shadow or mix-and-match standard library module names.
Duly noted! And thank y'all for the explanations!
Can I ask if it's bad practice to use a standard library module name as a
submodule, e.g. `example_module.math`?
TBH, this all arises because we use Bazel and wanted to modularize our tests -
we place them under `test/{name}.py` neighboring the target (sub)module, and
Bazel presently generates a wrapper script which is executed in the same
directory as the (sub)module, which is why this (unintended) shadowing occurs.
I would like still like to have a submodule named `math`, so I can just teach
our Bazel rules to not run the test in the same directory -- if having
submodules named `math` is not frowned upon.
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue32845>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32845] Mac: Local modules can shadow builtins (e.g. a local `math.py` can shadow `math`)
New submission from Eric Cousineau <eric.cousineau@tri.global>: We ran into an issue with our library; we build a submodule named `math.so`, and have a test `math_test` which is ran in the same directory as this module (since this is how we've written it with Bazel's generated Python binaries): https://github.com/RobotLocomotion/drake/issues/8041 This happens for us on HighSierra / Sierra MacOS (for *.py and *.so modules); Ubuntu Linux seems fine. I have a reproduction script just for *.py files here: https://github.com/EricCousineau-TRI/repro/blob/b08bc47/bug/mac_builtin_shadow/test.sh#L38 (attached this as `test.sh`) The module is exposed on the path given that it neighbors the first execution of the `import_test.py` script. The second execution (where the `import_test.py` script is moved up one level) succeeds on Mac (s.t. the `math.py` is not in `sys.path`). This behavior seems to violate the documentation: https://docs.python.org/2/tutorial/modules.html#the-module-search-path -- components: macOS files: test.sh messages: 312183 nosy: Eric Cousineau, ned.deily, ronaldoussoren priority: normal severity: normal status: open title: Mac: Local modules can shadow builtins (e.g. a local `math.py` can shadow `math`) type: behavior versions: Python 2.7 Added file: https://bugs.python.org/file47442/test.sh ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32845> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32377] Difference in ressurrection behavior with __del__ in py2 vs. py3
Eric Cousineau <eric.cousineau@tri.global> added the comment: You're welcome, and thank you for the prompt response! I will say that it feels a tad odd to only have `tp_finalize` be called once for the entire lifetime of the object, while still having the option of it being resurrected. Is there any way to somehow "un-mark" the object to enable this workflow that I would like to have? My current hack is to call `_PyGC_SET_FINALIZED(self, 0)` - may I ask if there is a simpler way to do this? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32377] Difference in ressurrection behavior with __del__ in py2 vs. py3
New submission from Eric Cousineau <eric.cousineau@tri.global>: Due to how `PyObject_CallFinalizer` is written in python3, `__del__` will only *ever* be called once. In my use case, I am experimenting with a feature in `pybind11` to prevent slicing with Python class instances that inherit from pybind11-C++ base classes, which involves detecting when an instance loses all reference in Python (`Py_REFCNT(...) == 0`) but still has reference in C++ (`shared_ptr::count() > 0`), and reviving the Python portion when this situation happens. In python2, I could do this without a hitch, as a resurrected object could have its `__del__` method called multiple times (through `tp_dealloc` I believe?). But in python3, the object is marked with `_PyGC_SET_FINALIZED(...)`, thus preventing `__del__` from being called again. It'd be nice to either (a) somehow allow `__del__` to be called naturally without too much fuss or, at the least, (b) have this reflected in the documentation: https://docs.python.org/3/reference/datamodel.html#object.__del__ See attached `revive_test`. Example execution: ``` $ python2 ./revive_test.py Revive Destroy [ Done ] $ python3 ./revive_test.py Revive [ Done ] ``` -- assignee: docs@python components: Documentation files: revive_test.py messages: 308660 nosy: Eric Cousineau, docs@python priority: normal severity: normal status: open title: Difference in ressurrection behavior with __del__ in py2 vs. py3 type: behavior versions: Python 3.4, Python 3.5, Python 3.6 Added file: https://bugs.python.org/file47339/revive_test.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15768] re.sub() with re.MULTILINE not replacing all occurrences
New submission from Eric Cousineau: [Copying post I made here: http://stackoverflow.com/questions/12082886/bug-in-python-regex-re-sub-with-re-multiline] I'm noticing some odd behavior in Python's Regex library, and I'm not sure if I'm doing something wrong. If I run a regex on it using re.sub(), with re.MULTILINE. It seems to only replace the first few occurrences. It replaces all occurrences if I turn off re.MULTILINE, use re.subn(..., count = 0, flags = re.MULTILINE), or compile the regex using re.compile(..., re.MULTILINE). I am running Python 2.7 on Ubuntu 12.04. I've posted a random example on: http://pastebin.com/49SU8Mm2 - Output from terminal http://codepad.org/2RO2iS4O - Script, confirming behavior (except for re.subn(), which is different on 2.5) -- components: Regular Expressions messages: 168909 nosy: eacousineau, ezio.melotti, mrabarnett priority: normal severity: normal status: open title: re.sub() with re.MULTILINE not replacing all occurrences type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15768 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15768] re.sub() with re.MULTILINE not replacing all occurrences
Eric Cousineau added the comment: Looks like I was just writing my code incorrectly - I should have been using re.sub(..., flags = re.MULTILINE). That explains it. -- resolution: - invalid status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15768 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
