Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016, at 14:04, MRAB wrote: > Instead of locking the object, could we keep the GIL, but have it > normally released? > A thread could then still call a function such as PyWeakref_GetObject() > that returns a borrowed reference, but only if it's holding the GIL. It > would be

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

By the way, just to finish playing this new fun game "Who'd I Borrow This Reference From?": On 10/10/2016 11:45 AM, Chris Angelico wrote: On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings wrote: I bet for every other spot in the API I can tell you from whom you're

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

> Huh? In all other circumstances, a "borrowed" reference is exactly that: X has a reference, and you are relying on X's reference to keep the object alive. Borrowing from a borrowed reference is simply a chain of these; Z borrows from Y, Y borrows from X, and X is the original person who did

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 11 October 2016 at 08:49, Guido van Rossum wrote: > On Mon, Oct 10, 2016 at 1:57 PM, Larry Hastings wrote: >> For now I'm going to leave the names as-is and just change the semantics >> everywhere. If this approach really works, and if we decide we want

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 3:27 PM, Greg Ewing wrote: > Nathaniel Smith wrote: >> >> IIRC to handle >> this gilectomy adds per-object mutexes that you have to hold whenever >> you're mucking around with that object's internals. > > > What counts as "mucking around with

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 9:52 AM, MRAB wrote: >> Also, I don't know when it would ever be safe to release the "memory >> deallocation lock". Just because it's safe for your thread doesn't mean >> it's safe for another thread. And if you do it on a thread-by-thread >>

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 22:14, Larry Hastings wrote: On 10/10/2016 09:36 PM, Chris Angelico wrote: Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory deallocation lock". While it is held (by any thread - it's a guard, more than a lock), Py_DECREF will not actually deallocate memory -

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 1:57 PM, Larry Hastings wrote: > > On 10/10/2016 06:37 PM, Guido van Rossum wrote: > > Modified +1: you can't change the behavior of the existing API, but > you can deprecate it and introduce a better one with a different name. > We'll have until Python

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Random832 wrote: Or is this a special kind of lock that you can "assert isn't locked" without locking it for yourself, and INCREF/DECREF does so? I don't think that would work. It might be unlocked at the moment you test it, but someone might lock it between then and the following

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Nathaniel Smith wrote: IIRC to handle this gilectomy adds per-object mutexes that you have to hold whenever you're mucking around with that object's internals. What counts as "mucking around with the object's internals", though? If I do the C equivalent of: x = somedict[5]

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 4:03 AM, Paul Moore wrote: If you have an object that forever owns a reference to another object, it's safe to return borrowed references. Even if there are such cases, it seems we're going to have to be a lot more careful about identifying them.

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings wrote: > But I don't think this fixes the problem. Consider: > > Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed reference. > Thread A then gets suspended, before it has a chance to Py_INCREF(Q). > Thread B

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10/10/2016 09:36 PM, Chris Angelico wrote: Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory deallocation lock". While it is held (by any thread - it's a guard, more than a lock), Py_DECREF will not actually deallocate memory - objects can fall to zero references without

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10/10/2016 06:37 PM, Guido van Rossum wrote: Modified +1: you can't change the behavior of the existing API, but you can deprecate it and introduce a better one with a different name. We'll have until Python 4.0 to carry through the deprecation anyways. And I doubt this is the only C API

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 20:36, Chris Angelico wrote: On Tue, Oct 11, 2016 at 5:24 AM, Random832 wrote: On Mon, Oct 10, 2016, at 14:04, MRAB wrote: Instead of locking the object, could we keep the GIL, but have it normally released? A thread could then still call a function such

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 5:24 AM, Random832 wrote: > On Mon, Oct 10, 2016, at 14:04, MRAB wrote: >> Instead of locking the object, could we keep the GIL, but have it >> normally released? >> >> A thread could then still call a function such as PyWeakref_GetObject() >> that

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016, at 14:04, MRAB wrote: > Instead of locking the object, could we keep the GIL, but have it > normally released? > > A thread could then still call a function such as PyWeakref_GetObject() > that returns a borrowed reference, but only if it's holding the GIL. It > would be

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 18:50, Nathaniel Smith wrote: On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore wrote: > On 10 October 2016 at 17:49, MRAB wrote: >> If you lookup something in a dict, it'll be a borrowed reference. >> >> If the dict is globals() and

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10 October 2016 at 18:50, Nathaniel Smith wrote: > I assume Larry is way ahead of us on this, Yeah, I'd imagine you're right on that :-) Paul ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore wrote: > On 10 October 2016 at 17:49, MRAB wrote: >> If you lookup something in a dict, it'll be a borrowed reference. >> >> If the dict is globals() and there's no GIL, another thread could delete the

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 4:03 AM, Paul Moore wrote: > On 10 October 2016 at 17:49, MRAB wrote: >> If you lookup something in a dict, it'll be a borrowed reference. >> >> If the dict is globals() and there's no GIL, another thread could delete the

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 3:49 AM, MRAB wrote: > On 2016-10-10 10:45, Chris Angelico wrote: >> >> On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings >> wrote: >>> >>> Huh? In all other circumstances, a "borrowed" reference is exactly that: >>> X >>>

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10 October 2016 at 17:49, MRAB wrote: > If you lookup something in a dict, it'll be a borrowed reference. > > If the dict is globals() and there's no GIL, another thread could delete the > item while your code had the borrowed reference. > > It looks like there

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 10:45, Chris Angelico wrote: On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings wrote: Huh? In all other circumstances, a "borrowed" reference is exactly that: X has a reference, and you are relying on X's reference to keep the object alive. Borrowing from a

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Modified +1: you can't change the behavior of the existing API, but you can deprecate it and introduce a better one with a different name. We'll have until Python 4.0 to carry through the deprecation anyways. And I doubt this is the only C API change needed for happy gil-free coding... On Mon,

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 4:17 AM, Larry Hastings wrote: > Given that the weakref doesn't have a reference to the object--merely a weak > reference, different thing--whose reference is it borrowing? As others have said, it doesn't really matter who's reference it was; just that

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

> On 2016-10-10, at 11:05 , Devin Jeanpierre wrote: > The term "borrowed" is supposed to imply a sensible scope during which you're > free to use the object, and weakrefs don't have that (except for what is > granted by the GIL), so this does sound wacky. I bet it was

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Larry Hastings wrote: In contrast, the "borrowed" reference returned by PyWeakRef_GetObject() seems to be "borrowed" from some unspecified entity. It's a delocalised quantum reference, borrowing a little bit from all strong references in existence. :-) -- Greg

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings wrote: > Huh? In all other circumstances, a "borrowed" reference is exactly that: X > has a reference, and you are relying on X's reference to keep the object > alive. Borrowing from a borrowed reference is simply a chain of

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10/10/2016 11:05 AM, Devin Jeanpierre wrote: > whose reference is it borrowing? I think this is a red herring sort of question. "borrowed" only means "unowned". But anyway, we borrowed from the weakref. (Borrowing from somebody doesn't imply they own a reference -- we can borrow from a

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

> It's my contention that this API is simply untenable under the Gilectomy Yeah, I agree with your contention. The only circumstance PyWeakref_GetObject would still be valid without a GIL is if you happened to know you had a reference elsewhere that was keeping it alive. But if you did, you