Re: Puzzling behaviour of Py_IncRef

[Barry Scott]

> On 27 Jan 2022, at 07:46, Tony Flury  wrote:
> 
> 
> On 26/01/2022 22:41, Barry wrote:
>> 
>> 
>> Run python and your code under a debugger and check the ref count of the 
>> object as you step through the code.
>> 
>> Don’t just step through your code but also step through the C python code.
>> That will allow you to see how this works at a low level.
>> Setting a watch point on the ref count will allow you run the code and just 
>> break as the ref count changes.
>> 
>> That is what I do when a see odd c api behaviour.
>> 
>> Barry
> 
> 
> Thanks - I have tried a few times on a few projects to run a debugger in 
> mixed language mode and never had any success.
> 
> I will have to try again.

You mean debugging Python and C/C++? In this case your python code is a simple 
test script and it's C that you care about.
Should not be difficult. I tend to use linux as my lead debug platform as its 
the easiest to work with. But Windows and macOS
also have very good debuggers.

Barry

> 
> 
>>> As posted in the original message - immediately before the call to the C 
>>> function/method sys.getrefcount reports the count to be 2 (meaning it is 
>>> actually a 1).
>>> 
>>> Inside the C function the ref count is incremented and the Py_REFCNT macro 
>>> reports the count as 3 inside the C function as expected (1 for the name in 
>>> the Python code, 1 for the argument as passed to the C function, and 1 for 
>>> the increment), so outside the function one would expect the ref count to 
>>> now be 2 (since the reference caused by calling the function is then 
>>> reversed).
>>> 
>>> However - Immediately outside the C function and back in the Python code 
>>> sys.getrefcount reports the count to be 2 again - meaning it is now really 
>>> 1. So that means that the refcount has been decremented twice in-between 
>>> the return of the C function and the execution of the immediate next python 
>>> statement. I understand one of those decrements - the parameter's ref count 
>>> is incremented on the way in so the same object is decremented on the way 
>>> out (so that calls don't leak references) but I don't understand where the 
>>> second decrement is coming from.
>>> 
>>> Again there is nothing in the Python code that would cause that decrement - 
>>> the decrement behavior is in the Python runtime.
>>> 
> -- 
> Anthony Flury
> email :anthony.fl...@btinternet.com
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 
>>> -- 
>>> Anthony Flury
>>> email :anthony.fl...@btinternet.com
> 
> -- 
> Anthony Flury
> email : anthony.fl...@btinternet.com
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

On 26/01/2022 22:41, Barry wrote:



Run python and your code under a debugger and check the ref count of 
the object as you step through the code.


Don’t just step through your code but also step through the C python code.
That will allow you to see how this works at a low level.
Setting a watch point on the ref count will allow you run the code and 
just break as the ref count changes.


That is what I do when a see odd c api behaviour.

Barry



Thanks - I have tried a few times on a few projects to run a debugger in 
mixed language mode and never had any success.


I will have to try again.


As posted in the original message - immediately before the call to 
the C function/method sys.getrefcount reports the count to be 2 
(meaning it is actually a 1).


Inside the C function the ref count is incremented and the Py_REFCNT 
macro reports the count as 3 inside the C function as expected (1 for 
the name in the Python code, 1 for the argument as passed to the C 
function, and 1 for the increment), so outside the function one would 
expect the ref count to now be 2 (since the reference caused by 
calling the function is then reversed).


However - Immediately outside the C function and back in the Python 
code sys.getrefcount reports the count to be 2 again - meaning it is 
now really 1. So that means that the refcount has been decremented 
twice in-between the return of the C function and the execution of 
the immediate next python statement. I understand one of those 
decrements - the parameter's ref count is incremented on the way in 
so the same object is decremented on the way out (so that calls don't 
leak references) but I don't understand where the second decrement is 
coming from.


Again there is nothing in the Python code that would cause that 
decrement - the decrement behavior is in the Python runtime.



--
Anthony Flury
email :anthony.fl...@btinternet.com

--
https://mail.python.org/mailman/listinfo/python-list


--
Anthony Flury
email :anthony.fl...@btinternet.com


--
Anthony Flury
email : anthony.fl...@btinternet.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Barry]

> On 25 Jan 2022, at 23:50, Tony Flury  wrote:
> 
> 
> 
> 
>> On 25/01/2022 22:28, Barry wrote:
>> 
 On 25 Jan 2022, at 14:50, Tony Flury via Python-list 
  wrote:
 
 
> On 20/01/2022 23:12, Chris Angelico wrote:
>>> On Fri, 21 Jan 2022 at 10:10, Greg Ewing  
>>> wrote:
>>> On 20/01/22 12:09 am, Chris Angelico wrote:
>>> At this point, the refcount has indeed been increased.
>>> 
   return self;
  }
>>> And then you say "my return value is this object".
>>> 
>>> So you're incrementing the refcount, then returning it without
>>> incrementing the refcount. Your code is actually equivalent to "return
>>> self".
>> Chris, you're not making any sense. This is C code, so there's no
>> way that "return x" can change the reference count of x.
> Yeah, I wasn't clear there. It was equivalent to *the Python code*
> "return self". My apologies.
> 
>  > The normal thing to do is to add a reference to whatever you're
>  > returning. For instance, Py_RETURN_NONE will incref None and then
>  > return it.
>  >
> 
> The OP understands that this is not a normal thing to do. He's
> trying to deliberately leak a reference for the purpose of diagnosing
> a problem.
> 
> It would be interesting to see what the actual refcount is after
> calling this function.
>>> After calling this without a double increment in the function the ref count 
>>> is still only 1 - which means that the 'return self' effectively does a 
>>> double decrement. My original message includes the Python code which calls 
>>> this 'leaky' function and you can see that despite the 'leaky POC' doing an 
>>> increment ref count drops back to one after the return.
>>> 
>>> You are right this is not a normal thing to do, I am trying to understand 
>>> the behaviour so my library does the correct thing in all cases - for 
>>> example - imagine you have two nodes in a tree :
>>> 
>>> A --- > B
>>> 
>>> And your Python code has a named reference to A, and B also maintains a 
>>> reference to A as it's parent.
>>> 
>>> In this case I would expect A to have a reference count of 2 (counted as 3 
>>> through sys.getrefcount() - one for the named reference in the Python code 
>>> - and one for the link from B back to A; I would also expect B to have a 
>>> reference count here of 1 (just the reference from A - assuming nothing 
>>> else referenced B).
>>> 
>>> My original code was incrementing the ref counts of A and B and then 
>>> returning A. within the Python test code A had a refcount of 1 (and not the 
>>> expected 2), but the refcount from B was correct as far as I could tell.
>>> 
>>> 
 Yes, and that's why I was saying it would need a *second* incref.
 
 ChrisA
>>> Thank you to all of you for trying to help - I accept that the only way to 
>>> make the code work is to do a 2nd increment.
>>> 
>>> I don't understand why doing a 'return self' would result in a double 
>>> decrement - that seems utterly bizzare behaviour - it obviously works, but 
>>> why.
>> The return self in C will not change the ref count.
>> 
>> I would suggest setting a break point in your code and stepping out of the 
>> function and seeing that python’s code does to the ref count.
>> 
>> Barry
> Barry,
> 
> something odd is going on because the Python code isn't doing anything that 
> would cause the reference count to go from 3 inside the C function to 1 once 
> the method call is complete.
> 
> As far as I know the only things that impact the reference counts are : 
> 
> Increments due to assigning a new name or adding it to a container.
> Increment due to passing the object to a function (since that binds a new 
> name) 
> Decrements due to deletion of a name
> Decrement due to going out of scope
> Decrement due to being removed from a container.
> None of those things are happening in the python code.
> 

Run python and your code under a debugger and check the ref count of the object 
as you step through the code.

Don’t just step through your code but also step through the C python code.
That will allow you to see how this works at a low level. 
Setting a watch point on the ref count will allow you run the code and just 
break as the ref count changes.

That is what I do when a see odd c api behaviour.

Barry
> As posted in the original message - immediately before the call to the C 
> function/method sys.getrefcount reports the count to be 2 (meaning it is 
> actually a 1).
> 
> Inside the C function the ref count is incremented and the Py_REFCNT macro 
> reports the count as 3 inside the C function as expected (1 for the name in 
> the Python code, 1 for the argument as passed to the C function, and 1 for 
> the increment), so outside the function one would expect the ref count to now 
> be 2 (since the reference caused by calling the function is then reversed).
> 
> However - Immediately outside the C function and 

Re: Puzzling behaviour of Py_IncRef

[Greg Ewing]

The convention for refcounting in CPython is that a function
takes borrowed references as arguments and returns a new
reference.

The 'self' argument passed in is a borrowed reference. If you
want to return it, you need to create a new reference by
increfing it.

So what you have written is just the correct way to write
a do-nothing function that returns its argument, i.e. the
equivalent of the Python function

   def f(self):
  return self

As others have said, if you want to leak a reference, you
need an extra incref besides the one that's a normal part
of the return convention.

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

On 26/01/2022 08:20, Chris Angelico wrote:

On Wed, 26 Jan 2022 at 19:04, Tony Flury via Python-list
 wrote:

So according to that I should increment twice if and only if the calling
code is using the result - which you can't tell in the C code - which is
very odd behaviour.

No, the return value from your C function will *always* have a
reference taken. Whether the return value is "used" or just dropped,
there's always going to be one ref used by the returning itself.

The standard way to return a value is always to incref it, then return
the pointer. That is exactly equivalent to Python saying "return
".

Incrementing twice is ONLY because you want to leak a reference.

ChrisA


Chris,

You keep saying I am leaking a reference - my original code (not the POC 
in the email) wasn't intending to leak a reference, it was incrementing 
the reference count in order to accurately count references, from other 
objects and i needed to double increment there so that the reference 
count remained correct outside of the C code.


I did try to be clear - my intention was never to leak a reference (I 
have been writing s/w long enough to know leaks are bad) - my POC code 
in the original message was the only code which deliberately leaked a 
reference in order to simply illustrate the problem.


I do appreciate the help you have tried to give - so thank you.

--
Anthony Flury
email : anthony.fl...@btinternet.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Chris Angelico]

On Wed, 26 Jan 2022 at 19:04, Tony Flury via Python-list
 wrote:
>
> So according to that I should increment twice if and only if the calling
> code is using the result - which you can't tell in the C code - which is
> very odd behaviour.

No, the return value from your C function will *always* have a
reference taken. Whether the return value is "used" or just dropped,
there's always going to be one ref used by the returning itself.

The standard way to return a value is always to incref it, then return
the pointer. That is exactly equivalent to Python saying "return
".

Incrementing twice is ONLY because you want to leak a reference.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

On 26/01/2022 01:29, MRAB wrote:

On 2022-01-25 23:50, Tony Flury via Python-list wrote:


On 25/01/2022 22:28, Barry wrote:


On 25 Jan 2022, at 14:50, Tony Flury via 
Python-list  wrote:




On 20/01/2022 23:12, Chris Angelico wrote:
On Fri, 21 Jan 2022 at 10:10, Greg 
Ewing  wrote:

On 20/01/22 12:09 am, Chris Angelico wrote:

At this point, the refcount has indeed been increased.


   return self;
  }

And then you say "my return value is this object".

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to 
"return

self".

Chris, you're not making any sense. This is C code, so there's no
way that "return x" can change the reference count of x.

Yeah, I wasn't clear there. It was equivalent to *the Python code*
"return self". My apologies.


  > The normal thing to do is to add a reference to whatever you're
  > returning. For instance, Py_RETURN_NONE will incref None and 
then

  > return it.
  >

The OP understands that this is not a normal thing to do. He's
trying to deliberately leak a reference for the purpose of 
diagnosing

a problem.

It would be interesting to see what the actual refcount is after
calling this function.
After calling this without a double increment in the function the 
ref count is still only 1 - which means that the 'return self' 
effectively does a double decrement. My original message includes 
the Python code which calls this 'leaky' function and you can see 
that despite the 'leaky POC' doing an increment ref count drops 
back to one after the return.


You are right this is not a normal thing to do, I am trying to 
understand the behaviour so my library does the correct thing in 
all cases - for example - imagine you have two nodes in a tree :


A --- > B

And your Python code has a named reference to A, and B also 
maintains a reference to A as it's parent.


In this case I would expect A to have a reference count of 2 
(counted as 3 through sys.getrefcount() - one for the named 
reference in the Python code - and one for the link from B back to 
A; I would also expect B to have a reference count here of 1 (just 
the reference from A - assuming nothing else referenced B).


My original code was incrementing the ref counts of A and B and 
then returning A. within the Python test code A had a refcount of 1 
(and not the expected 2), but the refcount from B was correct as 
far as I could tell.




Yes, and that's why I was saying it would need a *second* incref.

ChrisA
Thank you to all of you for trying to help - I accept that the only 
way to make the code work is to do a 2nd increment.


I don't understand why doing a 'return self' would result in a 
double decrement - that seems utterly bizzare behaviour - it 
obviously works, but why.

The return self in C will not change the ref count.

I would suggest setting a break point in your code and stepping out 
of the function and seeing that python’s code does to the ref count.


Barry


Barry,

something odd is going on because the Python code isn't doing anything
that would cause the reference count to go from 3 inside the C function
to 1 once the method call is complete.

As far as I know the only things that impact the reference counts are :

   * Increments due to assigning a new name or adding it to a container.
   * Increment due to passing the object to a function (since that binds
 a new name)
   * Decrements due to deletion of a name
   * Decrement due to going out of scope
   * Decrement due to being removed from a container.

None of those things are happening in the python code.

As posted in the original message - immediately before the call to the C
function/method sys.getrefcount reports the count to be 2 (meaning it is
actually a 1).

Inside the C function the ref count is incremented and the Py_REFCNT
macro reports the count as 3 inside the C function as expected (1 for
the name in the Python code, 1 for the argument as passed to the C
function, and 1 for the increment), so outside the function one would
expect the ref count to now be 2 (since the reference caused by calling
the function is then reversed).

However - Immediately outside the C function and back in the Python code
sys.getrefcount reports the count to be 2 again - meaning it is now
really 1. So that means that the refcount has been decremented twice
in-between the return of the C function and the execution of the
immediate next python statement. I understand one of those decrements -
the parameter's ref count is incremented on the way in so the same
object is decremented on the way out (so that calls don't leak
references) but I don't understand where the second decrement is coming
from.

Again there is nothing in the Python code that would cause that
decrement - the decrement behavior is in the Python runtime.


The function returns a result, an object.

The calling code is discarding the result, so it's being DECREFed.

For example:

    def foo():
 

Re: Puzzling behaviour of Py_IncRef

[MRAB]

On 2022-01-25 23:50, Tony Flury via Python-list wrote:


On 25/01/2022 22:28, Barry wrote:



On 25 Jan 2022, at 14:50, Tony Flury via Python-list  
wrote:



On 20/01/2022 23:12, Chris Angelico wrote:

On Fri, 21 Jan 2022 at 10:10, Greg Ewing  wrote:
On 20/01/22 12:09 am, Chris Angelico wrote:

At this point, the refcount has indeed been increased.


   return self;
  }

And then you say "my return value is this object".

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".

Chris, you're not making any sense. This is C code, so there's no
way that "return x" can change the reference count of x.

Yeah, I wasn't clear there. It was equivalent to *the Python code*
"return self". My apologies.


  > The normal thing to do is to add a reference to whatever you're
  > returning. For instance, Py_RETURN_NONE will incref None and then
  > return it.
  >

The OP understands that this is not a normal thing to do. He's
trying to deliberately leak a reference for the purpose of diagnosing
a problem.

It would be interesting to see what the actual refcount is after
calling this function.

After calling this without a double increment in the function the ref count is 
still only 1 - which means that the 'return self' effectively does a double 
decrement. My original message includes the Python code which calls this 
'leaky' function and you can see that despite the 'leaky POC' doing an 
increment ref count drops back to one after the return.

You are right this is not a normal thing to do, I am trying to understand the 
behaviour so my library does the correct thing in all cases - for example - 
imagine you have two nodes in a tree :

A --- > B

And your Python code has a named reference to A, and B also maintains a 
reference to A as it's parent.

In this case I would expect A to have a reference count of 2 (counted as 3 
through sys.getrefcount() - one for the named reference in the Python code - 
and one for the link from B back to A; I would also expect B to have a 
reference count here of 1 (just the reference from A - assuming nothing else 
referenced B).

My original code was incrementing the ref counts of A and B and then returning 
A. within the Python test code A had a refcount of 1 (and not the expected 2), 
but the refcount from B was correct as far as I could tell.



Yes, and that's why I was saying it would need a *second* incref.

ChrisA

Thank you to all of you for trying to help - I accept that the only way to make 
the code work is to do a 2nd increment.

I don't understand why doing a 'return self' would result in a double decrement 
- that seems utterly bizzare behaviour - it obviously works, but why.

The return self in C will not change the ref count.

I would suggest setting a break point in your code and stepping out of the 
function and seeing that python’s code does to the ref count.

Barry


Barry,

something odd is going on because the Python code isn't doing anything
that would cause the reference count to go from 3 inside the C function
to 1 once the method call is complete.

As far as I know the only things that impact the reference counts are :

   * Increments due to assigning a new name or adding it to a container.
   * Increment due to passing the object to a function (since that binds
 a new name)
   * Decrements due to deletion of a name
   * Decrement due to going out of scope
   * Decrement due to being removed from a container.

None of those things are happening in the python code.

As posted in the original message - immediately before the call to the C
function/method sys.getrefcount reports the count to be 2 (meaning it is
actually a 1).

Inside the C function the ref count is incremented and the Py_REFCNT
macro reports the count as 3 inside the C function as expected (1 for
the name in the Python code, 1 for the argument as passed to the C
function, and 1 for the increment), so outside the function one would
expect the ref count to now be 2 (since the reference caused by calling
the function is then reversed).

However - Immediately outside the C function and back in the Python code
sys.getrefcount reports the count to be 2 again - meaning it is now
really 1. So that means that the refcount has been decremented twice
in-between the return of the C function and the execution of the
immediate next python statement. I understand one of those decrements -
the parameter's ref count is incremented on the way in so the same
object is decremented on the way out (so that calls don't leak
references) but I don't understand where the second decrement is coming
from.

Again there is nothing in the Python code that would cause that
decrement - the decrement behavior is in the Python runtime.


The function returns a result, an object.

The calling code is discarding the result, so it's being DECREFed.

For example:

def foo():
return Node()

returns a new node, so its 

Re: Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

On 25/01/2022 22:28, Barry wrote:



On 25 Jan 2022, at 14:50, Tony Flury via Python-list  
wrote:



On 20/01/2022 23:12, Chris Angelico wrote:

On Fri, 21 Jan 2022 at 10:10, Greg Ewing  wrote:
On 20/01/22 12:09 am, Chris Angelico wrote:

At this point, the refcount has indeed been increased.


   return self;
  }

And then you say "my return value is this object".

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".

Chris, you're not making any sense. This is C code, so there's no
way that "return x" can change the reference count of x.

Yeah, I wasn't clear there. It was equivalent to *the Python code*
"return self". My apologies.


  > The normal thing to do is to add a reference to whatever you're
  > returning. For instance, Py_RETURN_NONE will incref None and then
  > return it.
  >

The OP understands that this is not a normal thing to do. He's
trying to deliberately leak a reference for the purpose of diagnosing
a problem.

It would be interesting to see what the actual refcount is after
calling this function.

After calling this without a double increment in the function the ref count is 
still only 1 - which means that the 'return self' effectively does a double 
decrement. My original message includes the Python code which calls this 
'leaky' function and you can see that despite the 'leaky POC' doing an 
increment ref count drops back to one after the return.

You are right this is not a normal thing to do, I am trying to understand the 
behaviour so my library does the correct thing in all cases - for example - 
imagine you have two nodes in a tree :

A --- > B

And your Python code has a named reference to A, and B also maintains a 
reference to A as it's parent.

In this case I would expect A to have a reference count of 2 (counted as 3 
through sys.getrefcount() - one for the named reference in the Python code - 
and one for the link from B back to A; I would also expect B to have a 
reference count here of 1 (just the reference from A - assuming nothing else 
referenced B).

My original code was incrementing the ref counts of A and B and then returning 
A. within the Python test code A had a refcount of 1 (and not the expected 2), 
but the refcount from B was correct as far as I could tell.



Yes, and that's why I was saying it would need a *second* incref.

ChrisA

Thank you to all of you for trying to help - I accept that the only way to make 
the code work is to do a 2nd increment.

I don't understand why doing a 'return self' would result in a double decrement 
- that seems utterly bizzare behaviour - it obviously works, but why.

The return self in C will not change the ref count.

I would suggest setting a break point in your code and stepping out of the 
function and seeing that python’s code does to the ref count.

Barry


Barry,

something odd is going on because the Python code isn't doing anything 
that would cause the reference count to go from 3 inside the C function 
to 1 once the method call is complete.


As far as I know the only things that impact the reference counts are :

 * Increments due to assigning a new name or adding it to a container.
 * Increment due to passing the object to a function (since that binds
   a new name)
 * Decrements due to deletion of a name
 * Decrement due to going out of scope
 * Decrement due to being removed from a container.

None of those things are happening in the python code.

As posted in the original message - immediately before the call to the C 
function/method sys.getrefcount reports the count to be 2 (meaning it is 
actually a 1).


Inside the C function the ref count is incremented and the Py_REFCNT 
macro reports the count as 3 inside the C function as expected (1 for 
the name in the Python code, 1 for the argument as passed to the C 
function, and 1 for the increment), so outside the function one would 
expect the ref count to now be 2 (since the reference caused by calling 
the function is then reversed).


However - Immediately outside the C function and back in the Python code 
sys.getrefcount reports the count to be 2 again - meaning it is now 
really 1. So that means that the refcount has been decremented twice 
in-between the return of the C function and the execution of the 
immediate next python statement. I understand one of those decrements - 
the parameter's ref count is incremented on the way in so the same 
object is decremented on the way out (so that calls don't leak 
references) but I don't understand where the second decrement is coming 
from.


Again there is nothing in the Python code that would cause that 
decrement - the decrement behavior is in the Python runtime.





--
Anthony Flury
email :anthony.fl...@btinternet.com

--
https://mail.python.org/mailman/listinfo/python-list


--
Anthony Flury
email :anthony.fl...@btinternet.com
--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Barry]

> On 25 Jan 2022, at 14:50, Tony Flury via Python-list  
> wrote:
> 
> 
>> On 20/01/2022 23:12, Chris Angelico wrote:
>>> On Fri, 21 Jan 2022 at 10:10, Greg Ewing  
>>> wrote:
>>> On 20/01/22 12:09 am, Chris Angelico wrote:
 At this point, the refcount has indeed been increased.
 
>   return self;
>  }
 And then you say "my return value is this object".
 
 So you're incrementing the refcount, then returning it without
 incrementing the refcount. Your code is actually equivalent to "return
 self".
>>> Chris, you're not making any sense. This is C code, so there's no
>>> way that "return x" can change the reference count of x.
>> Yeah, I wasn't clear there. It was equivalent to *the Python code*
>> "return self". My apologies.
>> 
>>>  > The normal thing to do is to add a reference to whatever you're
>>>  > returning. For instance, Py_RETURN_NONE will incref None and then
>>>  > return it.
>>>  >
>>> 
>>> The OP understands that this is not a normal thing to do. He's
>>> trying to deliberately leak a reference for the purpose of diagnosing
>>> a problem.
>>> 
>>> It would be interesting to see what the actual refcount is after
>>> calling this function.
> 
> After calling this without a double increment in the function the ref count 
> is still only 1 - which means that the 'return self' effectively does a 
> double decrement. My original message includes the Python code which calls 
> this 'leaky' function and you can see that despite the 'leaky POC' doing an 
> increment ref count drops back to one after the return.
> 
> You are right this is not a normal thing to do, I am trying to understand the 
> behaviour so my library does the correct thing in all cases - for example - 
> imagine you have two nodes in a tree :
> 
> A --- > B
> 
> And your Python code has a named reference to A, and B also maintains a 
> reference to A as it's parent.
> 
> In this case I would expect A to have a reference count of 2 (counted as 3 
> through sys.getrefcount() - one for the named reference in the Python code - 
> and one for the link from B back to A; I would also expect B to have a 
> reference count here of 1 (just the reference from A - assuming nothing else 
> referenced B).
> 
> My original code was incrementing the ref counts of A and B and then 
> returning A. within the Python test code A had a refcount of 1 (and not the 
> expected 2), but the refcount from B was correct as far as I could tell.
> 
> 
>> Yes, and that's why I was saying it would need a *second* incref.
>> 
>> ChrisA
> 
> Thank you to all of you for trying to help - I accept that the only way to 
> make the code work is to do a 2nd increment.
> 
> I don't understand why doing a 'return self' would result in a double 
> decrement - that seems utterly bizzare behaviour - it obviously works, but 
> why.

The return self in C will not change the ref count.

I would suggest setting a break point in your code and stepping out of the 
function and seeing that python’s code does to the ref count.

Barry
> 
> 
> 
> -- 
> Anthony Flury
> email : anthony.fl...@btinternet.com
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

On 20/01/2022 23:12, Chris Angelico wrote:

On Fri, 21 Jan 2022 at 10:10, Greg Ewing  wrote:

On 20/01/22 12:09 am, Chris Angelico wrote:

At this point, the refcount has indeed been increased.


   return self;
  }

And then you say "my return value is this object".

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".

Chris, you're not making any sense. This is C code, so there's no
way that "return x" can change the reference count of x.

Yeah, I wasn't clear there. It was equivalent to *the Python code*
"return self". My apologies.


  > The normal thing to do is to add a reference to whatever you're
  > returning. For instance, Py_RETURN_NONE will incref None and then
  > return it.
  >

The OP understands that this is not a normal thing to do. He's
trying to deliberately leak a reference for the purpose of diagnosing
a problem.

It would be interesting to see what the actual refcount is after
calling this function.


After calling this without a double increment in the function the ref 
count is still only 1 - which means that the 'return self' effectively 
does a double decrement. My original message includes the Python code 
which calls this 'leaky' function and you can see that despite the 
'leaky POC' doing an increment ref count drops back to one after the return.


You are right this is not a normal thing to do, I am trying to 
understand the behaviour so my library does the correct thing in all 
cases - for example - imagine you have two nodes in a tree :


A --- > B

And your Python code has a named reference to A, and B also maintains a 
reference to A as it's parent.


In this case I would expect A to have a reference count of 2 (counted as 
3 through sys.getrefcount() - one for the named reference in the Python 
code - and one for the link from B back to A; I would also expect B to 
have a reference count here of 1 (just the reference from A - assuming 
nothing else referenced B).


My original code was incrementing the ref counts of A and B and then 
returning A. within the Python test code A had a refcount of 1 (and not 
the expected 2), but the refcount from B was correct as far as I could tell.




Yes, and that's why I was saying it would need a *second* incref.

ChrisA


Thank you to all of you for trying to help - I accept that the only way 
to make the code work is to do a 2nd increment.


I don't understand why doing a 'return self' would result in a double 
decrement - that seems utterly bizzare behaviour - it obviously works, 
but why.




--
Anthony Flury
email : anthony.fl...@btinternet.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Chris Angelico]

On Fri, 21 Jan 2022 at 10:10, Greg Ewing  wrote:
>
> On 20/01/22 12:09 am, Chris Angelico wrote:
> > At this point, the refcount has indeed been increased.
> >
> >>   return self;
> >>  }
> >
> > And then you say "my return value is this object".
> >
> > So you're incrementing the refcount, then returning it without
> > incrementing the refcount. Your code is actually equivalent to "return
> > self".
>
> Chris, you're not making any sense. This is C code, so there's no
> way that "return x" can change the reference count of x.

Yeah, I wasn't clear there. It was equivalent to *the Python code*
"return self". My apologies.

>  > The normal thing to do is to add a reference to whatever you're
>  > returning. For instance, Py_RETURN_NONE will incref None and then
>  > return it.
>  >
>
> The OP understands that this is not a normal thing to do. He's
> trying to deliberately leak a reference for the purpose of diagnosing
> a problem.
>
> It would be interesting to see what the actual refcount is after
> calling this function.
>

Yes, and that's why I was saying it would need a *second* incref.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Greg Ewing]

On 20/01/22 12:09 am, Chris Angelico wrote:

At this point, the refcount has indeed been increased.


  return self;
 }


And then you say "my return value is this object".

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".


Chris, you're not making any sense. This is C code, so there's no
way that "return x" can change the reference count of x.

> The normal thing to do is to add a reference to whatever you're
> returning. For instance, Py_RETURN_NONE will incref None and then
> return it.
>

The OP understands that this is not a normal thing to do. He's
trying to deliberately leak a reference for the purpose of diagnosing
a problem.

It would be interesting to see what the actual refcount is after
calling this function.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Barry Scott]

> On 19 Jan 2022, at 10:57, Tony Flury via Python-list  
> wrote:
> 
> I am writing a C extension module for an AVL tree, and I am trying to ensure 
> reference counting is done correctly. I was having a problem with the 
> reference counting so I worked up this little POC of the problem, and I hope 
> someone can explain this.

The ref counting in the C API not easy to get right. Sometimes you must inc 
some times you must not inc.
There are a lot of libraries that take that problem away from you.

For example I maintain PyCXX that is a C++ library that allows your drive the C 
API without needed to worry about ref counting.
There are lots of other libraries as well that aim to do the same thing.

http://cxx.sourceforge.net/ 

Barry




> 
> Extension function :
> 
>   static PyObject *_Node_test_ref_count(PyObject *self)
>   {
>printf("\nIncrementing ref count for self - just for the hell
>   of it\n");
>printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
>Py_INCREF(self);
>printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
>fflush(stdout);
>return self;
>   }
> 
> As you can see this function purely increments the reference count of the 
> instance.
> 
> /Note: I understand normally this would be the wrong this to do, but this is 
> a POC of the issue, not live code. In the live code I am attaching a 2nd 
> nodes to each other, and the live code therefore increments the ref-count for 
> both objects - so even if the Python code deletes it's reference the 
> reference count for the instance should still be 1 in order to ensure it 
> doesn't get garbage collected./
> 
> This function is exposed as the test_ref method.
> 
> This is the test case :
> 
> def test_000_009_test_ref_count(self):
> node = _Node("Hello")
> self.assertEqual(sys.getrefcount(node), 2)
> node.test_ref()
> self.assertEqual(sys.getrefcount(node), 3)
> 
> The output of this test case is :
> 
> test_000_009_test_ref_count (__main__.TestNode) ...
> Incrementing ref count for self - just for the hell of it
> 
>  before self has a ref count of 2
> 
>  after self has a ref count of 3
> FAIL
> 
> ==
> FAIL: test_000_009_test_ref_count (__main__.TestNode)
> --
> Traceback (most recent call last):
>   File "/home/tony/Development/python/orderedtree/tests/test_orderedtree.py", 
> line 62, in test_000_009_test_ref_count
> self.assertEqual(sys.getrefcount(node), 3)
> AssertionError: 2 != 3
> 
> So I understand why the first assert will be true - when the 
> sys.getrefcount() function is called the ref count is incremented temporarily 
> (as a borrowed reference), so there are now two references - the 'node' 
> variable, and the borrowed reference in the function call.
> 
> We then call the 'test_ref' method, and again that call causes a borrowed 
> reference (hence the ref count being 2 initially within the method). The 
> 'test_ref' method increments the reference of the instance - as you can see 
> from the output we now have a ref count of 3 - (that count is the 'node' 
> variable in the test case, the borrowed reference due to the method call, and 
> the artificial increment from the 'ref_test' method).
> 
> When the 'ref_test' method exits I would expect the ref count of the instance 
> to now be 2 (one for the 'node' variable, and one as a result of the 
> artificial increment increment').
> 
> I would therefore expect the 2nd assertEqual in the test case to succeed. - 
> in this case the borrowed reference within sys.getfrefcount() should cause 
> the count to be 3.
> 
> As you see though that 2nd assertEqual fails - suggesting that the refcount 
> of 'node' is actually only 1 when the 'test_ref' method exits.
> 
> Can someone explain why the 'test_ref' method fails to change the refcount of 
> the 'node' instance.
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Chris Angelico]

On Thu, Jan 20, 2022 at 1:22 AM Tony Flury  wrote:
>
>
> On 19/01/2022 11:09, Chris Angelico wrote:
> > On Wed, Jan 19, 2022 at 10:00 PM Tony Flury via Python-list
> >  wrote:
> >> Extension function :
> >>
> >>  static PyObject *_Node_test_ref_count(PyObject *self)
> >>  {
> >>   printf("\nIncrementing ref count for self - just for the hell
> >>  of it\n");
> >>   printf("\n before self has a ref count of %ld\n", 
> >> Py_REFCNT(self));
> >>   Py_INCREF(self);
> >>   printf("\n after self has a ref count of %ld\n", 
> >> Py_REFCNT(self));
> >>   fflush(stdout);
> > At this point, the refcount has indeed been increased.
> >
> >>   return self;
> >>  }
> > And then you say "my return value is this object".
> >
> > The normal thing to do is to add a reference to whatever you're
> > returning. For instance, Py_RETURN_NONE will incref None and then
> > return it.
> >
> > So you're incrementing the refcount, then returning it without
> > incrementing the refcount. Your code is actually equivalent to "return
> > self".
> >
> > In order to actually leak a reference, you'd need to incref it twice.
> >
> > ChrisA
>
>
> Chris - I am still puzzled - does  doing 'return self' automatically
> decrement the ref count of the object ?, and why is that the desired
> behaviour ? Effectively it results in a decrement of two, since at the
> exit of the function the ref count is only 1 (as witnessed by the
> subsequent call to assertEqual).

Imagine you have a function which creates a brand new object. What
should its reference count be just before you return it? What should
the refcount be after it's been given to the caller? There has to be a
reference at all times, but only one, so that it can be properly
garbage collected; the act of returning it from a function has to
steal away a reference.

> (I am not suggesting that it should be changed - I understand that would
> be a breaking change !).
>
> You say I am returning it without incrementing, but I am explicitly
> incrementing it before the return.
>

Yes, you are incrementing it - but only once. The act of returning
something that you already have a reference to has to increment the
reference count and then return the thing. Picture this Python code:

_THING = object()
def get_thing():
return _THING

When you call get_thing(), it needs to give you back a new reference
to that object. Written in C, that would have to increment the
refcount, then return the thing. (If you disassemble that code in
CPython, what you'll see is that it does a LOAD_GLOBAL, which does the
reference count incrementing, and then it returns what's on the top of
the stack. In C, you simply return the pointer, so you manually
increment the refcount first.)

In order to increment the reference count and keep it higher, you need
to (1) increment the reference count, then (2) return the object,
which consists of incrementing the reference count and returning the
pointer.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

On 19/01/2022 11:09, Chris Angelico wrote:

On Wed, Jan 19, 2022 at 10:00 PM Tony Flury via Python-list
 wrote:

Extension function :

 static PyObject *_Node_test_ref_count(PyObject *self)
 {
  printf("\nIncrementing ref count for self - just for the hell
 of it\n");
  printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
  Py_INCREF(self);
  printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
  fflush(stdout);

At this point, the refcount has indeed been increased.


  return self;
 }

And then you say "my return value is this object".

The normal thing to do is to add a reference to whatever you're
returning. For instance, Py_RETURN_NONE will incref None and then
return it.

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".

In order to actually leak a reference, you'd need to incref it twice.

ChrisA



Chris - I am still puzzled - does  doing 'return self' automatically 
decrement the ref count of the object ?, and why is that the desired 
behaviour ? Effectively it results in a decrement of two, since at the 
exit of the function the ref count is only 1 (as witnessed by the 
subsequent call to assertEqual).


(I am not suggesting that it should be changed - I understand that would 
be a breaking change !).


You say I am returning it without incrementing, but I am explicitly 
incrementing it before the return.




--
https://mail.python.org/mailman/listinfo/python-list

Re: Puzzling behaviour of Py_IncRef

[Chris Angelico]

On Wed, Jan 19, 2022 at 10:00 PM Tony Flury via Python-list
 wrote:
> Extension function :
>
> static PyObject *_Node_test_ref_count(PyObject *self)
> {
>  printf("\nIncrementing ref count for self - just for the hell
> of it\n");
>  printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
>  Py_INCREF(self);
>  printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
>  fflush(stdout);

At this point, the refcount has indeed been increased.

>  return self;
> }

And then you say "my return value is this object".

The normal thing to do is to add a reference to whatever you're
returning. For instance, Py_RETURN_NONE will incref None and then
return it.

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".

In order to actually leak a reference, you'd need to incref it twice.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Puzzling behaviour of Py_IncRef

[Tony Flury via Python-list]

I am writing a C extension module for an AVL tree, and I am trying to 
ensure reference counting is done correctly. I was having a problem with 
the reference counting so I worked up this little POC of the problem, 
and I hope someone can explain this.


Extension function :

   static PyObject *_Node_test_ref_count(PyObject *self)
   {
    printf("\nIncrementing ref count for self - just for the hell
   of it\n");
    printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
    Py_INCREF(self);
    printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
    fflush(stdout);
    return self;
   }

As you can see this function purely increments the reference count of 
the instance.


/Note: I understand normally this would be the wrong this to do, but 
this is a POC of the issue, not live code. In the live code I am 
attaching a 2nd nodes to each other, and the live code therefore 
increments the ref-count for both objects - so even if the Python code 
deletes it's reference the reference count for the instance should still 
be 1 in order to ensure it doesn't get garbage collected./


This function is exposed as the test_ref method.

This is the test case :

    def test_000_009_test_ref_count(self):
    node = _Node("Hello")
    self.assertEqual(sys.getrefcount(node), 2)
    node.test_ref()
    self.assertEqual(sys.getrefcount(node), 3)

The output of this test case is :

test_000_009_test_ref_count (__main__.TestNode) ...
Incrementing ref count for self - just for the hell of it

 before self has a ref count of 2

 after self has a ref count of 3
FAIL

==
FAIL: test_000_009_test_ref_count (__main__.TestNode)
--
Traceback (most recent call last):
  File 
"/home/tony/Development/python/orderedtree/tests/test_orderedtree.py", 
line 62, in test_000_009_test_ref_count

    self.assertEqual(sys.getrefcount(node), 3)
AssertionError: 2 != 3

So I understand why the first assert will be true - when the 
sys.getrefcount() function is called the ref count is incremented 
temporarily (as a borrowed reference), so there are now two references - 
the 'node' variable, and the borrowed reference in the function call.


We then call the 'test_ref' method, and again that call causes a 
borrowed reference (hence the ref count being 2 initially within the 
method). The 'test_ref' method increments the reference of the instance 
- as you can see from the output we now have a ref count of 3 - (that 
count is the 'node' variable in the test case, the borrowed reference 
due to the method call, and the artificial increment from the 'ref_test' 
method).


When the 'ref_test' method exits I would expect the ref count of the 
instance to now be 2 (one for the 'node' variable, and one as a result 
of the artificial increment increment').


I would therefore expect the 2nd assertEqual in the test case to 
succeed. - in this case the borrowed reference within sys.getfrefcount() 
should cause the count to be 3.


As you see though that 2nd assertEqual fails - suggesting that the 
refcount of 'node' is actually only 1 when the 'test_ref' method exits.


Can someone explain why the 'test_ref' method fails to change the 
refcount of the 'node' instance.


--
https://mail.python.org/mailman/listinfo/python-list