[issue9417] Declaring a class creates circular references

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Kay, please ask your question on python-list or a similar forum.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Kay Hayen]

Kay Hayen  added the comment:

What I also meant to say, is that debug Python is not affected, and this had me 
deeply confused. Any ideas how that could happen?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Kay Hayen]

Kay Hayen  added the comment:

Today I changed my reference count tests to not use debug Python and came 
across this issue.

>From my testing, Python3.4 is the first affected version, Python3.3 was still 
>fine, so were 2.7 and 2.6. 

This leaks:

def simpleFunction39():
class Parent(str):
pass

This does not:

def simpleFunction39():
class Parent(object):
pass

When comparing (or attempted to) gc.get_objects() before and after, I was 
unable to point to any count that would be different, that was confusing. I can 
rule out a mistake in my changes to how the counts are achieved, because every 
other reference count test works.

--
nosy: +kayhayen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Georg Brandl]

Georg Brandl added the comment:

I don't think this can go somewhere useful.

--
resolution:  - wont fix
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Mark Lawrence]

Mark Lawrence added the comment:

Does anyone wish to take this forward, is it simply dead in the water or what?  
You might like to note that msg111963 refers to #812369 which was superseded by 
#18214 which was fixed in 3.4.

--
nosy: +BreamoreBoy
versions: +Python 3.5 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[David Stanek]

Changes by David Stanek dsta...@dstanek.com:


--
nosy: +dstanek

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

krisvale:

 What about my suggestion of simply providing a convention on how to disable 
 links manually, similar to how minidom.document does it with an unlink 
 method?  If these cases are documented, then there shouldn't be any extra 
 bother to do so.
We could provide an builtin unlink() function that calls any __unlink__ special 
method if present.


Maybe through this way such problems can be solved, but it seems it's not 
graceful and pythonic enough. The reason such an unlink method appears in 
xml.dom.minidom is because usually large xml document can eat up so much memory 
that we had to do something to make the memory garbaged sooner. But adding this 
as a standard __unlink__ method and requiring explicitly call the unlink() 
method make it similar as using c  c++, which require explicitly memory 
release. 

Besides, as you mentioned, exposes an implementation detail of C python, it's 
better that the object reference mechanism be fully controlled by python core 
and not be awared by users at all. That is also one of the purposes of 
designing python. 

I prefer terry's idea, which suggests adding a metaclass that produces rc-free 
class to stdlib. These class should behave almost the same as the ones type 
metaclass produces. In order not to introduce the slower lookup, we can also 
add a little differences on usage of such classes, for example, internally 
store mro list without the class itself, but construct a full mro list when 
getting the __mro__ attribute, and different super implementation, since 
explicitly getting the mro is not so common in practice.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Daniel Urban]

Changes by Daniel Urban urban.dani...@gmail.com:


--
nosy: +durban

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

I am opposed to a new 'unlink' builtin and '__unlink__' special method. There 
are more than enough builtin functions now, and this need is specific to those 
who both make lots temporary classes that they need garbage collected AND need 
to run with gc turned off. The intersection of these two small groups is 
probably very small. Since turning gc off is only an option with CPython (and 
PyPy?), this need is also implementation specific.

Before anything were added to the stdlib (and the need *might* be considered 
too rare for that) it should be tested in real use by more than one person. It 
appears that there are at least a couple of options for implementing a no_cr 
class. They could be attached to this issue, posted on the cookbook, or 
uploaded to PyPI.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

Andrea: for the purpose of this tracker, a 'bug' is a discrepancy between doc 
and behavior. The fact that (new-style) classes have circular references was 
known when they were introduced in 2.2. That fact is not a bug in the above 
sense. It is a design tradeoff that works OK for the typical case of classes 
remaining until the end of a program run. Changing the tradeoff embodied in 
type() is not likely to be accepted.

That said, an alternate no_cr (no circular reference) metaclass (perhaps a type 
subclass) that contructed cr-free classes, even at the cost of slower lookup, 
might be a good addition to the stdlib. Ray's ideas seems to me at least 
plausible. My first thought on location is in functools.

--
nosy: +terry.reedy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

What about my suggestion of simply providing a convention on how to disable 
links manually, similar to how minidom.document does it with an unlink 
method?  If these cases are documented, then there shouldn't be any extra 
bother to do so.
We could provide an builtin unlink() function that calls any __unlink__ special 
method if present.

This sounds like a feature request, really, and one that is unlikely to get 
accepted since it sort of exposes an implementation detail of C python.  Oh 
well.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

 However here's a proposed solution:

* for the __mro__: instead of using a tuple, use a new object that inherits 
from it. This new object should use weak reference for the first item and 
should return the real object (if available) only in __getitem__().

* __objclass__ can should become a property based on weak references.



I'm afraid doing so could cause some public C API change. For example, the 
PyDescr_TYPE, if we implemented all the Descriptor object's d_type(that is the 
__objclass__ of all types of descriptors) based on weakref, we could have all 
the callers who call the descriptor functions to check weather the 
weak-referented class object has gone away.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Andrea Colangelo]

Andrea Colangelo andrea.colang...@gmail.com added the comment:

I can confirm this bug should be addressed some way. On my high-traffic server, 
keeping GC enabled causes performance issues due to this bug, and disabling it 
causes an out-of-memory within hours.

--
nosy: +warp10

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

Is this due to your adopting of Django?
Are there other guys who using Django suffer the same problem?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Andrea Corbellini]

New submission from Andrea Corbellini corbellini.and...@gmail.com:

Creating a class (either using the 'class' statement or using type()) creates a 
circular reference.

I've attached a script that simply demonstrates this. The problem is caused by 
the fact that MyClass.__dict__['__dict__'].__objclass__ is MyClass.

Although most of the times classes are never deleted when the interpreted 
exits, some programs (like the popular Django framework) create temporary 
classes. And this is a pain because you can't disable the garbage collector.

--
components: Interpreter Core
files: test.py
messages: 111935
nosy: candrea
priority: normal
severity: normal
status: open
title: Declaring a class creates circular references
type: resource usage
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file18251/test.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Is there a reason why you want to disable the cyclic garbage collector?

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
versions:  -Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

This should not be a problem, I think. We cannot avoid such a circular 
reference. 

I believe the main reason of the circular reference is in MyClass.__mro__, this 
is a list and contains a reference to MyClass. You can add __slots__ = [] to 
MyClass's class body to avoid __dict__(the one you mentioned) and another 
member which contains two more references to MyClass, but the __mro__ still 
contains the last reference to MyClass.

--
nosy: +ysj.ray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Andrea Corbellini]

Andrea Corbellini corbellini.and...@gmail.com added the comment:

Disabling the GC can increase performances (although not significantly). But 
this bug is the cause of other problems too: what if the metaclass contains a 
__del__() method?

An another issue that I've found is that debugging is harder. I always try to 
avoid to create ref cycles in my code, also if my objects are collectable. In 
this way, I'm sure that I'll always be able to add a __del__ method in the 
future without problems. However, I can't easily check ref cycles without 
manually inspecting `gc.garbage`.

And also, specifying DEBUG_SAVEALL will put all the deleted classes and their 
attributes in the garbage, which makes debugging *very* hard in case of a 
leaking program.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

I couldn't imagine why a metaclass would want to have a __del__ method...

--
nosy: +georg.brandl

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Andrea Corbellini]

Andrea Corbellini corbellini.and...@gmail.com added the comment:

Having a __del__ inside a metaclass is strange, I know... but probably there 
are situations where you need to do so. Why shouldn't a developer be able to 
add a __del__ to a metaclass without creating uncollectable objects? I don't 
think this behavior is by design.

Also, doing random contributions to various projects I've seen many odd things. 
However I don't think that the bug tracker is the right place to discuss 
development practices. A bug that causes problems in unusual situations is 
still a bug. :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Whether this is a bug is not clear.  Sometimes it's just impossible not to 
create cycles, and classes may just be one instance of that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +krisvale

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Andrea Corbellini]

Andrea Corbellini corbellini.and...@gmail.com added the comment:

This is an unwanted an unexpected behavior, so this is a bug by definition. If 
it's not easy to fix, it's a different matter.

However here's a proposed solution:

* for the __mro__: instead of using a tuple, use a new object that inherits 
from it. This new object should use weak reference for the first item and 
should return the real object (if available) only in __getitem__().

* __objclass__ can should become a property based on weak references.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Alexander Belopolsky]

Alexander Belopolsky belopol...@users.sourceforge.net added the comment:

This was discussed in issue1545463 which was deemed not to be worth fixing.  
The particular bug described in issue1545463 would also be fixed by the still 
open issue812369.  So yes, there are cases where these cycles are a problem 
even with gc.

--
nosy: +belopolsky

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

This is not so easy: the __mro__ tuple, as its name says, is used internally 
for method resolution, or finding attributes on the type's bases. __objclass__ 
is used whenever the descriptor is accessed.  These operations are involved in 
every method call.

If you want this to go somewhere, you will have to produce a patch and 
demonstrate that the slowdown does not unsuitably impact performance.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9417] Declaring a class creates circular references

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Interesting.  Here is a case for disabling GC:  

We intentionally disable GC in Eve-online, because of the unpredictable lag 
spikes it produces.  In a server application with gigabytes of python objects, 
GC can cause hickups of some seconds, which is unacceptable when trying to keep 
the latency low.

We were considering using Django for our backend web framework once.  Good that 
we didn't, since it would have leaked.

We do try to keep our code free from circular references, and occasionally run 
single GC passes on our test cluster to weed out any that may have formed 
accidentally (gc.garbage with DEBUG_LEAKS)

IMHO, python should try to be as free from these as possible, although it is 
admittedly not always easy (see recursive closures, issue 7464)

An alternative, is to do something like the minidom module does:  Provide a 
unlink method (or similar) to manually nerf objects before forgetting them.  
Perhaps this could be standardized with an unlink keyword and a __unlink__ 
special method?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9417
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com