[Tcl Java] Fwd: TclBlend Ref Counting Bug (GC related)
Background: The relationship between the various objects are quite tricky. It takes some time to think them through. Basically, to reference a Tcl_Obj, the C structure representing a Tcl object in the C Tcl interpreter, TclBlend creates a TclObject, a Java class instance using the following relationship: TclObject (tcl.lang.TclObject, Java class) contains CObject (tcl.lang.CObject, Java class) contains a pointer to Tcl_Obj CObject is a subclass of InternalObject (tcl.lang.InternalObject). InternalObject is an abstract class containing an interface so that TclObject can be used as the common interface to not only Tcl_Obj, but also other types of TclBlend objects such as ReflectObject, TclInteger, etc, all of those are subclass of InternalObject. On the C side, the C Tcl interpreter needs access to Java objects defined in TclBlend, such as ReflectObject, TclInteger, etc. TclBlend uses a Tcl_Obj implementation to wrap around any TclObject using the following relationship: Tcl_Obj (A C struct with function pointers to functions that are specifically designed to manipulate TclBlend's TclObject. Defined in javaObj.c in TclBlend.) contains reference/pointer to the Java TclObject It seems that you can get infinite recusive containment as in: Tcl_Object contains Tcl_Obj contains Tcl_Object contains Tcl_Obj contains ... TclBlend is smart enough to detect such attempt. When it is detected, TclBlend removes the extra level containment so that most of the time you will have either: Tcl_Object contains Tcl_Obj or Tcl_Obj contains Tcl_Object The Problem: There are 2 conditions, which taken together causes problem with the above setup in TclBlend. 1. How are the objects been cleaned up (free'ed)? Tcl_Obj needs to be cleaned up excplictly using reference counting. Tcl_Object can be implicitly cleaned up using the GC. The original implementation in TclBlend is to use the GC to implicitly clean up Tcl_Object, which in turns cleans up the Tcl_Obj using an explict reference count decrement. By transitivity, this means that the Tcl_Obj are cleaned up implicitly by the GC. There is no problem by itself. 2. Tcl_Obj can only be free'ed by the same thread that created it (due to performance benefits). This behavior is by design and does not cause any problem by itself either because Tcl programs are usually written in C/C++, which always uses explicit memory management without anything like a GC thread. Taken 1 and 2 together causes a problem. The GC thread may be freeing a Tcl_Obj that is not created by the GC thread. This produces concurrency problems in Tcl. The proposed solution is to never use the GC thread to free Tcl_Obj and always use the thread that created the Tcl_Obj to free the Tcl_Obj. See below for more analysis of the problem. You can search the maling list archive for more discussion on this under the general subject of "GC". -- Jiang Wu [EMAIL PROTECTED] --- Start of forwarded message --- Subject: TclBlend Ref Counting Bug (GC related) To: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] From: Jiang Wu [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Date: 08 Aug 2000 00:12:48 PDT Currently, there is a plan to make the Java GC perform some cleanup on Tcl_Obj in TclBlend. Looking at the root cause for the GC trouble, I think there is a problem with how TclBlend uses Tcl_Obj reference counting. Here is my analysis. I think if we fix the reference counting, we don't need to do anything in the GC. TclBlend does not follow the normal Tcl_Obj reference counting usage. In TclBlend, a Java TclObject is used as the handle to the underlining Tcl_Obj: TclObject CObject Tcl_Obj Both Tcl_Obj and TclObject have reference counting. However, when incrementing or decrementing the reference count on the TclObject, this incrementing or decrementing is proxied into the native Tcl_Obj. This doesn't seem correct. In addition, when a new TclObject is created to wrap around a Tcl_Obj, the reference count of the TclObject is 0, but implicitly, the reference count of Tcl_Obj is incremented by 1. This doesn't seem right either. For example, in C, a call to Tcl_GetVar2Ex(...) returns a Tcl_Obj, of which, "The ref count for the returned object is _not_ incremented to reflect the returned reference; if you want to keep a reference to the object you must increment its ref count yourself." In TclBlend, the corresponding call is "interp-getVar()", which uses Tcl_GetVar2Ex(...). However, the resulting Tcl_Obj does have its reference count incremented by 1 implicitly by TclBlend. The problem with this implicit increment of native Tcl_Obj's reference count is that someone, namely the GC thread, must decrement the reference count. This causes all sort of undesirable behaviors such as the GC thread deadlocking because Tcl_Obj's
[Tcl Java] Fwd: TclBlend Ref Counting Bug (GC related)
"Jiang" == Jiang Wu [EMAIL PROTECTED] writes: Thanks, this description helps alot. Jiang Background: The relationship between the various objects Jiang are quite tricky. It takes some time to think them Jiang through. Basically, to reference a Tcl_Obj, the C Jiang structure representing a Tcl object in the C Tcl Jiang interpreter, TclBlend creates a TclObject, a Java class Jiang instance using the following relationship: Jiang TclObject (tcl.lang.TclObject, Java class) contains Jiang CObject (tcl.lang.CObject, Java class) contains Jianga pointer to Tcl_Obj In the case above what does the Tcl_Obj represent? Is it just a native tcl object such as a string, a list, or an array? Jiang CObject is a subclass of InternalObject Jiang (tcl.lang.InternalObject). InternalObject is an abstract Jiang class containing an interface so that TclObject can be used Jiang as the common interface to not only Tcl_Obj, but also other Jiang types of TclBlend objects such as ReflectObject, Jiang TclInteger, etc, all of those are subclass of Jiang InternalObject. Jiang On the C side, the C Tcl interpreter needs access to Java Jiang objects defined in TclBlend, such as ReflectObject, Jiang TclInteger, etc. TclBlend uses a Tcl_Obj implementation to Jiang wrap around any TclObject using the following relationship: Jiang Tcl_Obj (A C struct with function pointers to functions Jiang that are specifically designed to manipulate TclBlend's Jiang TclObject. Defined in javaObj.c in TclBlend.) contains Jiang reference/pointer to the Java TclObject This makes sense. The Tcl_Obj internal rep always holds a jobject handle to a java TclObject. Jiang It seems that you can get infinite recusive containment as Jiang in: Jiang Tcl_Object contains Jiang Tcl_Obj contains Jiang Tcl_Object contains Jiang Tcl_Obj contains ... Yeh this was confusing me, especially when I started thinking about the changes you made to the refcounting code. Jiang TclBlend is smart enough to detect such attempt. When it Jiang is detected, TclBlend removes the extra level containment Jiang so that most of the time you will have either: Is that what's going on in the setInternalRep method of the TclObject class? It checks for a CObject internal rep and creates a new reference to the current TclObject instance if the new rep is not of type CObject. Where else is this detection done? JiangTcl_Object contains Tcl_Obj Jiang or JiangTcl_Obj contains Tcl_Object Jiang The Problem: Jiang The proposed solution is to never use the GC thread to free Jiang Tcl_Obj and always use the thread that created the Tcl_Obj Jiang to free the Tcl_Obj. My initial impression is that this is the correct way to do it, though it can be tricky to get all of the reference counting right. -Dan The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: Fwd: TclBlend Ref Counting Bug (GC related)
On Fri, 13 October 2000, Daniel Wickstrom wrote: Jiang TclObject (tcl.lang.TclObject, Java class) contains Jiang CObject (tcl.lang.CObject, Java class) contains Jianga pointer to Tcl_Obj In the case above what does the Tcl_Obj represent? Is it just a native tcl object such as a string, a list, or an array? Yes. Typically, Tcl_Obj is a native C version of the Tcl list, array, int, ... But not always. See the end of the email for an example. Jiang Tcl_Object contains Jiang Tcl_Obj contains Jiang Tcl_Object contains Jiang Tcl_Obj contains ... Yeh this was confusing me, especially when I started thinking about the changes you made to the refcounting code. It is best to think that this never happens when trying to understand the general structure of the code. Otherwise, one can go quite insane :) Jiang TclBlend is smart enough to detect such attempt. When it Jiang is detected, TclBlend removes the extra level containment Jiang so that most of the time you will have either: Is that what's going on in the setInternalRep method of the TclObject class? It checks for a CObject internal rep and creates a new reference to the current TclObject instance if the new rep is not of type CObject. Where else is this detection done? Yes. It is guarding against recursion. Also see javaObj.c where it guards against having a Tcl_Obj contains TclObject contains Tcl_Obj contains ... type of recursion. My initial impression is that this is the correct way to do it, though it can be tricky to get all of the reference counting right. The rule of using reference counting properly is that if you want to keep the TclObject across calls to the Tcl interpreter, you must preserve it and release it later. However, TclBlend didn't need this rule before because it implicitly preserved and released Tcl_Obj. There are may be several places in the TclJava layer that contain reference counting bugs. These bugs typically results in core dumps because they would cause reference to already free'ed Tcl_Obj to be used. One example was in the ReflectObject, which allows you to create: Tcl_Obj contains (1) TclObject contains ReflectObject contains TclObject contains Tcl_Obj (2) Note that the above does have Tcl_Obj (1) containing another Tcl_Obj (2), which is a form of recursive containment. This type of recursion is not detected by TclBlend because you put another level of indirection in the chain of containment. The bug was that the ReflectObject did not increment the reference count of the TclObject even though the ReflectObject is going to keep the TclObject for the long term by putting it in a Hastable for use later. The lowest level Tcl_Obj (2) can be free'ed by Tcl interpreter when its reference count decreases to zero. Then later, the ReflectObject attempts to use the pointer to the free'ed Tcl_Obj (2), which can cause a segmentation fault. There may be other places in TclJava, where the above type of thing happen. -- Jiang Wu [EMAIL PROTECTED] -- [EMAIL PROTECTED] is brought to you by the Stanford Alumni Association and Critical Path. The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
