[Tcl Java] Re: TclBlend Initialization Mutex
On Wed, 9 Aug 2000, Scott Stanton wrote: Mo DeJong said: It sure would make life easier if we just required Tcl threads to use Tcl Blend. The notifier thing would go away because we could count on threads being there. I'm all in favor of this. As the threaded version of Tcl becomes more stable, I'd like to see it become the default, anyway. --Scott Lets see if we can list the pros and cons before we make a final decision. Pro: Tcl Blend would work, we would avoid strange edge cases. We would not need the notifier hacks Con: Non-thread safe extensions may have problems under Unix. Need to detect if Tcl is compiled with thread support. (not too hard) I am sure I have missed a few. Mo DeJong Red Hat Inc The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
-Original Message- From: Mo DeJong [mailto:[EMAIL PROTECTED]] Sent: Wednesday, August 09, 2000 1:34 PM To: [EMAIL PROTECTED] Subject: [Tcl Java] Re: TclBlend Initialization Mutex Con: Non-thread safe extensions may have problems under Unix. It seems that there are at least two reasons why non-thread safe extensions can have problems in a threaded process like Tcl or JVM. 1. access system level globals Hopefully, this can be fixed by compiling with -D_REENTRANT. 2. access Tcl globals What to do? Right now, I am experimenting with using -D_REENTRANT (but without -DTCL_THREADS) to compile a non-threaded Tcl. The Solaris problem did go away using this special Tcl compilation. -- Jiang Wu [EMAIL PROTECTED] The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
On Wed, 9 Aug 2000, Jiang Wu wrote: -Original Message- From: Mo DeJong [mailto:[EMAIL PROTECTED]] Sent: Wednesday, August 09, 2000 1:34 PM To: [EMAIL PROTECTED] Subject: [Tcl Java] Re: TclBlend Initialization Mutex Con: Non-thread safe extensions may have problems under Unix. It seems that there are at least two reasons why non-thread safe extensions can have problems in a threaded process like Tcl or JVM. 1. access system level globals Hopefully, this can be fixed by compiling with -D_REENTRANT. 2. access Tcl globals What to do? Right now, I am experimenting with using -D_REENTRANT (but without -DTCL_THREADS) to compile a non-threaded Tcl. The Solaris problem did go away using this special Tcl compilation. But you would need to recompile Tcl and all the extensions with -D_REENTRANT. While I would agree that this is a neat hack, I am not sure it is a good solution. Folks that use a pre-compiled version will not be able to use this. There is lots of history that suggests that core patches are not a good solution (Itcl). We may end up having to do away with pre-compiled versions of Tcl Blend all together, just so that people do not make this mistake. Mo DeJong Red Hat Inc The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
-Original Message- From: Mo DeJong [mailto:[EMAIL PROTECTED]] Sent: Wednesday, August 09, 2000 4:14 PM To: [EMAIL PROTECTED] Subject: [Tcl Java] Re: TclBlend Initialization Mutex Having to keep two separate versions of Tcl around is just not convenient. I am open to any solutions that can make these extensions work well with Java and Tcl. They need to be fixed. A threaded version of Tcl and a stub enabled extension will need to be the "minimum requirement" if you want to use Tcl Blend. Yes, they may be fixed in the future. Though I can't stop working waiting for them to be fixed. So I have to work with what I can get today. This is an unfortunate fact for UNIX due to its history and vendor variations. Not having pre-compiled versions do not solve the real problem. It only pushes the problem to the user. That may be the only solution, but it is very good in my mind. Well, the -D_REENTRANT thing is going to require a recompile of the extension, so I don't see how it is such a big deal. The point I was making is that having everyone compiling Tcl/TclBlend/extensions is big waste of time. Ideally, people can download and use so that they can focus on developing solutions using the tools rather than worrying about compiling the tools. But I guess that is as much a wishful thinking as hoping that the extensions will be fixed to be thread-safe. -- Jiang Wu [EMAIL PROTECTED] The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
-Original Message- From: Mo DeJong [mailto:[EMAIL PROTECTED]] Sent: Wednesday, August 09, 2000 5:14 PM To: [EMAIL PROTECTED] Subject: [Tcl Java] Re: TclBlend Initialization Mutex Yes, they may be fixed in the future. Though I can't stop working waiting for them to be fixed. So I have to work with what I can get today. Which means getting them working in threaded Tcl, right? Not always. The goal is to find tools and components that help with the fast development of products for our customers. Sometimes, we rather rewrite a piece of functionality from scratch, which makes it proprietary. We try to use as much off the shelf components as possible. But if the extension doesn't work off the shelf, then it is very hard to justify using it at all. We are not a Tcl only environment, if Tcl extensions don't work well, we start assimilating other technologies. (We are Borg ... :-) Binary downloads just don't work. I tired over an over again during the 1.2 process. People would complain that they did not want to compile, so I put up a binary version. Shock and horrors, people could not get that to work either. A binary has all sorts of problems, mismatched versions of Tcl, the JVM, wrong env settings, and so on. I certainly have a lot of sympathy for your frustration. I think the best approach is to just lock down Tcl Blend so that it only builds with a Thread enabled Tcl. People are going to need to download an build Tcl and Tcl Blend, there is just no way around it. It would be better to just work on the "Tcl SDK" idea, a common build environment would make things a lot easier. Let's say we all use "Tcl SDK". I get it and compile my own Tcl. Ship the binaries to my customers. They complain about things not working. The problem has not been solved by the SDK. There is a solution to the binary problem because many commercial companies do distribute binary programs and libraries. I guess the amount of resources involved in verifying these binaries is beyond the capability of one person with just a few pieces of equipment. It is certainly not right to blame any individual for non-working binary distributions of public domain software. Getting everyone to compile Tcl is not the right solution, but I suppose it is the cheapest solution and takes the heat off the people who already contributed a lot to the product. Maybe a company can provide binary distribution service and charge $99 per copy to cover the cost of the testing, follow the commercial Linux model. -- Jiang Wu [EMAIL PROTECTED] The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
Jiang Wu said: 1. In UNIX, TclBlend uses a pipe with non-blocking IO to wake up the Tcl event loop. If there are multiple event loops in several threads, will a single pipe based IO trigger all event loops to wake up? In the threaded version of Tcl, there are two distinct types of event loops. There is a single globally shared select loop (NotifierThreadProc). This thread performs the select() system call on behalf of all threads in the system. It has a single pipe that it includes in the select mask so other threads can wake the select thread up to force it to update its select mask. Whenever it detects an event, it notifies only the threads that have specified interest in that file descriptor. This typically only wakes up one thread since most of the time file descriptors are only owned by one thread. The second type of loop is the notifier loop created by each thread that waits for events. Each thread that uses the Tcl notifier APIs creates a separate event queue. If the event queue becomes empty, the notifier thread blocks waiting for notification from the select loop (or a timeout). So, with TclBlend, there should be one notifier loop per thread that contains an interpreter. There should be exactly one select loop for the entire process. 2. In Windows, a Win32 native window event is used to call Tcl_ServiceAll() to trigger the Tcl event loop. The same question applies, does Tcl_ServiceAll() trigger all Tcl event loops to wake up? In Windows, there is only one type of loop. A separate Win32 window event loop is used for each thread. The Tcl_ServiceAll() routine only affects a single thread's event queue. 3. Currently, I am seeing a problem on Solaris, where the non-blocking pipe used in TclBlend is triggering a busy loop on reading from the pipe. The similar code in Tcl 8.3.1 for the event loop has a slightly different implementation from that used in TclBlend with a comment about "reading only 1 byte at a time to avoid some race condition". I'm not sure what would be causing a busy loop. Can you tell what it is looping on? The basic procedure should be that one or more threads write a byte onto the pipe. The select thread will read one byte per select() iteration. The only way it could loop is if some thread was repeatedly writing to the pipe or if the select thread failed to consume data from the pipe. --Scott The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
On Tue, 08 August 2000, Scott Stanton wrote: Notes on the Tcl event loop Thanks for the information. It seems that multi-threaded Tcl already implemented some cross-platform thread notification, maybe TclBlend should use something like Tcl_ThreadAlert() to wake up the event loop. But this function is not available for non-threaded Tcl. So for single-threaded Tcl, TclBlend still needs to use the current custom notification system. used in TclBlend is triggering a busy loop on reading from the pipe. The similar code in Tcl 8.3.1 for the event loop has a slightly different implementation from that used in TclBlend with a comment about "reading only 1 byte at a time to avoid some race condition". I'm not sure what would be causing a busy loop. Can you tell what it is looping on? The basic procedure should be that one or more threads write a byte onto the pipe. The select thread will read one byte per select() iteration. The only way it could loop is if some thread was repeatedly writing to the pipe or if the select thread failed to consume data from the pipe. After spending some hours in gdb tracing the program, it seems that the problem occurs in tclUnixChan.c:FileInputProc() line 380: *errorCodePtr = errno; 1 byte is written to a non-blocking Tcl file channel. Then 1 byte is successfully read from the channel. Then Tcl calls FileInputProc() again to read more. There is no more data in the channel, so the call at tclUnixChan.c, line 376 bytesRead = read(fsPtr-fd, buf, (size_t) toRead); returns -1. However, for some reason, the system variable "errno" is always 0, not EAGAIN as expected by the caller function. This causes the caller function (tclIO.c:GetInput(), line 4865) to loop forever calling FileInputProc() over and over again. I have not determined what caused the problem with "errno". The Tcl library is loaded into a JVM at this point. So the process is running in multi-threaded mode. The Tcl library used is non-threaded, version 8.3.1. Can there be something bad about using "errno" from multiple threads? Another strange thing if I run the same Java program on a smaller example, then it seems to work OK. When the error does happen, there are many threads in the process. -- Jiang Wu [EMAIL PROTECTED] -- [EMAIL PROTECTED] is brought to you by the Stanford Alumni Association and Critical Path. The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
On 8 Aug 2000, Jiang Wu wrote: After spending some hours in gdb tracing the program, it seems that the problem occurs in tclUnixChan.c:FileInputProc() line 380: *errorCodePtr = errno; 1 byte is written to a non-blocking Tcl file channel. Then 1 byte is successfully read from the channel. Then Tcl calls FileInputProc() again to read more. There is no more data in the channel, so the call at tclUnixChan.c, line 376 bytesRead = read(fsPtr-fd, buf, (size_t) toRead); returns -1. However, for some reason, the system variable "errno" is always 0, not EAGAIN as expected by the caller function. This causes the caller function (tclIO.c:GetInput(), line 4865) to loop forever calling FileInputProc() over and over again. I have not determined what caused the problem with "errno". The Tcl library is loaded into a JVM at this point. So the process is running in multi-threaded mode. The Tcl library used is non-threaded, version 8.3.1. Can there be something bad about using "errno" from multiple threads? Under Solaris, errno becomes a function that returns a thread local variable when you compile with -D_REENTRANT. You compiled this version of Tcl with threads support, right? Another strange thing if I run the same Java program on a smaller example, then it seems to work OK. When the error does happen, there are many threads in the process. Sounds like a race condition dealing with ERRNO. I wonder if Tcl is not getting compiled without -D_REENTRANT or something. That seems unlikely. Tcl Blend just uses the CFLAGS set by Tcl. Mo DeJong Red Hat Inc The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
-Original Message- From: Mo DeJong [mailto:[EMAIL PROTECTED]] used is non-threaded, version 8.3.1. Can there be something bad about using "errno" from multiple threads? Under Solaris, errno becomes a function that returns a thread local variable when you compile with -D_REENTRANT. You compiled this version of Tcl with threads support, right? No :( I am using a regular non-thread Tcl library. Can I compile with -D_RENTRANT, but without threads? We have this requirement of being able to use Tcl extensions such as Scotty. So we are not using threaded Tcl. Darn! With problems like this, does it mean that there is no safe way of using a non-threaded Tcl in a JVM because the JVM is threaded by default. The JVM is going to allocate one thread to execute non-threaded Tcl code. Tcl tries to access system globals. JVM also tries to access the same system globals. -- Jiang Wu [EMAIL PROTECTED] The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
On Tue, 8 Aug 2000, Jiang Wu wrote: Darn! With problems like this, does it mean that there is no safe way of using a non-threaded Tcl in a JVM because the JVM is threaded by default. The JVM is going to allocate one thread to execute non-threaded Tcl code. Tcl tries to access system globals. JVM also tries to access the same system globals. That does seem like a problem. If we require "special hacks" on Tcl itself, then there is no point in trying to support a pre-compiled version of Tcl Blend that gets loaded into Tcl. Users will just have to download and build a threaded version of Tcl or use our special hacks. A "core patch" seems like a bad idea if you ask me. It sure would make life easier if we just required Tcl threads to use Tcl Blend. The notifier thing would go away because we could count on threads being there. Mo DeJong Red Hat Inc The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
I looked at the Window's version of Tcl. There is no special system level compiler flag to compile threaded versus non-threaded. So it seems that non-threaded Tcl on Windows may work fine with Java. I have to agree that requiring threaded Tcl may be the only option for UNIX. If non-threaded Tcl can interact badly with Java, so can these Tcl extensions that are not thread enabled. If that is the case, even if we can make Tcl work with the JVM, there is no guarantee that the extensions can work. -- Jiang Wu [EMAIL PROTECTED] From: Mo DeJong [EMAIL PROTECTED] Date: Tue, 8 Aug 2000 21:15:22 -0700 (PDT) To: [EMAIL PROTECTED] Subject: [Tcl Java] Re: TclBlend Initialization Mutex Resent-From: [EMAIL PROTECTED] Resent-Date: Tue, 8 Aug 2000 21:15:52 -0700 (PDT) On Tue, 8 Aug 2000, Jiang Wu wrote: Darn! With problems like this, does it mean that there is no safe way of using a non-threaded Tcl in a JVM because the JVM is threaded by default. The JVM is going to allocate one thread to execute non-threaded Tcl code. Tcl tries to access system globals. JVM also tries to access the same system globals. That does seem like a problem. If we require "special hacks" on Tcl itself, then there is no point in trying to support a pre-compiled version of Tcl Blend that gets loaded into Tcl. Users will just have to download and build a threaded version of Tcl or use our special hacks. A "core patch" seems like a bad idea if you ask me. It sure would make life easier if we just required Tcl threads to use Tcl Blend. The notifier thing would go away because we could count on threads being there. Mo DeJong Red Hat Inc The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
[Tcl Java] Re: TclBlend Initialization Mutex
Yes, you are correct. I noticed this after doing more research into the notifier and event loop implementation in Tcl. Per thread event loop can be achieve in Java easily. But there are some outstanding questions: 1. In UNIX, TclBlend uses a pipe with non-blocking IO to wake up the Tcl event loop. If there are multiple event loops in several threads, will a single pipe based IO trigger all event loops to wake up? 2. In Windows, a Win32 native window event is used to call Tcl_ServiceAll() to trigger the Tcl event loop. The same question applies, does Tcl_ServiceAll() trigger all Tcl event loops to wake up? 3. Currently, I am seeing a problem on Solaris, where the non-blocking pipe used in TclBlend is triggering a busy loop on reading from the pipe. The similar code in Tcl 8.3.1 for the event loop has a slightly different implementation from that used in TclBlend with a comment about "reading only 1 byte at a time to avoid some race condition". Your insights into these matter are greatly appreciated. -- Jiang Wu [EMAIL PROTECTED] From: Scott Stanton [EMAIL PROTECTED] Date: Mon, 07 Aug 2000 11:07:26 -0700 To: Jiang Wu [EMAIL PROTECTED] Cc: Mo DeJong [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: TclBlend Initialization Mutex Jiang Wu said: I added a global mutex to protect TclBlend initialization from mutiple threads. The patch is sent to Mo for check in. This mutex together with Mo's changes on removing the global lock and making JNIEnv thread local make TclBlend almost thread safe. I say "almost" because we still need to make one last change to the Java side of TclBlend to make the Notifier class per Interp. The Notifier really should be per-thread, not per-Interp. There can be many interpreters in a single thread and they should all share the same notifier. I don't know if there's a easy way to get this effect in Java. --Scott The TclJava mailing list is sponsored by Scriptics Corporation. To subscribe:send mail to [EMAIL PROTECTED] with the word SUBSCRIBE as the subject. To unsubscribe: send mail to [EMAIL PROTECTED] with the word UNSUBSCRIBE as the subject. To send to the list, send email to '[EMAIL PROTECTED]'. An archive is available at http://www.mail-archive.com/tcljava@scriptics.com