[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-09 Thread Mo DeJong 

On Wed, 9 Aug 2000, Scott Stanton wrote:

 Mo DeJong said:
   It sure would make life easier if we just required Tcl
   threads to use Tcl Blend. The notifier thing would go
   away because we could count on threads being there.
 
 I'm all in favor of this.  As the threaded version of Tcl becomes more 
 stable, I'd like to see it become the default, anyway.
 
 --Scott

Lets see if we can list the pros and cons before we make a final decision.

Pro:
  Tcl Blend would work, we would avoid strange edge cases.
  We would not need the notifier hacks

Con:
  Non-thread safe extensions may have problems under Unix.
  Need to detect if Tcl is compiled with thread support. (not too hard)

I am sure I have missed a few.

Mo DeJong
Red Hat Inc


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-09 Thread Jiang Wu 

 -Original Message-
 From: Mo DeJong [mailto:[EMAIL PROTECTED]]
 Sent: Wednesday, August 09, 2000 1:34 PM
 To: [EMAIL PROTECTED]
 Subject: [Tcl Java] Re: TclBlend Initialization Mutex

 Con:
   Non-thread safe extensions may have problems under Unix.

It seems that there are at least two reasons why non-thread safe extensions
can have problems in a threaded process like Tcl or JVM.

1. access system level globals

   Hopefully, this can be fixed by compiling with -D_REENTRANT.

2. access Tcl globals

   What to do?

Right now, I am experimenting with using -D_REENTRANT (but without
-DTCL_THREADS) to compile a non-threaded Tcl.  The Solaris problem did go
away using this special Tcl compilation.

-- Jiang Wu
   [EMAIL PROTECTED]
  


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-09 Thread Mo DeJong 

On Wed, 9 Aug 2000, Jiang Wu wrote:

  -Original Message-
  From: Mo DeJong [mailto:[EMAIL PROTECTED]]
  Sent: Wednesday, August 09, 2000 1:34 PM
  To: [EMAIL PROTECTED]
  Subject: [Tcl Java] Re: TclBlend Initialization Mutex
 
  Con:
Non-thread safe extensions may have problems under Unix.
 
 It seems that there are at least two reasons why non-thread safe extensions
 can have problems in a threaded process like Tcl or JVM.
 
 1. access system level globals
 
Hopefully, this can be fixed by compiling with -D_REENTRANT.
 
 2. access Tcl globals
 
What to do?
 
 Right now, I am experimenting with using -D_REENTRANT (but without
 -DTCL_THREADS) to compile a non-threaded Tcl.  The Solaris problem did go
 away using this special Tcl compilation.

But you would need to recompile Tcl and all the extensions
with -D_REENTRANT. While I would agree that this is a neat
hack, I am not sure it is a good solution. Folks that
use a pre-compiled version will not be able to use this.
There is lots of history that suggests that core patches
are not a good solution (Itcl). We may end up having
to do away with pre-compiled versions of Tcl Blend all
together, just so that people do not make this mistake.

Mo DeJong
Red Hat Inc


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-09 Thread Jiang Wu 

 -Original Message-
 From: Mo DeJong [mailto:[EMAIL PROTECTED]]
 Sent: Wednesday, August 09, 2000 4:14 PM
 To: [EMAIL PROTECTED]
 Subject: [Tcl Java] Re: TclBlend Initialization Mutex
 
  Having to keep two separate versions of Tcl around is just 
 not convenient.
  I am open to any solutions that can make these extensions 
 work well with
  Java and Tcl.
 
 They need to be fixed. A threaded version of Tcl and a stub enabled
 extension will need to be the "minimum requirement" if you want to
 use Tcl Blend.

Yes, they may be fixed in the future.  Though I can't stop working waiting
for them to be fixed.  So I have to work with what I can get today.

  This is an unfortunate fact for UNIX due to its history and vendor
  variations.  Not having pre-compiled versions do not solve 
 the real problem.
  It only pushes the problem to the user.  That may be the 
 only solution, but
  it is very good in my mind.
 
 Well, the -D_REENTRANT thing is going to require a recompile of
 the extension, so I don't see how it is such a big deal.

The point I was making is that having everyone compiling
Tcl/TclBlend/extensions is big waste of time.  Ideally, people can download
and use so that they can focus on developing solutions using the tools
rather than worrying about compiling the tools.  But I guess that is as much
a wishful thinking as hoping that the extensions will be fixed to be
thread-safe.

-- Jiang Wu
   [EMAIL PROTECTED]



The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-09 Thread Jiang Wu 

 -Original Message-
 From: Mo DeJong [mailto:[EMAIL PROTECTED]]
 Sent: Wednesday, August 09, 2000 5:14 PM
 To: [EMAIL PROTECTED]
 Subject: [Tcl Java] Re: TclBlend Initialization Mutex

  Yes, they may be fixed in the future.  Though I can't stop 
 working waiting
  for them to be fixed.  So I have to work with what I can get today.
 
 Which means getting them working in threaded Tcl, right?

Not always.  The goal is to find tools and components that help with the
fast development of products for our customers.  Sometimes, we rather
rewrite a piece of functionality from scratch, which makes it proprietary.
We try to use as much off the shelf components as possible.  But if the
extension doesn't work off the shelf, then it is very hard to justify using
it at all.  We are not a Tcl only environment, if Tcl extensions don't work
well, we start assimilating other technologies. (We are Borg ... :-)

 Binary downloads just don't work. I tired over an over again
 during the 1.2 process. People would complain that they did
 not want to compile, so I put up a binary version. Shock
 and horrors, people could not get that to work either.
 A binary has all sorts of problems, mismatched versions
 of Tcl, the JVM, wrong env settings, and so on.

I certainly have a lot of sympathy for your frustration.

 I think the best approach is to just lock down Tcl Blend
 so that it only builds with a Thread enabled Tcl. People
 are going to need to download an build Tcl and Tcl Blend,
 there is just no way around it. It would be better to
 just work on the "Tcl SDK" idea, a common build environment
 would make things a lot easier.

Let's say we all use "Tcl SDK".  I get it and compile my own Tcl.  Ship the
binaries to my customers.  They complain about things not working.  The
problem has not been solved by the SDK.

There is a solution to the binary problem because many commercial companies
do distribute binary programs and libraries.  I guess the amount of
resources involved in verifying these binaries is beyond the capability of
one person with just a few pieces of equipment.  It is certainly not right
to blame any individual for non-working binary distributions of public
domain software.

Getting everyone to compile Tcl is not the right solution, but I suppose it
is the cheapest solution and takes the heat off the people who already
contributed a lot to the product.  Maybe a company can provide binary
distribution service and charge $99 per copy to cover the cost of the
testing, follow the commercial Linux model.

-- Jiang Wu
   [EMAIL PROTECTED]


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-08 Thread Scott Stanton 


Jiang Wu said:
  1. In UNIX, TclBlend uses a pipe with non-blocking IO to wake up the Tcl
  event loop.  If there are multiple event loops in several threads, will 
a
  single pipe based IO trigger all event loops to wake up?

In the threaded version of Tcl, there are two distinct types of event 
loops.  There is a single globally shared select loop 
(NotifierThreadProc).  This thread performs the select() system call on 
behalf of all threads in the system.  It has a single pipe that it 
includes in the select mask so other threads can wake the select thread up 
to force it to update its select mask.  Whenever it detects an event, it 
notifies only the threads that have specified interest in that file 
descriptor.  This typically only wakes up one thread since most of the 
time file descriptors are only owned by one thread.

The second type of loop is the notifier loop created by each thread that 
waits for events.  Each thread that uses the Tcl notifier APIs creates a 
separate event queue.  If the event queue becomes empty, the notifier 
thread blocks waiting for notification from the select loop (or a timeout).

So, with TclBlend, there should be one notifier loop per thread that 
contains an interpreter.  There should be exactly one select loop for the 
entire process.

  2. In Windows, a Win32 native window event is used to call 
Tcl_ServiceAll()
  to trigger the Tcl event loop.  The same question applies, does
  Tcl_ServiceAll() trigger all Tcl event loops to wake up?

In Windows, there is only one type of loop.  A separate Win32 window event 
loop is used for each thread.  The Tcl_ServiceAll() routine only affects a 
single thread's event queue.
 
  3. Currently, I am seeing a problem on Solaris, where the non-blocking 
pipe
  used in TclBlend is triggering a busy loop on reading from the pipe.  
The
  similar code in Tcl 8.3.1 for the event loop has a slightly different
  implementation from that used in TclBlend with a comment about "reading 
only
  1 byte at a time to avoid some race condition".

I'm not sure what would be causing a busy loop.  Can you tell what it is 
looping on?  The basic procedure should be that one or more threads write 
a byte onto the pipe.  The select thread will read one byte per select() 
iteration.  The only way it could loop is if some thread was repeatedly 
writing to the pipe or if the select thread failed to consume data from 
the pipe.

--Scott



The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-08 Thread Jiang Wu 

On Tue, 08 August 2000, Scott Stanton wrote:
 Notes on the Tcl event loop

Thanks for the information.  It seems that multi-threaded Tcl already implemented some 
cross-platform thread notification, maybe TclBlend should use something like 
Tcl_ThreadAlert() to wake up the event loop.  But this function is not available for 
non-threaded Tcl.  So for single-threaded Tcl, TclBlend still needs to use the current 
custom notification system.

   used in TclBlend is triggering a busy loop on reading from the pipe.  
 The
   similar code in Tcl 8.3.1 for the event loop has a slightly different
   implementation from that used in TclBlend with a comment about "reading 
 only
   1 byte at a time to avoid some race condition".
 
 I'm not sure what would be causing a busy loop.  Can you tell what it is 
 looping on?  The basic procedure should be that one or more threads write 
 a byte onto the pipe.  The select thread will read one byte per select() 
 iteration.  The only way it could loop is if some thread was repeatedly 
 writing to the pipe or if the select thread failed to consume data from 
 the pipe.

After spending some hours in gdb tracing the program, it seems that the problem occurs 
in tclUnixChan.c:FileInputProc() line 380:  *errorCodePtr = errno;

1 byte is written to a non-blocking Tcl file channel.  Then 1 byte is successfully 
read from the channel.  Then Tcl calls FileInputProc() again to read more.  There is 
no more data in the channel, so the call at tclUnixChan.c, line 376

bytesRead = read(fsPtr-fd, buf, (size_t) toRead);

returns -1.  However, for some reason, the system variable "errno" is always 0, not 
EAGAIN as expected by the caller function.  This causes the caller function 
(tclIO.c:GetInput(), line 4865) to loop forever calling FileInputProc() over and over 
again.

I have not determined what caused the problem with "errno".  The Tcl library is loaded 
into a JVM at this point.  So the process is running in multi-threaded mode.  The Tcl 
library used is non-threaded, version 8.3.1.  Can there be something bad about using 
"errno" from multiple threads?

Another strange thing if I run the same Java program on a smaller example, then it 
seems to work OK.  When the error does happen, there are many threads in the process.

-- Jiang Wu
   [EMAIL PROTECTED]


--
[EMAIL PROTECTED] is brought to you by 
the Stanford Alumni Association and Critical Path.


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-08 Thread Mo DeJong 

On 8 Aug 2000, Jiang Wu wrote:

 After spending some hours in gdb tracing the program, it seems that the problem 
occurs in tclUnixChan.c:FileInputProc() line 380:  *errorCodePtr = errno;
 
 1 byte is written to a non-blocking Tcl file channel.  Then 1 byte is successfully 
read from the channel.  Then Tcl calls FileInputProc() again to read more.  There is 
no more data in the channel, so the call at tclUnixChan.c, line 376
 
 bytesRead = read(fsPtr-fd, buf, (size_t) toRead);
 
 returns -1.  However, for some reason, the system variable "errno" is always 0, not 
EAGAIN as expected by the caller function.  This causes the caller function 
(tclIO.c:GetInput(), line 4865) to loop forever calling FileInputProc() over and over 
again.
 
 I have not determined what caused the problem with "errno".
 The Tcl library is loaded into a JVM at this point.  So the
 process is running in multi-threaded mode.  The Tcl library
 used is non-threaded, version 8.3.1.  Can there be something
 bad about using "errno" from multiple threads?

Under Solaris, errno becomes a function that returns a thread
local variable when you compile with -D_REENTRANT. You compiled
this version of Tcl with threads support, right?

 Another strange thing if I run the same Java program on a smaller
 example, then it seems to work OK.  When the error does happen,
 there are many threads in the process.

Sounds like a race condition dealing with ERRNO. I wonder if
Tcl is not getting compiled without -D_REENTRANT or something.
That seems unlikely. Tcl Blend just uses the CFLAGS set
by Tcl.

Mo DeJong
Red Hat Inc


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-08 Thread Jiang Wu 

 -Original Message-
 From: Mo DeJong [mailto:[EMAIL PROTECTED]]
 
  used is non-threaded, version 8.3.1.  Can there be something
  bad about using "errno" from multiple threads?
 
 Under Solaris, errno becomes a function that returns a thread
 local variable when you compile with -D_REENTRANT. You compiled
 this version of Tcl with threads support, right?

No :(  I am using a regular non-thread Tcl library.  Can I compile with
-D_RENTRANT, but without threads?  We have this requirement of being able to
use Tcl extensions such as Scotty.  So we are not using threaded Tcl.

Darn!  With problems like this, does it mean that there is no safe way of
using a non-threaded Tcl in a JVM because the JVM is threaded by default.
The JVM is going to allocate one thread to execute non-threaded Tcl code.
Tcl tries to access system globals.  JVM also tries to access the same
system globals.

-- Jiang Wu
   [EMAIL PROTECTED]


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-08 Thread Mo DeJong 

On Tue, 8 Aug 2000, Jiang Wu wrote:

 Darn!  With problems like this, does it mean that there is no safe way of
 using a non-threaded Tcl in a JVM because the JVM is threaded by default.
 The JVM is going to allocate one thread to execute non-threaded Tcl code.
 Tcl tries to access system globals.  JVM also tries to access the same
 system globals.

That does seem like a problem. If we require "special hacks" on
Tcl itself, then there is no point in trying to support a
pre-compiled version of Tcl Blend that gets loaded into Tcl.
Users will just have to download and build a threaded version
of Tcl or use our special hacks. A "core patch" seems like
a bad idea if you ask me.

It sure would make life easier if we just required Tcl
threads to use Tcl Blend. The notifier thing would go
away because we could count on threads being there.

Mo DeJong
Red Hat Inc


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-08 Thread Jiang Wu 

I looked at the Window's version of Tcl.  There is no special system level
compiler flag to compile threaded versus non-threaded.  So it seems that
non-threaded Tcl on Windows may work fine with Java.

I have to agree that requiring threaded Tcl may be the only option for UNIX.
If non-threaded Tcl can interact badly with Java, so can these Tcl
extensions that are not thread enabled.  If that is the case, even if we can
make Tcl work with the JVM, there is no guarantee that the extensions can
work.

-- Jiang Wu
   [EMAIL PROTECTED]

 From: Mo DeJong [EMAIL PROTECTED]
 Date: Tue, 8 Aug 2000 21:15:22 -0700 (PDT)
 To: [EMAIL PROTECTED]
 Subject: [Tcl Java] Re: TclBlend Initialization Mutex
 Resent-From: [EMAIL PROTECTED]
 Resent-Date: Tue, 8 Aug 2000 21:15:52 -0700 (PDT)
 
 On Tue, 8 Aug 2000, Jiang Wu wrote:
 
 Darn!  With problems like this, does it mean that there is no safe way of
 using a non-threaded Tcl in a JVM because the JVM is threaded by default.
 The JVM is going to allocate one thread to execute non-threaded Tcl code.
 Tcl tries to access system globals.  JVM also tries to access the same
 system globals.
 
 That does seem like a problem. If we require "special hacks" on
 Tcl itself, then there is no point in trying to support a
 pre-compiled version of Tcl Blend that gets loaded into Tcl.
 Users will just have to download and build a threaded version
 of Tcl or use our special hacks. A "core patch" seems like
 a bad idea if you ask me.
 
 It sure would make life easier if we just required Tcl
 threads to use Tcl Blend. The notifier thing would go
 away because we could count on threads being there.
 
 Mo DeJong
 Red Hat Inc
 
 
 The TclJava mailing list is sponsored by Scriptics Corporation.
 To subscribe:send mail to [EMAIL PROTECTED]
 with the word SUBSCRIBE as the subject.
 To unsubscribe:  send mail to [EMAIL PROTECTED]
 with the word UNSUBSCRIBE as the subject.
 To send to the list, send email to '[EMAIL PROTECTED]'.
 An archive is available at http://www.mail-archive.com/tcljava@scriptics.com
 


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com

[Tcl Java] Re: TclBlend Initialization Mutex

2000-08-07 Thread Jiang Wu 

Yes, you are correct.  I noticed this after doing more research into the
notifier and event loop implementation in Tcl.

Per thread event loop can be achieve in Java easily.  But there are some
outstanding questions:

1. In UNIX, TclBlend uses a pipe with non-blocking IO to wake up the Tcl
event loop.  If there are multiple event loops in several threads, will a
single pipe based IO trigger all event loops to wake up?

2. In Windows, a Win32 native window event is used to call Tcl_ServiceAll()
to trigger the Tcl event loop.  The same question applies, does
Tcl_ServiceAll() trigger all Tcl event loops to wake up?

3. Currently, I am seeing a problem on Solaris, where the non-blocking pipe
used in TclBlend is triggering a busy loop on reading from the pipe.  The
similar code in Tcl 8.3.1 for the event loop has a slightly different
implementation from that used in TclBlend with a comment about "reading only
1 byte at a time to avoid some race condition".

Your insights into these matter are greatly appreciated.

-- Jiang Wu
   [EMAIL PROTECTED]

 From: Scott Stanton [EMAIL PROTECTED]
 Date: Mon, 07 Aug 2000 11:07:26 -0700
 To: Jiang Wu [EMAIL PROTECTED]
 Cc: Mo DeJong [EMAIL PROTECTED], [EMAIL PROTECTED]
 Subject: Re: TclBlend Initialization Mutex
 
 
 Jiang Wu said:
 I added a global mutex to protect TclBlend initialization from mutiple
 threads.  The patch is sent to Mo for check in.  This mutex together
 with
 Mo's changes on removing the global lock and making JNIEnv thread local
 make
 TclBlend almost thread safe.  I say "almost" because we still need to
 make
 one last change to the Java side of TclBlend to make the Notifier class
 per
 Interp.
 
 The Notifier really should be per-thread, not per-Interp.  There can be
 many interpreters in a single thread and they should all share the same
 notifier.  I don't know if there's a easy way to get this effect in Java.
 
 --Scott
 
 


The TclJava mailing list is sponsored by Scriptics Corporation.
To subscribe:send mail to [EMAIL PROTECTED]  
 with the word SUBSCRIBE as the subject.
To unsubscribe:  send mail to [EMAIL PROTECTED] 
 with the word UNSUBSCRIBE as the subject.
To send to the list, send email to '[EMAIL PROTECTED]'. 
An archive is available at http://www.mail-archive.com/tcljava@scriptics.com