On Jan 28, 2011, at 13:37 , Greg Roelofs wrote: > Keith Wiley wrote: > >> However, it is also not run in any concurrent fashion except >> w.r.t. to the JVM itself. For example, my map task doesn't >> make parallel calls through JNI to the native code on concurrent >> threads at the Java level, nor does the native code itself spawn >> any threads (like I said, it isn't even compiled with pthreads). > > Is speculative execution enabled?
I suppose I hadn't specifically disabled that, but two points: (1) Speculative execution would occur on a completely different node, so there definitely isn't any thread cross-talk (in memory). So long as they don't rely on reading/writing temp files from HDFS I don't see how they could have any effect on one another. (2) I am also getting seg faults when I run in noncluster standalone mode, which is a single nonspeculated thread......I presume. Can you explain your thoughts on speculative execution w.r.t. the problems I'm having? >> So, the question is, in the scenario I have described, is there >> any reason to suspect that the cause of my problems is some >> sort of thread trampling between the native code and something >> else in the surrounding environment (the JVM or something like >> that), especially in the context of the surrounding Hadoop >> infrastructure? It doesn't really make any sense to me, but >> I'm running out of ideas. > > I don't see any obvious possibilities except speculative execution, and > even that would depend on how the shared library was written. Does it > contain any global or static variables? If so, it's almost certainly > not thread-safe (unless, say, a global variable were basically write- > only and advisory only, e.g., used only in an error message or a summary > message at the end). Yes, not thread safe, but what difference could that make if I don't use the library in a multi-threaded fashion. One map task, one node, one Java thread calling JNI and using the native code? How do thread safety issues factor into this? I admit, it's my theory that threads might be involved somehow, but I don't understand how, I'm just shooting in the dark since I can't solve this problem any other way yet. Thanks for the input. Can you tell me what you're thinking w.r.t. speculative execution? I'll try it without, but I don't see how it could alter the standalone behavior. Cheers! ________________________________________________________________________________ Keith Wiley [email protected] www.keithwiley.com "Luminous beings are we, not this crude matter." -- Yoda ________________________________________________________________________________
