On Fri, Aug 1, 2014 at 9:00 AM, Joseph Naegele <[email protected]> wrote:
> This is how Lua enables reentrancy and it's generally not considered a > nightmare. We are also talking about two different issues. One is allowing > Julia to run code in multiple threads using one runtime, GC, etc. which I'm > sure is relatively high priority for you guys. The other issue is of the > ability to run multiple Julia "instances" in one binary. From a high-level > perspective Julia operates just like an interpreter, but it currently > suffers from the same issue as Python in that you can only embed one > interpreter instance in a running process. Lua solves this by simply > requiring all interpreter state to be passed by the client code to all Lua > API functions. It's a pretty common paradigm. > Lua was largely designed for this use case – i.e. to be a small, simple, embeddable language – so it's not too surprising that it allows this. However, this approach does constrict the language runtime quite a bit. I'd be curious to know, for example, if LuaJIT also allows multiple independent instances in a single process. In any case, Julia isn't primarily designed for embedding and this is approach is unlikely to happen in Julia for a number of reasons. One issue is that LLVM is not reentrant. All the many Julia instances in a single process could lock on the shared LLVM library – we have to lock on code gen for threading anyway. But I suspect every switch would also require swapping out a lot of global LLVM state, which seems likely to be slow and very annoying. There's also the issue that we can and do turn Julia functions into C-callable function pointers that can be invoked from C as if they were C function pointers – this currently has zero overhead and if the Julia function is fast, then calling it from C will also be fast. If these require interpreter state, then that would need to be a function call argument to every C-callable function, which is at odds with many C APIs (although good libraries do allow for a void* data argument). Maybe this could be made to work, but my suspicion is that it would introduce too much overhead and destroy our current ability to do zero-cost two-way interop with C. I'm also not really convinced this is a terribly useful thing to do. If you want multiple independent Julia instances, why not have multiple processes?
