On Wed, Mar 10, 2010 at 10:13 PM, Dewey Gaedcke <de...@minggl.com> wrote: > Javier, > Thanks very much for your detailed reply. I think the first part of my > email with Andre was truncated, so you didn't have all the context for my > question. I've practiced for years as a MySQL scalability architect, so > I'm very familiar with the various options on the persistence side of the > problem…it's the in-memory Lua stuff that's a bit foreign to me. > Here was part of my original thread if that helps you think of other > recommendations for us.
I did read the original question, and I wasn't thinking about persistence (although it might be easier to just use a RDBMS and get it too). If I understand it correctly, you have a big bunch of data and you want to read it from more than one process at a time. Even if it might be easier to have all the data directly available on Lua tables, it just wasn't designed with concurrent access in mind. I don't know if you're familiar with embedding Lua in C programs. In short, there's a C function that creates a Lua 'space' where scripts are run. it manages the VM state and the whole data managed by Lua. One Lua space can only be executed by one thread at a time. there's no concurrent running in a single space (*). A C program can create several Lua spaces, and those can run separately, on different threads/processes. But there's no (simple) way for code in one space to access data held in other space. Therefore, if you want to have your 8mill-record data in one space, and have several CGIs acess it, you have to choose the execution model: - cooperative multitasking (Copas and similar) everything on a single Lua space; but no real concurrency. - any of several Lua concurrent extensions: each thread/process/task/lane works in a different Lua space. there's no shared data, use message passing to fetch the data from the space that has it. - external shared resource. typically a RDBMS (*) there's LuaThreads, that implements a global interpreter lock (GIL) (**), allowing you to have several threads on the same Lua space, but the GIL is so heavily contended that most of the time is spend locking and unlocking it instead of executing Lua code. (**) from what i remember, it wasn't so 'excluding' as most GILs out there; in theory it could concurrently execute Lua code, but any non-local variable access has to use it.... well, you know about scalability, so you see why it's not so hot anymore, all the cool kids are using multiple Lua states and message passing. -- Javier _______________________________________________ Kepler-Project mailing list Kepler-Project@lists.luaforge.net http://lists.luaforge.net/cgi-bin/mailman/listinfo/kepler-project http://www.keplerproject.org/
