Javier, Thanks very much for your detailed reply. I think the first part of my email with Andre was truncated, so you didn't have all the context for my question. I've practiced for years as a MySQL scalability architect, so I'm very familiar with the various options on the persistence side of the problem…it's the in-memory Lua stuff that's a bit foreign to me.
Here was part of my original thread if that helps you think of other recommendations for us. I have a large graph of about 8 million rows (and growing quickly) represented in two Lua tables (dictionaries, not arrays). Two tables (but same rows totaling about 1.5 gigs of RAM) because each table is keyed differently. (a batch Lua process is recalculating them both constantly) The graph is changing quite frequently and I want to let multiple simultaneous processes run analysis (read-only) algorithms on these two tables….each analysis can take up to 3-4 seconds so I can't use coroutines which would block for that period of time Given the volume of data, and the rate of change, it seems crazy (and expensive) to instantiate those tables again and again in each Lua (or Lanes, Rings, fCGI, etc) instance. Changes are flushed to disk (mysql) periodically, but load-time latency, DB load and RAM requirements will still be prohibitive if each web-service call needs it's own 1.5 gig copy. So instead, I'm wondering if some other tool/technique will allow me to achieve read-only access to those tables from other parallel processes? If not, is there another good way to do this without forcing each of my read processes (web service threads) to consume that much ram and be offline every 30 minutes while they repetitively reload?? Thanks for any help or advice you can provide. Dewey On Mar 10, 2010, at 8:17 PM, Javier Guerra Giraldez wrote: > On Wed, Mar 10, 2010 at 8:37 PM, Dewey Gaedcke <[email protected]> wrote: >> So where I'm still lost is that I don't understand how Copas, Xavante, >> FastCGI or any other technology is able to read (in parallel) a Lua table >> that exists in my dedicated persistent Lua memory process. > > although Copas looks like a multitasking scheduler; it's in fact just > a wrapper around select() that picks the right coroutine to execute. > So, all the Copas 'threads' are coroutines on the same Lua space. > There's no parallel access. > > You have several different alternatives to your problem: > > - Use Copas or something similar. here it's all on one Lua space, so > there's no problem accessing the data from any 'thread'. it's easy to > write TCP servers that access the common resource. > > - Use any of different Lua multithreading systems. Like LuaLanes, or > luaproc (and many more). These handle several separate Lua spaces > running on parallel;with several different interprocess communication > facilities. this wouldn't let you directly access the data from any > other thread; but it's easy to write a 'fetch' function that sends > some parameters to a 'server' which does the query and returns the > requested data. IOW, the thread that holds the data works like a kind > of database server for the other threads. usually the client fetch > functions could be very short, less than 5 lines each > > - put the data on some external resource. like a database (SQLite > could be perfect, MySQL if you do a lot of reads and few writes, Tokyo > Tyrant if it's best described as a key/value store, memcached, > CouchDB, MongoDB, redis, etc.) > > > -- > Javier > > _______________________________________________ > Kepler-Project mailing list > [email protected] > http://lists.luaforge.net/cgi-bin/mailman/listinfo/kepler-project > http://www.keplerproject.org/
_______________________________________________ Kepler-Project mailing list [email protected] http://lists.luaforge.net/cgi-bin/mailman/listinfo/kepler-project http://www.keplerproject.org/
