I guess I got off track from my original subject line - once I started writing I realized read performance wasn't a LevelDB issue (I originally thought that maybe it was) but that the bottleneck must be our utilization of the API...
On Jul 26, 2012, at 5:18 PM, Parnell Springmeyer wrote: > I'm using Riak in a 5 node cluster with LevelDB for the backend (we store A > LOT of archivable data) on FreeBSD. > > The data is mapped out as follows: I have a set of database objects that are > closely linked to user accounts - I needed to be able to compose complex > queries on these objects including joins with the user data; so it made sense > to keep those objects in MySQL. > > I have software that takes those database objects and produces DAILY stats > for each object (so we have months/years of data for each database object). > These stats are what we store in Riak. > > Now, that application also updates the MySQL database object with the key > under which the stat object is stored in Riak for quick and easy compiling of > the "latest" data (since it's just a GET operation and not a M/R job). > > Mashing up this data for small sets of MySQL database objects is quick and > painless. But once it starts approaching > 1000 objects I notice it slows to > a crawl and i notice Riak being pegged pretty hard (IOW it is Riak's > response). > > Now; here's the issue: with my web application I haven't figured out how to > use the RiakPBC connector - so we are going through the HTTP API. I have a > feeling this is where that bottle neck is occurring. > > Why do you ask? Because our Python web app is multi-threaded and the PBC > sockets don't play nice here. I'm not finding my experiments to solve this > very successful. So I wanted to ask the greater community if anyone HAS or is > willing to HELP me solve it! > > Thanks :)
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
