My main concern about poolboy is the fact that it wraps another gen_server process around the actual worker process. This makes you send an extra message every time you query the database, which is quite an overhead when you have a lot of concurrent users. Since you guys are fighting for performance, that could be an issue for you.
At Mochi, we have a client of our own, which is somewhat close to gen_pool behaviour I implemented and open-sourced. This client incomplete in many ways, and uses parametrized modules, which a lot of people tend to dislike, but it is very fast, stable and production-proven. Nevertheless, I would really like to hear about Basho's opinion on poolboy too. Not only they included it into Riak distribution, but they also spent a handful of effort to test it through and fix a number of quite serious errors. On Mon, Sep 24, 2012 at 12:56 PM, Yuri Lukyanov <[email protected]> wrote: > Hi Dmitry, ;) > > Yes, thank you. I'm also thinking about this approach. I just wanted > to know how people use poolboy for that (It seems they do). > But even considering your case, imagine that you have quite a big > cluster and quite a big amount of rps. > In this case you may want to have more than one load balancer to > spread the load and be more tolerant to LB failures. > Exactly the same situation arises. > It's not a problem to implement a pooler in this case on my own. I'm > just asking about using poolboy for that. > > On Mon, Sep 24, 2012 at 12:08 PM, Dmitry Demeshchuk > <[email protected]> wrote: > > Why not use load balancer on top of Riak cluster, independently from > > clients? If load balancing is sophisticated enough, it can even do a > greater > > job than just uniformly spreading requests between machines. > > > > Consider the following situation. You happen to send several requests > for a > > huge piece of data (each can be just a simple get, or even a complicated > > map-reduce query), but the rest of the nodes, or just some of them, > receive > > much smaller queries. > > > > What an external load balancer can do is gathering some basic stats from > > each machine (like load average, memory and IO consumption, number of > rps to > > Riak and so on) and balancing the requests according to this data. > Another > > benefit from this approach is that you can easily add any other Riak > client > > (Python, Ruby, whatever else) on top of it, and load will be still nicely > > distributed between the machines. > > > > On Mon, Sep 24, 2012 at 11:38 AM, Yuri Lukyanov <[email protected]> > wrote: > >> > >> Hi, > >> > >> I'm trying to use poolboy to organize a pool of connections to riak > >> nodes with protobuf client. > >> > >> This has come from the suggestion here: > >> > >> > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-September/009346.html > >> > >> The quote: > >> "Using Poolboy is convenient because it comes as a dependency of > >> riak_core. > >> > >> If you use Poolboy, you'll have to modify riakc_pb_socket slightly to > >> account for the way poolboy initializes connections (add a > start_link/1), > >> or create a simple module to pass the initialization from poolboy to > >> riakc_pb_socket." > >> > >> What I'm pondering about is what would be a convenient way to start > >> riakc_pb_socket clients in a pool. > >> For example, if I have 10 riak nodes in a cluster and want to open a > >> single connection to each of them (using riakc_pb_socket), I need one > >> poolboy worker per one riakc_pb_socket instance. In this case, I need > >> a host/port of a single riak node for each > >> riakc_pb_socket:start_link(). So I use WorkerArgs in > >> poolboy:start_link/3 to pass a list of riak nodes. And here comes a > >> confusion. poolboy passes WorkerArgs to each of its workers, so each > >> worker gets the list of riak nodes. Now I need to somehow choose a > >> node, maybe like proposed here: > >> https://github.com/basho/riak-erlang-client/pull/45/files > >> But I don't think it's a good solution. > >> > >> Maybe my idea of using riakc_pb_socket together with poolboy is not > >> convenient? Maybe I should start a pool per each riak node? Maybe > >> poolboy is not that convenient in this case as it seems? > >> > >> Would you share your experiences and thoughts on the subject? > >> > >> Thank you. > >> > >> _______________________________________________ > >> riak-users mailing list > >> [email protected] > >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > > > -- > > Best regards, > > Dmitry Demeshchuk > -- Best regards, Dmitry Demeshchuk
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
