On my 'newhttp' branch, the ConnectionManager class handles all connections to the server(s) for the transports (many connections to many servers). The transport can worry about what goes onto the wire, and the CM worries about the underlying connections.
Today, there are multiple transports, each attempting to manage connections in various ways, and frankly... doing a poor job of it. The HTTP transport opens a new connection for every request. To borrow a phrase from Scott Conant on 'Chopped', that "drives me to anger" :-) So yes, my changes are a big improvement in that we now have *persistent* connections to the server, with the commensurate performance improvement. The basic problem is that "RiakTransport" conflates wire formatting *and* connection management. I'm trying to separate those two, in order to improve the connections. Once they are separated, then connection policies (as you describe) will be much easier to implement. Your timeout work is great, and I completely agree with you: timeouts are a "must have" in a production environment. I don't see any problem adding timeouts; my comment on your github commit was more about 2.5 compatibility, than denial. And I consider it my problem to figure out how to make it work in a 2.5 environment. Cheers, -g On Thu, Sep 8, 2011 at 13:47, Brett Hoerner <br...@bretthoerner.com> wrote: > I own the pull request that adds (very basic) pooling logic to the > Riak client. I left Transports alone (each is a single connection) and > decided to have pools be another class you pick just like transport. > This allows you instantly use any pooling logic (never remove down > servers, delete down servers, round robin, whatever) with *any* > transport class: Http, Http keep-alive, PBC, CachedPBC, etc. > > I've added pooling and timeouts and use both in production because I'm > honestly not sure how you use Riak in a highly available way without > them... so we need to make sure they both work well/easily under this > new scheme. > > Do your changes make anything new possible or are they just about cleanup? > > > > On Thu, Sep 8, 2011 at 5:17 AM, Greg Stein <gst...@gmail.com> wrote: >> Yup, agreed. I can certainly see use-cases where a client may have >> particular knowledge about data/server locality, and wants to open >> connections to *just* those servers. The transports that I am >> advocating open connections to servers described by one layer higher. >> That layer could say "all Riak servers in the cluster", or it could >> say "these three servers holding copies of <this> data". >> >> An auto-discovery layer (to add and remove) in-between the client and >> the transport would be handy. I can envision that, and my proposed >> code makes it possible, but (at my current stage) I don't have a use >> for it. I'd hope that others can build on my connection pooling work >> to see that through. >> >> Cheers, >> -g >> >> On Thu, Sep 8, 2011 at 07:49, Phil Stanhope <stanh...@gmail.com> wrote: >>> I like the idea of RiakTransport as you describe it. It opens the door to >>> other potential underlying transports and isolating a client from knowledge >>> of those (websockets, tcp, zeromq messaging come to mind). I'm not >>> suggesting that RIAK will ever support these transports by this comment, >>> however. >>> I also agree with the need to have RiakRingAwareTransport as an additional >>> layer that *might* be used by a client. There may be valid reasons why a >>> client might want to force particular traffic onto a subset of the ring >>> (e.g. M/R config, Search Config, forcing read/write traffic onto different >>> nodes, etc). Again, I'm not suggesting that using a subset of the ring for >>> particular operations is the best practice. But it may be necessary to do so >>> in order to validate and do certain types of testing to prove or disprove >>> certain access patterns. >>> -phil >>> >>> On Thu, Sep 8, 2011 at 7:31 AM, Greg Stein <gst...@gmail.com> wrote: >>>> >>>> Hey, all... >>>> >>>> After a couple comments on my recent work, and some archaeology on the >>>> Python/Riak work over the past month... I've realized that I might >>>> have a very different view of RiakTransport compared to what I'm >>>> seeing in the current work. I figured it best to bring that to the >>>> forefront and discuss: >>>> >>>> In my view, RiakTransport is used by RiakClient (and others) to "talk >>>> to the Riak server". >>>> >>>> Some of the current work, and some proposed pull requests, seem to >>>> take the position that a RiakTransport is "one connection to a server, >>>> and the client should manage those". >>>> >>>> Needless to say, I'm in favor of my own position :-) ... I think it is >>>> best to transfer *all* responsibility for talking to the server(s) to >>>> the transport layer. I really don't think the client/bucket/object >>>> layers should know anything about talking to the server(s). I'd like >>>> to see the transport layer be told about all server(s) available, and >>>> then it Just Works. >>>> >>>> I'm still a newbie with all this code, and need to keep plugging away >>>> at the higher levels of functionality and compensation for problems. >>>> I'd like to build up some code that contacts *one* given server, asks >>>> for all of the ring servers, and then opens connections to those >>>> servers. And then, it should (automatically) maintain client >>>> connections based on what is happening with the Riak cluster. The >>>> current (proposed) code manages connections to N servers, but has no >>>> automatic add/remove based on changes in the cluster status. I think >>>> this happens at a layer *just* above the actual transport. ie. >>>> something tracks the changes in the ring status and its servers, and >>>> transmits those changes into the transport layer, which alters its >>>> communication with that cluster (regardless of whether that >>>> communication is via HTTP or protobuf). >>>> >>>> >>>> Okee doke. That's the end of my brain dump and future thoughts on the >>>> transport and communication layer. I'd really like some feedback, >>>> review, and thoughts. >>>> >>>> Thanks! >>>> -g >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
