Alright. Just took a look at my old code base. It seems like I tried to hack the async stuff into the methods that calls networking (in the transports) and used some threading to do it. It's ugly and it pretty much breaks everything, which is probably why I abandoned it at the time.
Just want to gather some opinion as to how to do the async stuff correctly. Is it better to have async networking calls for all operations? This means we can no longer do things like bucket.get(key) and expect to return an object, as we need to pass it a call back or do some other tricks like that. Also, what about threads vs greenlets? Python people tends to hate Thread, so I'm thinking greenlet maybe better suited for our purposes since it's just an IO call at the end of the day. Shuhao On Tue, Jul 24, 2012 at 6:40 PM, Philipp Borgers <[email protected]> wrote: > On Tue, 2012-07-24 at 17:00 -0400, Shuhao Wu wrote: >> On Tue, Jul 24, 2012 at 11:27 AM, Jeremy Thurgood <[email protected]> wrote: >> > On 24 July 2012 15:53, Sean Cribbs <[email protected]> wrote: >> >> > >> > So, on to my grandiose plans for the future of the Python Riak client. >> > We're currently using a combination of this client and Riakasaurus, >> > which is a Twisted-based client library that we maintain. Riakasaurus >> > is based (indirectly) on a previous version of the Python client, but >> > is different enough that it's impossible to pull in upstream changes >> > directly. The main reason for the divergence is that the Python client >> > assumes blocking network I/O operations that will return when they >> > have the required data, while Twisted uses an asynchronous >> > callback-based I/O mechanism. >> > >> > I think it would be possible to restructure the Python client somewhat >> > to decouple the higher-level operations from the lower-level network >> > operations in a way that would let us switch between Twisted-style >> > async I/O and more traditional blocking I/O. This would probably be a >> > fairly invasive change in some ways, but shouldn't have any noticeable >> > impact on the external API. I'm hoping it'll make the >> > ConnectionManager replacement cleaner and more flexible, but I don't >> > have a good enough mental model of the system to be sure. >> >> Actually. I already have a prototype of just a separation between the >> higher level and lower level API. I'm planning to add async IO (done >> in some local branch a couple months ago, but very crude) but haven't >> figured out a "good" way of doing it (callbacks, twisted style defer. >> Greenlet vs threads. etc.) just yet. > > I'm highly interested in the async stuff! Would be great if you could > share your local branch or ideas. > >> >> In addition I also added sibling handlings, which I believe is >> missing/lacking in the riak-python-client, which is a center part of >> my higher level design. >> >> If you want to take a look and see if this is something you think is >> what you mean, here's a link: >> https://github.com/ultimatebuster/riak-python-client2 >> >> It's also a prototyping repo for me in terms of what to try to bring >> to the official clients via pull requests. >> >> > As the person proposing this, I'm happy to put some effort into >> > hacking up a prototype to see if this is actually a workable idea and >> > to do the bulk of the implementation work if it is -- it'll save on a >> > bunch of Riakasaurus maintenance in the future if we can pull it off. >> > I'm proposing some pretty drastic surgery to the existing codebase, so >> > I'd like to know if other people (especially Sean) think it's a good >> > idea before I sink too much time into it. >> > >> > Thanks, >> > --J >> > >> > _______________________________________________ >> > riak-users mailing list >> > [email protected] >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
