For the getkeys method i experimented with passing in a callback handler that will be passed each key as it is read. I set up a hook in the curl write function to be able to grab the data as it streams instead of waiting for the entire result set to come over the wire. I will probably port this into the new transport library as well.
https://github.com/gaiaops/riak-php-client/blob/pool/riak.php#L1032 We could use this approach for anything that is likely to send huge data responses, like map reduce operations or getKeys. The other interesting thing i did with the http client was allow the http request to be attached to a pool object that will issue many requests in parallel and do a curl_mult_select on the attached resources to grab back the data as it comes over the wire. Using this approach I could queue up and run many hundreds of requests in parallel and get far better results than any of the clients I've seen so far when many requests need to be made in parallel. I could probably do the same for the protocol buffer transports as well, since writing a socket select pool isn't terribly difficult. Just want to make sure there is an actual use-case. One use-case I can see so far is when you need to bulk add or delete objects for migration of data. ~ John ----- Original Message ----- From: "Mark Steele" <[email protected]> To: "John Loehrer" <[email protected]> Cc: "Joseph Lambert" <[email protected]>, "riak-users Users" <[email protected]> Sent: Friday, December 23, 2011 7:55:56 AM Subject: Re: php 5.3 protocol buffer client In case it's helpful: http://www.control-alt-del.org/2011/09/06/building-a-better-riak-client-for-php/ Write up a did a while back on how to improve/re-write the php client for PB. We're using something very similar to what I wrote up (with a custom C protobuf pecl extension) with good results. There's probably loads of room for improvement there, I believe we're seeing much higher throughput. My biggest gripes with the stock client were the lack of streaming support for results and the class structure and obviously the HTTP interface (with no keepalives!). I highly recommend using something like an iterator to handle streaming results sets to not have to buffer everything in RAM. Also one oft overlooked performance booster is making sure the client isn't retrieving the body on puts if you want to do conflict resolution on reads (returnbody should be false!), or don't care about siblings. Cheers, Mark Steele, CISSP, CSM Bering Media Inc. On Fri, Dec 23, 2011 at 10:31 AM, John Loehrer < [email protected] > wrote: I did some xdebug code profiling on the http client and found that parsing http headers was consuming 15% of the time. swapping that one part out with the pecl-http function if it is available speeds it up to as fast as drslump: http://php.net/manual/en/function.http-parse-headers.php writes: 254 reads: 357 This illustrates how a one line change in the code can make a huge difference. I will continue to profile the different transport methods to see what can be done. ~ John ----- Original Message ----- From: "John Loehrer" < [email protected] > To: "Joseph Lambert" < [email protected] > Cc: "riak-users Users" < [email protected] > Sent: Friday, December 23, 2011 6:13:18 AM Subject: Re: php 5.3 protocol buffer client I tested each transport method against each other. Each writes 10,000 objects into riak, then reads those same objects back in serial order. The benchmark doesn't really measure riak performance, just useful for comparing the difference between each transport method since the backend is the same for all the clients. Http is the slowest and the protobuf transport is more than 2x faster on reads. per sec: w - r ====================== http: 233 - 303 drslump: 250 - 374 pb: 310 - 444 protobuf: 391 - 654 ~ John "riak-users Users" < [email protected] > Sent: Thursday, December 22, 2011 9:17:33 PM Subject: Re: php 5.3 protocol buffer client I have compared drslump vs 2 others, and found the fastest so far is the one based off of: https://github.com/bramp/protoc-gen-php By a factor of 2x. So far DrSlump is the slowest of the three. The refactored code based off of pb4php comes in at a close second: http://code.google.com/p/pb4php/ My refactoring didn't make it any faster or slower, just made the api cleaner and more to my liking. Still working on it, and will post both parts as standalone components on github once i am done refactoring. ~ John ----- Original Message ----- From: "Joseph Lambert" < [email protected] > To: "John Loehrer" < [email protected] > Cc: "riak-users Users" < [email protected] > Sent: Thursday, December 22, 2011 5:52:19 PM Subject: Re: php 5.3 protocol buffer client Nice! I had actually done the same thing using DrSlump's protobuf library and found that it worked quite well and was about 40% faster than HTTP. I'm curious what your results might be performance-wise comparing what you have written to DrSlumps (I noticed there is a commit comment about comparing to DrSlumps). - Joe Lambert On Fri, Dec 23, 2011 at 5:39 AM, John Loehrer < [email protected] > wrote: Sorry if it wasn't clear, all the changes are in the transport branch: https://github.com/gaiaops/riak-php-client/tree/transport It can be merged into master once we are sure it is production ready. ~ John ----- Original Message ----- From: "John Loehrer" < [email protected] > To: "riak-users Users" < [email protected] > Sent: Thursday, December 22, 2011 1:34:36 PM Subject: php 5.3 protocol buffer client Past few days I have been experimenting with a protocol buffers client for riak in php 5.3. https://github.com/gaiaops/riak-php-client https://github.com/basho/riak-php-client/pull/20 It is a port of the existing php client and should be backward compatible with the existing code. Everything seems to work as expected and passes the unit tests. Anyone willing to play with it in their test environments to help me find bugs? Also, interested in getting help finishing it, as maintaining the code is a lot of work. ~ John _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
