I had tried a similar approach but couldn't get it right. I tend to avoid curl, so here's how you could deal with streaming results in pure socket land:
http://www.control-alt-del.org/2011/09/09/streaming-results-from-riak-over-http-in-php/ As for the socket pool stuff, I've got mixed feelings about the idea. Sending many requests simultaneously involves keeping track of a lot more things that are in flight and tends to greatly increase the complexity of the application. I've often found parallelism to be implemented more easily by creating worker processes and queuing things via other mechanisms (eg: IPC, AMQP, etc...).That my 2 cents, YMMV Mark Steele, CISSP, CSM Bering Media Inc. On Fri, Dec 23, 2011 at 11:08 AM, John Loehrer <[email protected]>wrote: > For the getkeys method i experimented with passing in a callback handler > that will be passed each key as it is read. I set up a hook in the curl > write function to be able to grab the data as it streams instead of waiting > for the entire result set to come over the wire. I will probably port this > into the new transport library as well. > > https://github.com/gaiaops/riak-php-client/blob/pool/riak.php#L1032 > > We could use this approach for anything that is likely to send huge data > responses, like map reduce operations or getKeys. > > The other interesting thing i did with the http client was allow the http > request to be attached to a pool object that will issue many requests in > parallel and do a curl_mult_select on the attached resources to grab back > the data as it comes over the wire. Using this approach I could queue up > and run many hundreds of requests in parallel and get far better results > than any of the clients I've seen so far when many requests need to be made > in parallel. > > I could probably do the same for the protocol buffer transports as well, > since writing a socket select pool isn't terribly difficult. Just want to > make sure there is an actual use-case. One use-case I can see so far is > when you need to bulk add or delete objects for migration of data. > > > ~ John > > > ----- Original Message ----- > From: "Mark Steele" <[email protected]> > To: "John Loehrer" <[email protected]> > Cc: "Joseph Lambert" <[email protected]>, "riak-users Users" < > [email protected]> > Sent: Friday, December 23, 2011 7:55:56 AM > Subject: Re: php 5.3 protocol buffer client > > In case it's helpful: > http://www.control-alt-del.org/2011/09/06/building-a-better-riak-client-for-php/ > > > Write up a did a while back on how to improve/re-write the php client for > PB. We're using something very similar to what I wrote up (with a custom C > protobuf pecl extension) with good results. > > > There's probably loads of room for improvement there, I believe we're > seeing much higher throughput. > > > My biggest gripes with the stock client were the lack of streaming support > for results and the class structure and obviously the HTTP interface (with > no keepalives!). I highly recommend using something like an iterator to > handle streaming results sets to not have to buffer everything in RAM. Also > one oft overlooked performance booster is making sure the client isn't > retrieving the body on puts if you want to do conflict resolution on reads > (returnbody should be false!), or don't care about siblings. > > > Cheers, > > Mark Steele, CISSP, CSM > Bering Media Inc. > > > > > On Fri, Dec 23, 2011 at 10:31 AM, John Loehrer < [email protected]> > wrote: > > > I did some xdebug code profiling on the http client and found that parsing > http headers was consuming 15% of the time. swapping that one part out with > the pecl-http function if it is available speeds it up to as fast as > drslump: > > http://php.net/manual/en/function.http-parse-headers.php > > writes: 254 > reads: 357 > > This illustrates how a one line change in the code can make a huge > difference. I will continue to profile the different transport methods to > see what can be done. > > > ~ John > > ----- Original Message ----- > From: "John Loehrer" < [email protected] > > > To: "Joseph Lambert" < [email protected] > > Cc: "riak-users Users" < [email protected] > > > > > Sent: Friday, December 23, 2011 6:13:18 AM > Subject: Re: php 5.3 protocol buffer client > > I tested each transport method against each other. Each writes 10,000 > objects into riak, then reads those same objects back in serial order. The > benchmark doesn't really measure riak performance, just useful for > comparing the difference between each transport method since the backend is > the same for all the clients. Http is the slowest and the protobuf > transport is more than 2x faster on reads. > > > per sec: w - r > ====================== > http: 233 - 303 > drslump: 250 - 374 > pb: 310 - 444 > protobuf: 391 - 654 > > > ~ John > > > "riak-users Users" < [email protected] > > Sent: Thursday, December 22, 2011 9:17:33 PM > Subject: Re: php 5.3 protocol buffer client > > I have compared drslump vs 2 others, and found the fastest so far is the > one based off of: > > https://github.com/bramp/protoc-gen-php > > By a factor of 2x. So far DrSlump is the slowest of the three. The > refactored code based off of pb4php comes in at a close second: > > http://code.google.com/p/pb4php/ > > My refactoring didn't make it any faster or slower, just made the api > cleaner and more to my liking. Still working on it, and will post both > parts as standalone components on github once i am done refactoring. > > > ~ John > > ----- Original Message ----- > From: "Joseph Lambert" < [email protected] > > To: "John Loehrer" < [email protected] > > Cc: "riak-users Users" < [email protected] > > Sent: Thursday, December 22, 2011 5:52:19 PM > Subject: Re: php 5.3 protocol buffer client > > Nice! > > > I had actually done the same thing using DrSlump's protobuf library and > found that it worked quite well and was about 40% faster than HTTP. > > > I'm curious what your results might be performance-wise comparing what you > have written to DrSlumps (I noticed there is a commit comment about > comparing to DrSlumps). > > - Joe Lambert > > > > On Fri, Dec 23, 2011 at 5:39 AM, John Loehrer < [email protected] > > wrote: > > > Sorry if it wasn't clear, all the changes are in the transport branch: > > https://github.com/gaiaops/riak-php-client/tree/transport > > It can be merged into master once we are sure it is production ready. > > ~ John > > > > ----- Original Message ----- > From: "John Loehrer" < [email protected] > > To: "riak-users Users" < [email protected] > > Sent: Thursday, December 22, 2011 1:34:36 PM > Subject: php 5.3 protocol buffer client > > Past few days I have been experimenting with a protocol buffers client for > riak in php 5.3. > > https://github.com/gaiaops/riak-php-client > https://github.com/basho/riak-php-client/pull/20 > > > It is a port of the existing php client and should be backward compatible > with the existing code. Everything seems to work as expected and passes the > unit tests. > > Anyone willing to play with it in their test environments to help me find > bugs? Also, interested in getting help finishing it, as maintaining the > code is a lot of work. > > > ~ John > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
