So when I try to use pagination, it doesn't seem to be picking up my continuation. I'm having trouble parsing the json I get back using stream=true (and there is still a timeout) so I went to just using pagination. Perhaps I'm doing it wrong, (likely, it has been a long day) but riak seems to be ignoring my continuation:
(pardon the sanitization) curl 'http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=5' {"keys":["1","2","3","4","5"],"continuation":"g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=="} curl 'http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=5&continuation=g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ==' {"keys":["1","2","3","4","5"],"continuation":"g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=="} The same keys and continuation value are returned regardless of whether my request contains a continuation value. I've tried swapping the order of max_results and continuation without any luck. I also made sure that my continuation value was url encoded. Hopefully I'm not missing something obvious here. Well, come to think of it, hopefully I am missing something obvious! Sean On Jul 26, 2013, at 6:43 PM, Russell Brown <[email protected]> wrote: > For a work around you could use streaming and pagination. > > Request smaller pages of data (i.e. sub 60 seconds worth) and use streaming > to get the results to your client sooner. > > In HTTP this would look like > > http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=10000&stream=true > > your results will include a continuation like > > {"continuation":"g2gCYgAAFXttAAAABDU0OTk="} > > and you can use that to get the next N results. Breaking your query up that > way should duck the timeout. > > Furthermore, adding &stream=true will mean the first results is received very > rapidly. > > I don't think the Ruby client is up to date for the new 2i features, but you > could monkeypatch as before. > > Cheers > > Russell > > On 26 Jul 2013, at 19:00, Sean McKibben <[email protected]> wrote: > >> Thank you for looking in to this. This is a major problem for our production >> cluster, and we're in a bit of a bind right now trying to figure out a >> workaround in the interim. It sounds like maybe a mapreduce might handle the >> timeout properly, so hopefully we can do that in the meantime. >> If there is any way we can have a hotfix ASAP though, that would be >> preferable. Certainly would not be a problem for us to edit a value in the >> config file (and given the lack of support in the ruby client for the >> timeout setting, the ability to edit the global default would be preferred). >> In the ruby client i had to monkeypatch it like this to even submit the >> timeout value, which is not ideal: >> >> module Riak >> class Client >> class HTTPBackend >> def get_index(bucket, index, query) >> bucket = bucket.name if Bucket === bucket >> path = case query >> when Range >> raise ArgumentError, t('invalid_index_query', :value => >> query.inspect) unless String === query.begin || Integer === query.end >> index_range_path(bucket, index, query.begin, query.end) >> when String, Integer >> index_eq_path(bucket, index, query, 'timeout' => '260000') >> else >> raise ArgumentError, t('invalid_index_query', :value => >> query.inspect) >> end >> response = get(200, path) >> JSON.parse(response[:body])['keys'] >> end >> end >> end >> end >> >> Thanks for the update, >> Sean >> >> >> >> On Jul 26, 2013, at 4:49 PM, Russell Brown <[email protected]> wrote: >> >>> Hi Sean, >>> I'm very sorry to say that you've found a featurebug. >>> >>> There was a fix put in here https://github.com/basho/riak_core/pull/332 >>> >>> But that means that the default timeout of 60 seconds is now honoured. In >>> the past it was not. >>> >>> As far as I can see the 2i endpoint never accepted a timeout argument, and >>> it still does not. >>> >>> The fix would be to add the timeout to the 2i API endpoints, and I'll do >>> that straight away. >>> >>> In the meantime, I wonder if streaming the results would help, or if you'd >>> still hit the overall timeout? >>> >>> Very sorry that you've run into this. Let me know if streaming helps, I've >>> raised an issue here[1] if you want to track this bug >>> >>> Cheers >>> >>> Russell >>> >>> [1] https://github.com/basho/riak_kv/issues/610 >>> >>> >>> On 26 Jul 2013, at 17:59, Sean McKibben <[email protected]> wrote: >>> >>>> I should have mentioned that I also tried: >>>> curl -H "X-Riak-Timeout: 260000" >>>> "http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?timeout=260000"; >>>> -i >>>> but still receive the 500 error below exactly at the 60 second mark. Is >>>> this a bug? >>>> >>>> Secondary to getting this working at all, is this documented anywhere? and >>>> any way to set this timeout using the ruby riak client? >>>> >>>> Stream may well work, but I'm going to have to make a number of changes on >>>> the client side to deal with the results. >>>> >>>> Sean >>>> >>>> On Jul 26, 2013, at 3:53 PM, Brian Roach <[email protected]> wrote: >>>> >>>>> Sean - >>>>> >>>>> The timeout isn't via a header, it's a query param -> &timeout=xxxx >>>>> >>>>> You can also use stream=true to stream the results. >>>>> >>>>> - Roach >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On Jul 26, 2013, at 3:43 PM, Sean McKibben <[email protected]> wrote: >>>>> >>>>>> We just upgraded to 1.4 and are having a big problem with some of our >>>>>> larger 2i queries. We have a few key queries that takes longer than 60 >>>>>> seconds (usually about 110 seconds) to execute, but after going to 1.4 >>>>>> we can't seem to get around a 60 second timeout. >>>>>> >>>>>> I've tried: >>>>>> curl -H "X-Riak-Timeout: 260000" >>>>>> "http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?x-riak-timeout=260000"; >>>>>> -i >>>>>> >>>>>> But I always get >>>>>> HTTP/1.1 500 Internal Server Error >>>>>> Vary: Accept-Encoding >>>>>> Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact) >>>>>> Date: Fri, 26 Jul 2013 21:41:28 GMT >>>>>> Content-Type: text/html >>>>>> Content-Length: 265 >>>>>> Connection: close >>>>>> >>>>>> <html><head><title>500 Internal Server >>>>>> Error</title></head><body><h1>Internal Server Error</h1>The server >>>>>> encountered an error while processing this >>>>>> request:<br><pre>{error,{error,timeout}}</pre><P><HR><ADDRESS>mochiweb+webmachine >>>>>> web server</ADDRESS></body></html> >>>>>> >>>>>> Right at the 60 second mark. What can I set to give my secondary index >>>>>> queries more time?? >>>>>> >>>>>> This is causing major problems for us :( >>>>>> >>>>>> Sean >>>>>> _______________________________________________ >>>>>> riak-users mailing list >>>>>> [email protected] >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >> > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
