The solution turned out to be simple. The index of a view is a B-tree, so looking for range between two IDs is of logarithmic complexity, and that's ok, but skipping exact count of records requires linear count of operations. I.e., skipping 5M records requires O(5M) operations, but search for two bounds in a database of 1G elements requieres O(30) operations.
On Wednesday, 13 July 2016 12:51:47 UTC+3, Andrey Rogozhnikov wrote: > > Nick, did you find a way to query records with "large" skips? I've > experienced similar behaviour: a view with hundreds of millions of records, > query.skip about 10M or 50M, and the message: > > undefined:1 > > SyntaxError: Unexpected end of input > at parse (native) > > > On Tuesday, 12 May 2015 07:55:44 UTC+3, Nick Wood wrote: >> >> I'm running a query against a view that emits ~5M rows. The key that's >> being emitted for each row is 32 characters and the value is < 10 >> characters. I'm using the NodeJS 2.0.8 client and running a query with >> 'skip' set to 1,600,000 (give or take 100,000) and with 'limit' set to >> 10,000 (I've also tried 100,000 with the same results). >> >> Sometimes it works. Other times it gives the following error after about >> 1 minute 15 seconds: >> >> undefined:0 >> SyntaxError: Unexpected end of input >> at parse (native) >> >> I've tried increasing the operationTimeout to 5 minutes with no apparent >> affect. >> >> I can't find anything in the logs. Any ideas? >> >> A second related question is that it seems like setting the limit to 10k >> when pulling from the beginning of the view (i.e. skip is set to 0) causes >> the view query to return results very quickly. But when setting 'skip' to >> a much higher value, the query takes a lot longer, even though 'limit' is >> still set to 10k. I assumed the query time would be linear with the same >> limit regardless of where in the full data set the results are coming from. >> Is this expected behavior? >> >> Nick >> > -- You received this message because you are subscribed to the Google Groups "Couchbase" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
