I see three options. 1) Use query streaming - http://ayende.com/blog/161249/ravendbs-querying-streaming-unbounded-results
I have tried using the streaming api in the past (not too long ago) and found it to be no more performant than using #3 (in fact it was slower.) 2) Increase the max requests per connection It's generally recommended to keep the max requests limit in-tact although you certainly can increase it. Even with the increase though you'll still have to chunk your results. 3) Break your rows into chunks of 128 I'd recommend that you throw an operation in the middle that breaks up results in chunks of 128 and yields out the lists. Then for each row in the raven operation you open a new connection, run through the 128 items and then close it. Here is an example: https://gist.github.com/nathanpalmer/5384121 Nathan Palmer On Thu, Mar 21, 2013 at 10:55 AM, Wayne Douglas <codingvi...@googlemail.com>wrote: > Hi > > I want to pull all the documents from a rdb to start an ETL process off. > > What is the best way of doing this? > > So far i have the following but it craps out at 128 records? > > public class ReadRavenDupeOperation : AbstractOperation > { > private readonly IDocumentStore store; > private IDocumentSession session; > > public ReadRavenDupeOperation(IDocumentStore store) > { > this.store = store; > session = store.OpenSession(); > session.Advanced.MaxNumberOfRequestsPerSession = 1000; > } > > public override IEnumerable<Row> Execute(IEnumerable<Row> rows) > { > var stats = new RavenQueryStatistics(); > var count = session.Query<DupeTable>().Statistics(out > stats).ToList(); > > var loops = stats.TotalResults / 100; > > for (int i = 0; i < loops; i++) > { > if (session == null) > { > session = store.OpenSession(); > session.Advanced.MaxNumberOfRequestsPerSession = 1000; > } > > var docs = > session.Query<DupeTable>().ToList().Skip(i*100).Take(100); > > if (Statistics.OutputtedRows > 0 && > Statistics.OutputtedRows%1000 == 0) > { > session.Dispose(); > session = null; > } > > foreach (var doc in docs) > { > yield return Row.FromObject(doc); > } > } > } > } > > -- > Cheers, > > w:// > > -- > You received this message because you are subscribed to the Google Groups > "Rhino Tools Dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to rhino-tools-dev+unsubscr...@googlegroups.com. > To post to this group, send email to rhino-tools-dev@googlegroups.com. > Visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to rhino-tools-dev+unsubscr...@googlegroups.com. To post to this group, send email to rhino-tools-dev@googlegroups.com. Visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
