So I would refactor the logic to use Iterables and the required changes in MongoDocumentStore. For the RDB one I would just wrap current impl. Should be able to push the change once Load 19 is cut.
Would that work for you Julian? Chetan Mehrotra On Thu, Mar 20, 2014 at 2:39 PM, Julian Reschke <[email protected]> wrote: > On 2014-03-20 10:02, Thomas Mueller wrote: >> >> Hi, >> >> For the database case, what we could do is return an iterator that >> internally chunks, that is: >> >> 1) run the query "select * from datastore where id > ? limit 10000 order >> by id" >> 2) if the results set is empty, then iterator.hasNext is false >> 3) else read this in memory and close the connection >> 4) return the data from the in-memory buffer >> 5) if the in-memory buffer is empty, start with 1) replacing ? with the >> latest id >> >> This might also make sense for the MongoDB case. It's not very different >> from what we do now, but the chunking logic would be on the DocumentStore >> side, not on the NodeStore side. So chunking or not would be an >> implementation detail of the DocumentStore. >> >> Regards, >> Thomas > > > SGTM. > > Best regards, Julian >
