On 2014-03-20 10:02, Thomas Mueller wrote:
Hi,
For the database case, what we could do is return an iterator that
internally chunks, that is:
1) run the query "select * from datastore where id > ? limit 10000 order
by id"
2) if the results set is empty, then iterator.hasNext is false
3) else read this in memory and close the connection
4) return the data from the in-memory buffer
5) if the in-memory buffer is empty, start with 1) replacing ? with the
latest id
This might also make sense for the MongoDB case. It's not very different
from what we do now, but the chunking logic would be on the DocumentStore
side, not on the NodeStore side. So chunking or not would be an
implementation detail of the DocumentStore.
Regards,
Thomas
SGTM.
Best regards, Julian
