On Wednesday, 19 February 2014 at 09:21:48 UTC, Tarman wrote:
Hi,
We're doing some "super computing" "big data" style stuff with
D. We have a system where we're comparing associative arrays
with billions of entries.
However in this system we need to fairly consider possible
solutions for many "units" at a time within a single thread.
So we'd like to... say, first iterate over the first 10 million
for each "unit" then iterate over the next 10 million for the
next unit so that each unit gets a fair share of CPU time.
However in this case we can't:
int count = 0;
foreach (item; associativeArray) {
if (++count == 10_000_000) {
// here we wish to somehow save the iteration state
}
}
Then on the next iteration:
foreach (resume from previous iteration point) {
}
We've tried copying the keys into a non-associative array and
sure this works, but it is far far far less optimal than an
equivalent C++ solution we wrote where we use an
std::unordered_set and can simply store the iterator.
You could iterate over byValue/byKey/zip(byKey,byValue) which
gives you a range to start again from. You'd need to iterate by
hand (with while,for) though, since your range won't get consumed
by foreach, I guess. Or use RefRange.
not tested:
auto values = refRange(&associativeArray.byValue);
foreach(item; values)
{
if(someday) break;
}
// here values should have been advanced accordingly
// and you can just continue your iteration
foreach(item; values)
{
}
