@bearophile:Thanks for your suggestion! Yes we've tried this but unfortunately the performance doesn't work for us, maybe because all our CPUs are quite saturated.
@simendsjo:Thanks also for your suggestion, we are using tasks and yield in our code but in this case we felt that using it just to resume iteration was perhaps a misuse of the feature although we would be interested to see what the overhead is.
@Mr Pankrath: Thanks that looks like what we need!