[julia-users] Re: Need help/direction on how to optimize some Julia code

[Yuuki Soho]

I don't know too much about the subject but it seems for your problem 
description that the limiting factor will be network bandwidth. 
If I understand correctly now you are doing only 5 requests at the same 
time? It seems to me you could do much more. 

It does depend on the server though, some will block you if you are doing 
too many requests.

At some point I was doing some web crawling, using a single process and a 
recursive function:

function exploreSite(...)
    ....
    for i=1:length(links)
        refs = [refs; remotecall(1, exploreSite, depth-1, links[i]) ]
    end

    for r in refs
        wait(r)
    end

This would fire hundreds of queries asynchronously, and saturate either my 
network or the server.