Uni networks are typically high bandwidth and low latency and are optimally peered. My home network, unfortunately, exhibits none of those traits :)
In any case, even a 70-millisecond resolution time results in 70 seconds of delay for 1000 uncached resolutions. On Monday, August 24, 2015 at 7:42:16 AM UTC-7, Uwe Fechner wrote: > > Did you try the DNS servers from Google, e.g. 8.8.8.8 ? > I never saw a reply that needs more than one second. > (Well, in our university network.) > > Am Montag, 24. August 2015 16:25:06 UTC+2 schrieb Seth: >> >> Name resolution delays are generally an issue with network latency. >> Trying to resolve 1000 uncached names will take a while on any system: >> >> seth@schroeder:~$ time host www.julialang.org >> www.julialang.org is an alias for julialang.github.io. >> julialang.github.io is an alias for github.map.fastly.net. >> github.map.fastly.net has address 23.235.47.133 >> >> >> real 0m3.268s >> user 0m0.003s >> sys 0m0.020s >> >> Unless your server is localhost, you're going to have delays. >> >> On Monday, August 24, 2015 at 6:49:47 AM UTC-7, Jonathan Malmaud wrote: >>> >>> Thanks for the report, Andrei - Would you mind filing this is an issue >>> at https://github.com/JuliaWeb/Requests.jl? >>> >>> On Mon, Aug 24, 2015 at 9:17 AM, Andrei Zh <[email protected]> wrote: >>> >>>> Jonathan, thanks for your support. So far I noticed that DNS gives >>>> pretty large delay. E.g. resolving IP addresses for 1000 URLs took 80 >>>> seconds in serial code and 26 seconds in muli-task code: >>>> >>>> >>>> Serial execution: >>>> >>>> julia> @time for url in urls >>>> begin >>>> Base.getaddrinfo(URI(url).host) >>>> end >>>> end >>>> elapsed time: 80.071810293 seconds (732400 bytes allocated) >>>> >>>> >>>> Multitask execution: >>>> >>>> >>>> julia> @time @sync for url in urls >>>> @async begin >>>> Base.getaddrinfo(URI(url).host) >>>> end >>>> end >>>> >>>> elapsed time: 26.241893516 seconds (4277968 bytes allocated) >>>> >>>> So I'll try to pre-resolve IPs and test again. >>>> >>>> >>>> On Monday, August 24, 2015 at 4:01:44 PM UTC+3, Jonathan Malmaud wrote: >>>> >>>>> As one of the maintainers of Requests.jl, I'm especially interested in >>>>> its use for high-performance applications so don't hesitate to file an >>>>> issue if it gives you any performance problems. >>>>> >>>>> On Sunday, August 23, 2015 at 7:40:08 PM UTC-4, Andrei Zh wrote: >>>>>> >>>>>> Hi Steven, >>>>>> >>>>>> thanks for your answer! It turns out I misunderstood @async long time >>>>>> ago, assuming it also makes a remote call to other processes and thus >>>>>> introduces true multi-tasking. So now I need to rethink my approach >>>>>> before >>>>>> going further. >>>>>> >>>>>> Just to clarify: my goal is to perform as many requests as possible >>>>>> at the same time, so I want to use both - multiple processes (to start >>>>>> several requests at several cores in parallel) and tasks (to launch new >>>>>> requests while old ones are still waiting for IO to complete). >>>>>> >>>>>> So I will update my approach and come back with results or new >>>>>> questions. >>>>>> >>>>>> >>>>>> >>>>>> On Monday, August 24, 2015 at 2:13:23 AM UTC+3, Steven G. Johnson >>>>>> wrote: >>>>>>> >>>>>>> @parallel in Julia is for executing separate parallel processes >>>>>>> (true multi-tasking, with separate address spaces). @async is for >>>>>>> "tasks", >>>>>>> which are "green threads" and represent cooperative multitasking >>>>>>> (within >>>>>>> the same process and the same address space). >>>>>>> >>>>>>> I/O in Julia is asynchronous — while one task is blocked waiting on >>>>>>> I/O, another task will wake up and start running. (This is based on >>>>>>> the >>>>>>> libuv library, which is designed for high-performance asynchronous I/O.) >>>>>>> >>>>>>> The first question is whether you want to fetch URLs in separate OS >>>>>>> processes, or you want to use green threads within the same process. >>>>>>> It >>>>>>> sounds like you want the latter, in which case @async is the right >>>>>>> thing. >>>>>>> >>>>>>> The second question is whether something about the Requests.jl >>>>>>> package is serializing things somehow; for that you might file an issue >>>>>>> at >>>>>>> Requests.jl. >>>>>>> >>>>>> >>>
