Did you try the DNS servers from Google, e.g. 8.8.8.8 ? I never saw a reply that needs more than one second. (Well, in our university network.)
Am Montag, 24. August 2015 16:25:06 UTC+2 schrieb Seth: > > Name resolution delays are generally an issue with network latency. Trying > to resolve 1000 uncached names will take a while on any system: > > seth@schroeder:~$ time host www.julialang.org > www.julialang.org is an alias for julialang.github.io. > julialang.github.io is an alias for github.map.fastly.net. > github.map.fastly.net has address 23.235.47.133 > > > real 0m3.268s > user 0m0.003s > sys 0m0.020s > > Unless your server is localhost, you're going to have delays. > > On Monday, August 24, 2015 at 6:49:47 AM UTC-7, Jonathan Malmaud wrote: >> >> Thanks for the report, Andrei - Would you mind filing this is an issue at >> https://github.com/JuliaWeb/Requests.jl? >> >> On Mon, Aug 24, 2015 at 9:17 AM, Andrei Zh <[email protected]> wrote: >> >>> Jonathan, thanks for your support. So far I noticed that DNS gives >>> pretty large delay. E.g. resolving IP addresses for 1000 URLs took 80 >>> seconds in serial code and 26 seconds in muli-task code: >>> >>> >>> Serial execution: >>> >>> julia> @time for url in urls >>> begin >>> Base.getaddrinfo(URI(url).host) >>> end >>> end >>> elapsed time: 80.071810293 seconds (732400 bytes allocated) >>> >>> >>> Multitask execution: >>> >>> >>> julia> @time @sync for url in urls >>> @async begin >>> Base.getaddrinfo(URI(url).host) >>> end >>> end >>> >>> elapsed time: 26.241893516 seconds (4277968 bytes allocated) >>> >>> So I'll try to pre-resolve IPs and test again. >>> >>> >>> On Monday, August 24, 2015 at 4:01:44 PM UTC+3, Jonathan Malmaud wrote: >>> >>>> As one of the maintainers of Requests.jl, I'm especially interested in >>>> its use for high-performance applications so don't hesitate to file an >>>> issue if it gives you any performance problems. >>>> >>>> On Sunday, August 23, 2015 at 7:40:08 PM UTC-4, Andrei Zh wrote: >>>>> >>>>> Hi Steven, >>>>> >>>>> thanks for your answer! It turns out I misunderstood @async long time >>>>> ago, assuming it also makes a remote call to other processes and thus >>>>> introduces true multi-tasking. So now I need to rethink my approach >>>>> before >>>>> going further. >>>>> >>>>> Just to clarify: my goal is to perform as many requests as possible at >>>>> the same time, so I want to use both - multiple processes (to start >>>>> several >>>>> requests at several cores in parallel) and tasks (to launch new requests >>>>> while old ones are still waiting for IO to complete). >>>>> >>>>> So I will update my approach and come back with results or new >>>>> questions. >>>>> >>>>> >>>>> >>>>> On Monday, August 24, 2015 at 2:13:23 AM UTC+3, Steven G. Johnson >>>>> wrote: >>>>>> >>>>>> @parallel in Julia is for executing separate parallel processes (true >>>>>> multi-tasking, with separate address spaces). @async is for "tasks", >>>>>> which >>>>>> are "green threads" and represent cooperative multitasking (within the >>>>>> same >>>>>> process and the same address space). >>>>>> >>>>>> I/O in Julia is asynchronous — while one task is blocked waiting on >>>>>> I/O, another task will wake up and start running. (This is based on the >>>>>> libuv library, which is designed for high-performance asynchronous I/O.) >>>>>> >>>>>> The first question is whether you want to fetch URLs in separate OS >>>>>> processes, or you want to use green threads within the same process. It >>>>>> sounds like you want the latter, in which case @async is the right thing. >>>>>> >>>>>> The second question is whether something about the Requests.jl >>>>>> package is serializing things somehow; for that you might file an issue >>>>>> at >>>>>> Requests.jl. >>>>>> >>>>> >>
