Wouldn't this be simpler with pmap, e.g. http://gist.github.com/399269
although to be honest I don't really know how the automatically parallelized clojure functions decide how many threads to use. Is the JVM smart enough to only create as many system-level threads as make sense on my hardware? Best -- Eric Nurullah Akkaya <nurul...@nakkaya.com> writes: > Since you don't need coordination or keep some sort of state, IMHO > future is better suited for this. > Following gist is my take, it first reads the file that contains the > list of URLs to be downloaded then splits the list into number of > thread pieces. Each future object gets a piece of the list and start > processing in its own thread. Each URL is written to disk using a > UUID. > > http://gist.github.com/399127 > > Cheers... > -- > Nurullah Akkaya > http://nakkaya.com > > > > On Wed, May 12, 2010 at 9:29 PM, nickikt <nick...@gmail.com> wrote: >> Hallo all, >> >> A friend of mine ask if there is a smart way to get the html code of >> couple thousand links but with a script, it takes for ever since it >> always has takes a couple of seconds to get the connection. >> >> I needs to be multi threaded so we can use all of the download rate. >> So I sad I could try it with clojure but I am pretty new in clojure >> and java (almost through the Programming Clojure book but nothing >> practical) and thought i just ask here instead of trying to copy some >> java code in clojure and hack something bad. >> >> My idea would be to span a agent for every dump and control that there >> won't be more then a 10-20 threads. And dump it into a file. >> >> How would you implement this? >> A function that does all of it in a agent and control it with a >> counter from a other function? >> >> Is there a clojure way to write a file or should I use the java way? >> Same for reading a file. >> >> Here is some javacode that does part of what I want. Is there a better >> way then proxy all of it? >> >> url = new URL("random page"); >> URLConnection conn = url.openConnection(); >> DataInputStream in = new DataInputStream ( conn.getInputStream >> ( ) ) ; >> BufferedReader d = new BufferedReader(new InputStreamReader(in)); >> while(d.ready()) >> { >> System.out.println( d.readLine()); >> } >> >> Just that this prints to console instead of a file. >> >> So tanks for reading. I'm working on it any tip or suggestion would >> help. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
