No, it's still lazy. Partitions are handed off to threads as they are needed, and the results are collected in the main thread. I found the same code in the O'Reilly book after I figured it out, in the "Parallelism on the Cheap" section. It points out the problem with pmap over small computations, and has the workaround with partition-all and doall. Apparently I should have started there. :-p
On Monday, October 14, 2013 10:13:09 PM UTC-7, Sean Corfield wrote: > > I nearly suggested that but it sounded so counter-intuitive and I > didn't have time to construct a test-bed for it... glad you figured it > out. That means that your float-seq has to be fully realized in memory > tho', right? > > On Mon, Oct 14, 2013 at 9:58 PM, Brian Craft > <craft...@gmail.com<javascript:>> > wrote: > > Answering my own question, my grouping example failed because of the > > laziness of the expressions being computed in the pmap: the threads were > not > > evaluating the expressions until they were evaluated (sequentially) by > the > > main thread. Adding a doall gives the result I was hoping for. > > > > > > On Sunday, October 13, 2013 4:10:16 PM UTC-7, Brian Craft wrote: > >> > >> I'm walking a seq of many millions of floats, encoding them for the > >> persistence layer, and getting sequence ids from the db. So, > conceptually > >> there are two parts: the slow part, and the side-effecting part. > Vaguely > >> like > >> > >> (map get-ids (map encode float-seq)) > >> > >> which is later reduced while writing to disk. In the get-ids step the > >> order matters. My first attempt to make the slow part parallel was to > use > >> pmap, > >> > >> (map get-ids (pmap encode float-seq)) > >> > >> However that's actually slower. I expect this is because even though > >> "encode" is the bottleneck, it's still faster than the overhead of > pmap. I > >> next tried pmap over groups of floats, a bit like > >> > >> (map get-ids (flatten (pmap #(map encode %) (partition-all 20000 > >> float-seq)))) > >> > >> (sorry for any typos, I'm just pseudo-coding here) This was still > slower, > >> which surprised me. I understand the first pmap result, but this one is > >> puzzling to me. Even if I partition half the length of the seq (so in > theory > >> it can run two threads, each of which will run five or six seconds), > it's no > >> faster than map. Part of this seems to be the overhead of creating > more > >> intermediate seqs. Perhaps I'm misunderstanding what's happening during > >> partition-all. > >> > >> Is there some obvious way to approach this scenario? I looked briefly > at > >> the reducers library, however it was unclear to me how to deal with the > >> side-effecting portion of the operation. The second (fast) map > operation > >> needs to be done in order. > > > > -- > > -- > > You received this message because you are subscribed to the Google > > Groups "Clojure" group. > > To post to this group, send email to clo...@googlegroups.com<javascript:> > > Note that posts from new members are moderated - please be patient with > your > > first post. > > To unsubscribe from this group, send email to > > clojure+u...@googlegroups.com <javascript:> > > For more options, visit this group at > > http://groups.google.com/group/clojure?hl=en > > --- > > You received this message because you are subscribed to the Google > Groups > > "Clojure" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to clojure+u...@googlegroups.com <javascript:>. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > -- > Sean A Corfield -- (904) 302-SEAN > An Architect's View -- http://corfield.org/ > World Singles, LLC. -- http://worldsingles.com/ > > "Perfection is the enemy of the good." > -- Gustave Flaubert, French realist novelist (1821-1880) > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
