Re: Pmap on hash maps

2016-04-18 Thread myriam abramson 
I like pmap too. I've set the number of processors call with a defonce to
minimize the calls to this function to speed up pmap. What do ya think?

(defn rt-num-processors []
  (.availableProcessors (Runtime/getRuntime)))

(defonce num-processors (rt-num-processors))


On Mon, Apr 18, 2016 at 1:31 AM, Mars0i  wrote:

> On Sunday, April 17, 2016 at 4:23:32 PM UTC-5, JvJ wrote:
>>
>> Orders of magnitude?  Is it even worth running things in parallel in this
>> case?
>>
>
> I got a 2X-5X speed improvement with pmap, but as adrian says, it depends.
>
> The nice thing about  pmap, though, is that using it just involves adding
> a "p" to the front of a map call.  So if you're already using map for
> something, it's easy to test whether pmap helps (preferably using
> Criterium ).
>
> One thing that can help make pmap worthwhile is rearranging code so that
> the function passed to a single map/pmap call does a lot of work with each
> element of the collection passed to it.  If the amount of computation for
> each element is sufficiently high, pmap will be faster.e.g. rather than
> (map f1 (map f2 (map f3 my-collection))), do (map (comp f1 f2 f3)
> my-collection), and then try adding the "p" on the front.  This avoids
> constructing unnecessary intermediate sequences, too.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pmap on hash maps

2016-04-17 Thread Mars0i 
On Sunday, April 17, 2016 at 4:23:32 PM UTC-5, JvJ wrote:
>
> Orders of magnitude?  Is it even worth running things in parallel in this 
> case?
>

I got a 2X-5X speed improvement with pmap, but as adrian says, it depends.

The nice thing about  pmap, though, is that using it just involves adding a 
"p" to the front of a map call.  So if you're already using map for 
something, it's easy to test whether pmap helps (preferably using Criterium 
).  

One thing that can help make pmap worthwhile is rearranging code so that 
the function passed to a single map/pmap call does a lot of work with each 
element of the collection passed to it.  If the amount of computation for 
each element is sufficiently high, pmap will be faster.e.g. rather than 
(map f1 (map f2 (map f3 my-collection))), do (map (comp f1 f2 f3) 
my-collection), and then try adding the "p" on the front.  This avoids 
constructing unnecessary intermediate sequences, too.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pmap on hash maps

2016-04-17 Thread Erik Assum 
There was this talk on this a while ago by Leon Barrett
https://m.youtube.com/watch?v=BzKjIk0vgzE
Also, I guess this from this years Clojure West might be useful, all though I 
haven't watched it yet:
https://m.youtube.com/watch?list=PLZdCLR02grLq4e8-1P2JNHBKUOLFTX3kb&v=fA9ZXy4N13s

Erik. 
-- 
i farta

> Den 17. apr. 2016 kl. 23.23 skrev JvJ :
> 
> Orders of magnitude?  Is it even worth running things in parallel in this 
> case?
> 
>> On Sunday, 17 April 2016 07:20:30 UTC-7, tbc++ wrote:
>> The cost of creating that extra collection will be vastly overshadowed by 
>> the cost of running pmap. pmap involves the use of several locks, and thread 
>> co-ordination primitives, these will most likely be orders of magnitude more 
>> expensive than an extra allocation. 
>> 
>> Timothy
>> 
>>> On Sun, Apr 17, 2016 at 12:38 AM, JvJ  wrote:
>>> Perhaps I should have rephrased.  I meant calling it in such a way that the 
>>> result is a hash map.
>>> 
>>> I know I could do something like:
>>> 
>>> (into a (pmap f a))
>>> 
>>> This will give the same result, but there's an extra collection in there 
>>> that I'm not sure is necessary.
>>> 
>>> 
>>>> On Saturday, 16 April 2016 20:15:51 UTC-7, Alex Miller wrote:
>>>> You can call pmap on hash maps!
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to clojure+u...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> 
>> -- 
>> “One of the main causes of the fall of the Roman Empire was that–lacking 
>> zero–they had no way to indicate successful termination of their C programs.”
>> (Robert Firth)
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pmap on hash maps

2016-04-17 Thread adrian . medina 
It depends on your specific use case. Sometimes the overhead of parallelism 
overshadows the performance gains. The rule of thumb for any language is to 
first identify bottlenecks in your code through profiling and optimize 
accordingly. 

For a concrete example, a while ago I was created a program with Clojure 
which interprets 3D game assets in one format and re-exports them to 
another format. I needed to run this on thousands of files. After 
profiling, I found that parallelizing the conversion process for one asset 
was a red herring as I could barely saturate one thread before it finished. 
So instead of parallelizing the internals of the conversion process, I 
found it was much more efficient (and less complex) to just run the 
conversion process in parallel on many assets at the same time. This meant 
I could scale the import/export procedure linearly with CPUs, and it was 
incredibly simple to do with Clojure out of the box without increasing the 
complexity of the program at all. 

I know this is a relatively obvious use of parallelism, but my point is 
that while sometimes it's tempting to try to parallelize things all the way 
down my experience has taught me that parallelism is often more important 
at the I/O boundaries of your program, whatever that may be. 

On Sunday, April 17, 2016 at 5:23:32 PM UTC-4, JvJ wrote:
>
> Orders of magnitude?  Is it even worth running things in parallel in this 
> case?
>
> On Sunday, 17 April 2016 07:20:30 UTC-7, tbc++ wrote:
>>
>> The cost of creating that extra collection will be vastly overshadowed by 
>> the cost of running pmap. pmap involves the use of several locks, and 
>> thread co-ordination primitives, these will most likely be orders of 
>> magnitude more expensive than an extra allocation. 
>>
>> Timothy
>>
>> On Sun, Apr 17, 2016 at 12:38 AM, JvJ  wrote:
>>
>>> Perhaps I should have rephrased.  I meant calling it in such a way that 
>>> the result is a hash map.
>>>
>>> I know I could do something like:
>>>
>>> (into a (pmap f a))
>>>
>>> This will give the same result, but there's an extra collection in there 
>>> that I'm not sure is necessary.
>>>
>>>
>>> On Saturday, 16 April 2016 20:15:51 UTC-7, Alex Miller wrote:
>>>>
>>>> You can call pmap on hash maps!
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to clojure+u...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> -- 
>> “One of the main causes of the fall of the Roman Empire was that–lacking 
>> zero–they had no way to indicate successful termination of their C 
>> programs.”
>> (Robert Firth) 
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pmap on hash maps

2016-04-17 Thread JvJ 
Orders of magnitude?  Is it even worth running things in parallel in this 
case?

On Sunday, 17 April 2016 07:20:30 UTC-7, tbc++ wrote:
>
> The cost of creating that extra collection will be vastly overshadowed by 
> the cost of running pmap. pmap involves the use of several locks, and 
> thread co-ordination primitives, these will most likely be orders of 
> magnitude more expensive than an extra allocation. 
>
> Timothy
>
> On Sun, Apr 17, 2016 at 12:38 AM, JvJ > 
> wrote:
>
>> Perhaps I should have rephrased.  I meant calling it in such a way that 
>> the result is a hash map.
>>
>> I know I could do something like:
>>
>> (into a (pmap f a))
>>
>> This will give the same result, but there's an extra collection in there 
>> that I'm not sure is necessary.
>>
>>
>> On Saturday, 16 April 2016 20:15:51 UTC-7, Alex Miller wrote:
>>>
>>> You can call pmap on hash maps!
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com 
>> 
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> “One of the main causes of the fall of the Roman Empire was that–lacking 
> zero–they had no way to indicate successful termination of their C 
> programs.”
> (Robert Firth) 
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pmap on hash maps

2016-04-17 Thread Timothy Baldridge 
The cost of creating that extra collection will be vastly overshadowed by
the cost of running pmap. pmap involves the use of several locks, and
thread co-ordination primitives, these will most likely be orders of
magnitude more expensive than an extra allocation.

Timothy

On Sun, Apr 17, 2016 at 12:38 AM, JvJ  wrote:

> Perhaps I should have rephrased.  I meant calling it in such a way that
> the result is a hash map.
>
> I know I could do something like:
>
> (into a (pmap f a))
>
> This will give the same result, but there's an extra collection in there
> that I'm not sure is necessary.
>
>
> On Saturday, 16 April 2016 20:15:51 UTC-7, Alex Miller wrote:
>>
>> You can call pmap on hash maps!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pmap on hash maps

2016-04-16 Thread JvJ 
Perhaps I should have rephrased.  I meant calling it in such a way that the 
result is a hash map.

I know I could do something like:

(into a (pmap f a))

This will give the same result, but there's an extra collection in there 
that I'm not sure is necessary.

On Saturday, 16 April 2016 20:15:51 UTC-7, Alex Miller wrote:
>
> You can call pmap on hash maps!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pmap on hash maps

2016-04-16 Thread Alex Miller 
You can call pmap on hash maps!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pmap on hash maps

2016-04-16 Thread JvJ 

Does anyone know if there exists a convenient method to call something like 
pmap on a hash map?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.