Re: group-by vs. reducers?
Hey,
Reducers is fascinating and quite complex at the same time. I feel like
best practices around it has not quite solidified yet.
Here is how I made your example work:
(ns group-by-reducers.core
(:require [clojure.core.reducers :as r :only [fold reduce map]])
(:require [criterium.core :as c]))
(defn group-by-naive [f coll]
(reduce
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
{}
coll))
(defn group-by-red [f coll]
(letfn [(reduce-groups
([] {})
([g1 g2]
(merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
(defn -main
[ args]
(c/quick-bench (group-by #(mod % 29) (range 1)))
(c/quick-bench (group-by-naive #(mod % 29) (range 1)))
(c/quick-bench (group-by-red #(mod % 29) (range 1
(available as a gist here: https://gist.github.com/4232044)
The core and naive versions perform roughly equally (~4ms) while the
reducers version takes 10x as much time (~40ms) on my two-core laptop.
I'm absolutely sure I'm doing something wrong (there is probably too much
memory allocated by the map function) and would like to hear the opinion of
someone more knowledgeable with reducers. I just did not want the subject
to be buried.
Hope that still helps a bit,
Balint
ps. There are some tips and tricks in this
post: http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
Hi,
As I was trying to wrap my head around the reducers library[1], I thought
implementing group-by would be a good exercise to gain some insight.
After spending a few hours with it, I'm still pretty much clueless, so
hope to find someone here to help me out:
So if I understood the reducer lingo introduced in [2],[3] and group-by
correctly, it reduces the following reducing function on a collection
(fn group-by-reducef [keyfn ret x]
(let [k (keyfn x)]
(assoc ret k (conj (get ret k []) x
where keyfn is provided by a partial function application.
fold needs a combining function that takes two result maps that have
already been grouped and merges them.
A naive implementation could look like
(defn group-by-combinef
([] {})
([g1 g2]
(persistent!
(reduce (fn [res k v]
(assoc! res k (into (get res k []) v)))
(transient g1) g2
(defn group-by [f coll]
(fold (partial gr-by-reducef f) gr-by-combinef coll))
Now couple of questions:
1) I expected fold to actually perform the operation, how can I force it
to give me the result?
2) Can somehow the actual reducing at the leaf nodes still take advantage
of transient collections?
3) I took a look at flatten as it seems the closest match. Again, if I
call (flatten [[1 2] [2 4]]), I don't actually get the result. How do I get
to the result?
Thanks!
[1]
https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj
[2]
http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html
[3] http://clojure.com/blog/2012/05/15/anatomy-of-reducer.html
--
László Török
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: group-by vs. reducers?
BTW I understood the most about reducers (still not quite there yet, though
:) ) from Rich Hickey's talk at EuroClojure:
https://vimeo.com/45561411
On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
Hey,
Reducers is fascinating and quite complex at the same time. I feel like
best practices around it has not quite solidified yet.
Here is how I made your example work:
(ns group-by-reducers.core
(:require [clojure.core.reducers :as r :only [fold reduce map]])
(:require [criterium.core :as c]))
(defn group-by-naive [f coll]
(reduce
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
{}
coll))
(defn group-by-red [f coll]
(letfn [(reduce-groups
([] {})
([g1 g2]
(merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
(defn -main
[ args]
(c/quick-bench (group-by #(mod % 29) (range 1)))
(c/quick-bench (group-by-naive #(mod % 29) (range 1)))
(c/quick-bench (group-by-red #(mod % 29) (range 1
(available as a gist here: https://gist.github.com/4232044)
The core and naive versions perform roughly equally (~4ms) while the
reducers version takes 10x as much time (~40ms) on my two-core laptop.
I'm absolutely sure I'm doing something wrong (there is probably too much
memory allocated by the map function) and would like to hear the opinion of
someone more knowledgeable with reducers. I just did not want the subject
to be buried.
Hope that still helps a bit,
Balint
ps. There are some tips and tricks in this post:
http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
Hi,
As I was trying to wrap my head around the reducers library[1], I thought
implementing group-by would be a good exercise to gain some insight.
After spending a few hours with it, I'm still pretty much clueless, so
hope to find someone here to help me out:
So if I understood the reducer lingo introduced in [2],[3] and group-by
correctly, it reduces the following reducing function on a collection
(fn group-by-reducef [keyfn ret x]
(let [k (keyfn x)]
(assoc ret k (conj (get ret k []) x
where keyfn is provided by a partial function application.
fold needs a combining function that takes two result maps that have
already been grouped and merges them.
A naive implementation could look like
(defn group-by-combinef
([] {})
([g1 g2]
(persistent!
(reduce (fn [res k v]
(assoc! res k (into (get res k []) v)))
(transient g1) g2
(defn group-by [f coll]
(fold (partial gr-by-reducef f) gr-by-combinef coll))
Now couple of questions:
1) I expected fold to actually perform the operation, how can I force it
to give me the result?
2) Can somehow the actual reducing at the leaf nodes still take advantage
of transient collections?
3) I took a look at flatten as it seems the closest match. Again, if I
call (flatten [[1 2] [2 4]]), I don't actually get the result. How do I get
to the result?
Thanks!
[1]
https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj
[2]
http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html
[3] http://clojure.com/blog/2012/05/15/anatomy-of-reducer.html
--
László Török
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: [ANN] sublime-lispindent, clojure indenting for sublime text 2
You're welcome :) On Fri, Dec 7, 2012 at 1:12 AM, Anton Holmberg antonholmber...@gmail.comwrote: This is so awesome! You are my hero. Been searching for this for a while now. Den måndagen den 12:e november 2012 kl. 17:25:38 UTC+1 skrev Jonathan Fischer Friberg: Dear clojure mailing list, As the indenting for clojure (and lisp in general) was very lacking in sublime, I decided to make a plugin: https://github.com/odyssomay/**sublime-lispindenthttps://github.com/odyssomay/sublime-lispindent I hope someone finds this useful. By the way, if someone with a mac could try the keyboard shortcuts that would be great. I don't own a mac so I cannot test them. Sincerely, Jonathan -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
How was the skills matters conference in London (6. Dec)?
Hi, I couldn't make it to the skill matters conference in London ))-: just being curious,... how was it? Any news? was there a main theme? Many Greetings John -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: How was the skills matters conference in London (6. Dec)?
It was really great, all the videos are up already: http://skillsmatter.com/event/scala/clojure-exchange-2012 Have a look. Thomas On Friday, December 7, 2012 10:59:32 AM UTC, john wrote: Hi, I couldn't make it to the skill matters conference in London ))-: just being curious,... how was it? Any news? was there a main theme? Many Greetings John -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: group-by vs. reducers?
Hi,
There are indeed too much allocationfoing on in your r/map.
You don't need the rmap,
Start from a plain old reduce like your reduce-by-naive, replace reduce by
r/fold, remove the seed and add the combine-fn (which now provides the
seed):
(defn group-by-red [f coll]
(r/fold
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Crude benchmark;
(let [v (vec (range 100))]
(dotimes [n 5]
(println Run n)
(time (group-by #(mod % 29) v))
(time (group-by-naive #(mod % 29) v))
(time (group-by-red #(mod % 29) v
Run 0
Elapsed time: 938.878 msecs
Elapsed time: 778.161 msecs
Elapsed time: 1035.997 msecs
Run 1
Elapsed time: 715.352 msecs
Elapsed time: 771.913 msecs
Elapsed time: 840.264 msecs
Run 2
Elapsed time: 656.588 msecs
Elapsed time: 700.114 msecs
Elapsed time: 777.323 msecs
Run 3
Elapsed time: 731.781 msecs
Elapsed time: 713.547 msecs
Elapsed time: 875.801 msecs
Run 4
Elapsed time: 689.711 msecs
Elapsed time: 710.728 msecs
Elapsed time: 1112.609 msecs
nil
Let's play with the granularity (defaults to 512)
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Re-benchmark:
Run 0
Elapsed time: 810.676 msecs
Elapsed time: 736.764 msecs
Elapsed time: 547.651 msecs
Run 1
Elapsed time: 681.249 msecs
Elapsed time: 850.046 msecs
Elapsed time: 521.27 msecs
Run 2
Elapsed time: 669.385 msecs
Elapsed time: 712.15 msecs
Elapsed time: 518.502 msecs
Run 3
Elapsed time: 673.108 msecs
Elapsed time: 745.688 msecs
Elapsed time: 542.837 msecs
Run 4
Elapsed time: 654.196 msecs
Elapsed time: 723.074 msecs
Elapsed time: 506.861 msecs
Note that you are not exactly comparing apples to apples since group-by-red
is returning maps whose values are unrealized lazy sequence (concats of
concats of ocncats of vactors) instead of vectors
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with into)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Run 0
Elapsed time: 763.455 msecs
Elapsed time: 763.501 msecs
Elapsed time: 681.075 msecs
Run 1
Elapsed time: 645.52 msecs
Elapsed time: 731.545 msecs
Elapsed time: 476.381 msecs
Run 2
Elapsed time: 660.775 msecs
Elapsed time: 728.19 msecs
Elapsed time: 475.543 msecs
Run 3
Elapsed time: 657.255 msecs
Elapsed time: 725.995 msecs
Elapsed time: 494.038 msecs
Run 4
Elapsed time: 647.53 msecs
Elapsed time: 731.085 msecs
Elapsed time: 538.649 msecs
hth,
Christophe
On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi balint.e...@gmail.com wrote:
BTW I understood the most about reducers (still not quite there yet,
though :) ) from Rich Hickey's talk at EuroClojure:
https://vimeo.com/45561411
On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
Hey,
Reducers is fascinating and quite complex at the same time. I feel like
best practices around it has not quite solidified yet.
Here is how I made your example work:
(ns group-by-reducers.core
(:require [clojure.core.reducers :as r :only [fold reduce map]])
(:require [criterium.core :as c]))
(defn group-by-naive [f coll]
(reduce
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
{}
coll))
(defn group-by-red [f coll]
(letfn [(reduce-groups
([] {})
([g1 g2]
(merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
(defn -main
[ args]
(c/quick-bench (group-by #(mod % 29) (range 1)))
(c/quick-bench (group-by-naive #(mod % 29) (range 1)))
(c/quick-bench (group-by-red #(mod % 29) (range 1
(available as a gist here:
https://gist.github.com/**4232044https://gist.github.com/4232044
)
The core and naive versions perform roughly equally (~4ms) while the
reducers version takes 10x as much time (~40ms) on my two-core laptop.
I'm absolutely sure I'm doing something wrong (there is probably too much
memory allocated by the map function) and would like to hear the opinion of
someone more knowledgeable with reducers. I just did not want the subject
to be buried.
Hope that still helps a bit,
Balint
ps. There are some tips and tricks in this post: http://www.thebusby.com/
**2012/07/tips-tricks-with-**clojure-reducers.htmlhttp://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
Hi,
As I was trying to wrap my head around the reducers library[1], I
thought implementing group-by would be a good exercise to gain some insight.
After spending a few hours with it, I'm still pretty much clueless, so
hope to find someone here to help me out:
So if I understood the reducer lingo introduced in [2],[3] and group-by
correctly, it reduces the following reducing function on a collection
(fn group-by-reducef [keyfn ret x]
(let [k (keyfn x)]
(assoc ret
Re: How was the skills matters conference in London (6. Dec)?
Thanks a lot Thomas ... I am looking forward to the not yet published podcast 18:00 - late Clojurex drinks Am Freitag, 7. Dezember 2012 12:33:20 UTC+1 schrieb Thomas: It was really great, all the videos are up already: http://skillsmatter.com/event/scala/clojure-exchange-2012 Have a look. Thomas On Friday, December 7, 2012 10:59:32 AM UTC, john wrote: Hi, I couldn't make it to the skill matters conference in London ))-: just being curious,... how was it? Any news? was there a main theme? Many Greetings John -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: group-by vs. reducers?
Hi,
shouldn't
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
be
(fn [groups a]
(let [k (f a)]
(assoc groups k (conj (get groups k []) a
?
Las
2012/12/7 Christophe Grand christo...@cgrand.net
Hi,
There are indeed too much allocationfoing on in your r/map.
You don't need the rmap,
Start from a plain old reduce like your reduce-by-naive, replace reduce by
r/fold, remove the seed and add the combine-fn (which now provides the
seed):
(defn group-by-red [f coll]
(r/fold
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Crude benchmark;
(let [v (vec (range 100))]
(dotimes [n 5]
(println Run n)
(time (group-by #(mod % 29) v))
(time (group-by-naive #(mod % 29) v))
(time (group-by-red #(mod % 29) v
Run 0
Elapsed time: 938.878 msecs
Elapsed time: 778.161 msecs
Elapsed time: 1035.997 msecs
Run 1
Elapsed time: 715.352 msecs
Elapsed time: 771.913 msecs
Elapsed time: 840.264 msecs
Run 2
Elapsed time: 656.588 msecs
Elapsed time: 700.114 msecs
Elapsed time: 777.323 msecs
Run 3
Elapsed time: 731.781 msecs
Elapsed time: 713.547 msecs
Elapsed time: 875.801 msecs
Run 4
Elapsed time: 689.711 msecs
Elapsed time: 710.728 msecs
Elapsed time: 1112.609 msecs
nil
Let's play with the granularity (defaults to 512)
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Re-benchmark:
Run 0
Elapsed time: 810.676 msecs
Elapsed time: 736.764 msecs
Elapsed time: 547.651 msecs
Run 1
Elapsed time: 681.249 msecs
Elapsed time: 850.046 msecs
Elapsed time: 521.27 msecs
Run 2
Elapsed time: 669.385 msecs
Elapsed time: 712.15 msecs
Elapsed time: 518.502 msecs
Run 3
Elapsed time: 673.108 msecs
Elapsed time: 745.688 msecs
Elapsed time: 542.837 msecs
Run 4
Elapsed time: 654.196 msecs
Elapsed time: 723.074 msecs
Elapsed time: 506.861 msecs
Note that you are not exactly comparing apples to apples since
group-by-red is returning maps whose values are unrealized lazy sequence
(concats of concats of ocncats of vactors) instead of vectors
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with into)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Run 0
Elapsed time: 763.455 msecs
Elapsed time: 763.501 msecs
Elapsed time: 681.075 msecs
Run 1
Elapsed time: 645.52 msecs
Elapsed time: 731.545 msecs
Elapsed time: 476.381 msecs
Run 2
Elapsed time: 660.775 msecs
Elapsed time: 728.19 msecs
Elapsed time: 475.543 msecs
Run 3
Elapsed time: 657.255 msecs
Elapsed time: 725.995 msecs
Elapsed time: 494.038 msecs
Run 4
Elapsed time: 647.53 msecs
Elapsed time: 731.085 msecs
Elapsed time: 538.649 msecs
hth,
Christophe
On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi balint.e...@gmail.comwrote:
BTW I understood the most about reducers (still not quite there yet,
though :) ) from Rich Hickey's talk at EuroClojure:
https://vimeo.com/45561411
On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
Hey,
Reducers is fascinating and quite complex at the same time. I feel like
best practices around it has not quite solidified yet.
Here is how I made your example work:
(ns group-by-reducers.core
(:require [clojure.core.reducers :as r :only [fold reduce map]])
(:require [criterium.core :as c]))
(defn group-by-naive [f coll]
(reduce
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
{}
coll))
(defn group-by-red [f coll]
(letfn [(reduce-groups
([] {})
([g1 g2]
(merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
(defn -main
[ args]
(c/quick-bench (group-by #(mod % 29) (range 1)))
(c/quick-bench (group-by-naive #(mod % 29) (range 1)))
(c/quick-bench (group-by-red #(mod % 29) (range 1
(available as a gist here:
https://gist.github.com/**4232044https://gist.github.com/4232044
)
The core and naive versions perform roughly equally (~4ms) while the
reducers version takes 10x as much time (~40ms) on my two-core laptop.
I'm absolutely sure I'm doing something wrong (there is probably too
much memory allocated by the map function) and would like to hear the
opinion of someone more knowledgeable with reducers. I just did not want
the subject to be buried.
Hope that still helps a bit,
Balint
ps. There are some tips and tricks in this post:
http://www.thebusby.com/**2012/07/tips-tricks-with-**
clojure-reducers.htmlhttp://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
Hi,
As I was trying to wrap my head around the reducers library[1], I
thought implementing group-by would be a good exercise to gain some
Re: group-by vs. reducers?
Absolutely
On Fri, Dec 7, 2012 at 1:29 PM, László Török ltoro...@gmail.com wrote:
Hi,
shouldn't
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
be
(fn [groups a]
(let [k (f a)]
(assoc groups k (conj (get groups k []) a
?
Las
2012/12/7 Christophe Grand christo...@cgrand.net
Hi,
There are indeed too much allocationfoing on in your r/map.
You don't need the rmap,
Start from a plain old reduce like your reduce-by-naive, replace reduce
by r/fold, remove the seed and add the combine-fn (which now provides the
seed):
(defn group-by-red [f coll]
(r/fold
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Crude benchmark;
(let [v (vec (range 100))]
(dotimes [n 5]
(println Run n)
(time (group-by #(mod % 29) v))
(time (group-by-naive #(mod % 29) v))
(time (group-by-red #(mod % 29) v
Run 0
Elapsed time: 938.878 msecs
Elapsed time: 778.161 msecs
Elapsed time: 1035.997 msecs
Run 1
Elapsed time: 715.352 msecs
Elapsed time: 771.913 msecs
Elapsed time: 840.264 msecs
Run 2
Elapsed time: 656.588 msecs
Elapsed time: 700.114 msecs
Elapsed time: 777.323 msecs
Run 3
Elapsed time: 731.781 msecs
Elapsed time: 713.547 msecs
Elapsed time: 875.801 msecs
Run 4
Elapsed time: 689.711 msecs
Elapsed time: 710.728 msecs
Elapsed time: 1112.609 msecs
nil
Let's play with the granularity (defaults to 512)
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Re-benchmark:
Run 0
Elapsed time: 810.676 msecs
Elapsed time: 736.764 msecs
Elapsed time: 547.651 msecs
Run 1
Elapsed time: 681.249 msecs
Elapsed time: 850.046 msecs
Elapsed time: 521.27 msecs
Run 2
Elapsed time: 669.385 msecs
Elapsed time: 712.15 msecs
Elapsed time: 518.502 msecs
Run 3
Elapsed time: 673.108 msecs
Elapsed time: 745.688 msecs
Elapsed time: 542.837 msecs
Run 4
Elapsed time: 654.196 msecs
Elapsed time: 723.074 msecs
Elapsed time: 506.861 msecs
Note that you are not exactly comparing apples to apples since
group-by-red is returning maps whose values are unrealized lazy sequence
(concats of concats of ocncats of vactors) instead of vectors
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with into)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Run 0
Elapsed time: 763.455 msecs
Elapsed time: 763.501 msecs
Elapsed time: 681.075 msecs
Run 1
Elapsed time: 645.52 msecs
Elapsed time: 731.545 msecs
Elapsed time: 476.381 msecs
Run 2
Elapsed time: 660.775 msecs
Elapsed time: 728.19 msecs
Elapsed time: 475.543 msecs
Run 3
Elapsed time: 657.255 msecs
Elapsed time: 725.995 msecs
Elapsed time: 494.038 msecs
Run 4
Elapsed time: 647.53 msecs
Elapsed time: 731.085 msecs
Elapsed time: 538.649 msecs
hth,
Christophe
On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi balint.e...@gmail.comwrote:
BTW I understood the most about reducers (still not quite there yet,
though :) ) from Rich Hickey's talk at EuroClojure:
https://vimeo.com/45561411
On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
Hey,
Reducers is fascinating and quite complex at the same time. I feel like
best practices around it has not quite solidified yet.
Here is how I made your example work:
(ns group-by-reducers.core
(:require [clojure.core.reducers :as r :only [fold reduce map]])
(:require [criterium.core :as c]))
(defn group-by-naive [f coll]
(reduce
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
{}
coll))
(defn group-by-red [f coll]
(letfn [(reduce-groups
([] {})
([g1 g2]
(merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
(defn -main
[ args]
(c/quick-bench (group-by #(mod % 29) (range 1)))
(c/quick-bench (group-by-naive #(mod % 29) (range 1)))
(c/quick-bench (group-by-red #(mod % 29) (range 1
(available as a gist here:
https://gist.github.com/**4232044https://gist.github.com/4232044
)
The core and naive versions perform roughly equally (~4ms) while the
reducers version takes 10x as much time (~40ms) on my two-core laptop.
I'm absolutely sure I'm doing something wrong (there is probably too
much memory allocated by the map function) and would like to hear the
opinion of someone more knowledgeable with reducers. I just did not want
the subject to be buried.
Hope that still helps a bit,
Balint
ps. There are some tips and tricks in this post:
http://www.thebusby.com/**2012/07/tips-tricks-with-**
clojure-reducers.htmlhttp://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
Hi,
As I was trying to wrap my head around the
Re: group-by vs. reducers?
Thank you, Christophe, that makes total sense.
Interestingly, on my machine, the reducers version is still slower:
Run # 0
Elapsed time: 1105.586 msecs
Elapsed time: 685.8 msecs
Elapsed time: 549.969 msecs
Run # 1
Elapsed time: 497.831 msecs
Elapsed time: 489.932 msecs
Elapsed time: 520.38 msecs
Run # 2
Elapsed time: 363.873 msecs
Elapsed time: 489.529 msecs
Elapsed time: 556.117 msecs
Run # 3
Elapsed time: 357.822 msecs
Elapsed time: 429.894 msecs
Elapsed time: 540.975 msecs
Run # 4
Elapsed time: 425.276 msecs
Elapsed time: 431.819 msecs
Elapsed time: 528.8 msecs
I guess the JVM finds some serious optimizations during the runs for the
non-reducer versions. I saw the same difference when using criterium for
benchmarking.
Balint
On Friday, December 7, 2012 1:44:58 PM UTC+1, Christophe Grand wrote:
Absolutely
On Fri, Dec 7, 2012 at 1:29 PM, László Török ltor...@gmail.comjavascript:
wrote:
Hi,
shouldn't
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
be
(fn [groups a]
(let [k (f a)]
(assoc groups k (conj (get groups k []) a
?
Las
2012/12/7 Christophe Grand chris...@cgrand.net javascript:
Hi,
There are indeed too much allocationfoing on in your r/map.
You don't need the rmap,
Start from a plain old reduce like your reduce-by-naive, replace reduce
by r/fold, remove the seed and add the combine-fn (which now provides the
seed):
(defn group-by-red [f coll]
(r/fold
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Crude benchmark;
(let [v (vec (range 100))]
(dotimes [n 5]
(println Run n)
(time (group-by #(mod % 29) v))
(time (group-by-naive #(mod % 29) v))
(time (group-by-red #(mod % 29) v
Run 0
Elapsed time: 938.878 msecs
Elapsed time: 778.161 msecs
Elapsed time: 1035.997 msecs
Run 1
Elapsed time: 715.352 msecs
Elapsed time: 771.913 msecs
Elapsed time: 840.264 msecs
Run 2
Elapsed time: 656.588 msecs
Elapsed time: 700.114 msecs
Elapsed time: 777.323 msecs
Run 3
Elapsed time: 731.781 msecs
Elapsed time: 713.547 msecs
Elapsed time: 875.801 msecs
Run 4
Elapsed time: 689.711 msecs
Elapsed time: 710.728 msecs
Elapsed time: 1112.609 msecs
nil
Let's play with the granularity (defaults to 512)
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with concat)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Re-benchmark:
Run 0
Elapsed time: 810.676 msecs
Elapsed time: 736.764 msecs
Elapsed time: 547.651 msecs
Run 1
Elapsed time: 681.249 msecs
Elapsed time: 850.046 msecs
Elapsed time: 521.27 msecs
Run 2
Elapsed time: 669.385 msecs
Elapsed time: 712.15 msecs
Elapsed time: 518.502 msecs
Run 3
Elapsed time: 673.108 msecs
Elapsed time: 745.688 msecs
Elapsed time: 542.837 msecs
Run 4
Elapsed time: 654.196 msecs
Elapsed time: 723.074 msecs
Elapsed time: 506.861 msecs
Note that you are not exactly comparing apples to apples since
group-by-red is returning maps whose values are unrealized lazy sequence
(concats of concats of ocncats of vactors) instead of vectors
(defn group-by-red [f coll]
(r/fold 2048
(partial merge-with into)
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
coll))
Run 0
Elapsed time: 763.455 msecs
Elapsed time: 763.501 msecs
Elapsed time: 681.075 msecs
Run 1
Elapsed time: 645.52 msecs
Elapsed time: 731.545 msecs
Elapsed time: 476.381 msecs
Run 2
Elapsed time: 660.775 msecs
Elapsed time: 728.19 msecs
Elapsed time: 475.543 msecs
Run 3
Elapsed time: 657.255 msecs
Elapsed time: 725.995 msecs
Elapsed time: 494.038 msecs
Run 4
Elapsed time: 647.53 msecs
Elapsed time: 731.085 msecs
Elapsed time: 538.649 msecs
hth,
Christophe
On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi
balin...@gmail.comjavascript:
wrote:
BTW I understood the most about reducers (still not quite there yet,
though :) ) from Rich Hickey's talk at EuroClojure:
https://vimeo.com/45561411
On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
Hey,
Reducers is fascinating and quite complex at the same time. I feel
like best practices around it has not quite solidified yet.
Here is how I made your example work:
(ns group-by-reducers.core
(:require [clojure.core.reducers :as r :only [fold reduce map]])
(:require [criterium.core :as c]))
(defn group-by-naive [f coll]
(reduce
(fn [groups a]
(assoc groups (f a) (conj (get groups a []) a)))
{}
coll))
(defn group-by-red [f coll]
(letfn [(reduce-groups
([] {})
([g1 g2]
(merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
(defn -main
[ args]
(c/quick-bench (group-by #(mod % 29) (range 1)))
(c/quick-bench (group-by-naive #(mod % 29) (range 1)))
(c/quick-bench (group-by-red
Re: [lein-cljsbuild] exclude a cljs from compilation
Hi Evan,
Brenton Ashworth said that this kind of options should not be added to the
compiler, but to the tools that use the compiler.
Could yowl please give me some advice from where to start patching
lein-cljsbuild following Brenton advice?
Thank so much
Mimmo
Here is Brenton comment on :exclude option.
http://dev.clojure.org/jira/browse/CLJS-419?focusedCommentId=30185page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-30185
On Dec 7, 2012, at 7:04 AM, Evan Mezeske emeze...@gmail.com wrote:
Yeah, I saw that. Very cool stuff! Glad you could contribute.
On Thursday, December 6, 2012 7:14:21 PM UTC-8, Mimmo Cosenza wrote:
Hi Evan,
just to let you that I submitted on clojurescript repo the patch to exclude a
file and/or a directory of cljs from compilation.
the new compiler option is :exclude and accepts both a name of a file or
directory o a vector of files and/or directories.
Hope it helps
mimmo
On Nov 16, 2012, at 10:39 AM, Mimmo Cosenza mimmo@gmail.com wrote:
On Friday, November 16, 2012 8:28:18 AM UTC+1, Evan Mezeske wrote:
If you wouldn't mind, please create a new issue, or maybe just add to this
one: https://github.com/emezeske/lein-cljsbuild/issues/108 .
done. added issue #157 with reference to #108 and viceversa.
thanks
mimmo
Thanks,
-Evan
On Thursday, November 15, 2012 6:56:45 AM UTC-7, Mimmo Cosenza wrote:
Hi all,
my question is the following:
- I like to have the usual brepl connect to the browser during development
- I like to have the same source-base (e.g. :source-path src/cljs) for
both the development and production builds (e.g. :builds {:prod {.}
{:dev {.} of :cljsbuild keyword)
Is there a way in lein-cljsbuild to reach this goal? I'd like to write
something like this in :prod build
(defproject
:cljsbuild {:builds
{:dev {:source-path src/cljs
:compiler :output-to
resources/public/js/main_debug.js
:optimizations :whitespace
:pretty-print true}}
:prod {:source-path src/cljs
;;; here a key/values to exclude some file in
:source-path from compilation, something like :exclude [afil.cljs
another-file.cljs]
:compiler :output-to
resources/public/js/main.js
:optimizations :advanced}})
Thanks for the attention
Mimmo
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
[ANN] New functional programming group (Western Massachusetts)
With help from some friends, I am starting a new FP group here in Western Massachusetts: http://www.meetup.com/Functional-Programming-Connoisseurs/ From the meetup.com description: Open to any and all that have functional and logical tastes in programming languages and data models. Clojure, Scala, Erlang, Schemes, Prologs, F#, MLs, concatenative (e.g. Factor), and so on. Many of us have experience and interest in building distributed, large-scale systems. Many of us also happen to be independent contractors, solo/entrepreneurs, business owners, and otherwise interested in building killer businesses as well as awesome software. Depending on the week/month, expect a mix of free-for-all conversation and hacking, lightning talks, and the occasional prepared presentation. Our first meeting will be in January (date and location TBD). If you're in the area and on this list, stop by and say 'hi'! Of course, if you would like to give a relevant talk, presentation, or demonstration, by all means shoot me an email. Cheers, - Chas -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [lein-cljsbuild] exclude a cljs from compilation
I chimed in on the patch. Taking Brenton's approach sounds like it would be
significantly simpler and require a very small patch to lein-cljsbuild.
On Fri, Dec 7, 2012 at 1:24 PM, Mimmo Cosenza mimmo.cose...@gmail.comwrote:
Hi Evan,
Brenton Ashworth said that this kind of options should not be added to the
compiler, but to the tools that use the compiler.
Could yowl please give me some advice from where to start
patching lein-cljsbuild following Brenton advice?
Thank so much
Mimmo
Here is Brenton comment on :exclude option.
http://dev.clojure.org/jira/browse/CLJS-419?focusedCommentId=30185page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-30185
On Dec 7, 2012, at 7:04 AM, Evan Mezeske emeze...@gmail.com wrote:
Yeah, I saw that. Very cool stuff! Glad you could contribute.
On Thursday, December 6, 2012 7:14:21 PM UTC-8, Mimmo Cosenza wrote:
Hi Evan,
just to let you that I submitted on clojurescript repo the patch to
exclude a file and/or a directory of cljs from compilation.
the new compiler option is :exclude and accepts both a name of a file or
directory o a vector of files and/or directories.
Hope it helps
mimmo
On Nov 16, 2012, at 10:39 AM, Mimmo Cosenza mimmo@gmail.com wrote:
On Friday, November 16, 2012 8:28:18 AM UTC+1, Evan Mezeske wrote:
If you wouldn't mind, please create a new issue, or maybe just add to
this one:
https://github.com/**emezeske/lein-cljsbuild/**issues/108https://github.com/emezeske/lein-cljsbuild/issues/108.
done. added issue #157 with reference to #108 and viceversa.
thanks
mimmo
Thanks,
-Evan
On Thursday, November 15, 2012 6:56:45 AM UTC-7, Mimmo Cosenza wrote:
Hi all,
my question is the following:
- I like to have the usual brepl connect to the browser during
development
- I like to have the same source-base (e.g. :source-path src/cljs)
for both the development and production builds (e.g. :builds {:prod {.}
{:dev {.} of :cljsbuild keyword)
Is there a way in lein-cljsbuild to reach this goal? I'd like to write
something like this in :prod build
(defproject
:cljsbuild {:builds
{:dev {:source-path src/cljs
:compiler :output-to resources/public/js/main_
**debug.js
:optimizations :whitespace
:pretty-print true}}
:prod {:source-path src/cljs
;;; here a key/values to exclude some file
in :source-path from compilation, something like :exclude [afil.cljs
another-file.cljs]
:compiler :output-to
resources/public/js/main.js
:optimizations :advanced}})
Thanks for the attention
Mimmo
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+u...@**googlegroups.com
For more options, visit this group at
http://groups.google.com/**group/clojure?hl=enhttp://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: [ANN] sublime-lispindent, clojure indenting for sublime text 2
I opened a report on ST2's userecho a long time ago about the terrible indentation. If you'd like to see if fixed in Sublime Text 2 itself, please upvote: http://sublimetext.userecho.com/topic/98139-clojure-auto-indentation-is-almost-never-correct/ On Monday, November 12, 2012 10:25:38 AM UTC-6, Jonathan Fischer Friberg wrote: Dear clojure mailing list, As the indenting for clojure (and lisp in general) was very lacking in sublime, I decided to make a plugin: https://github.com/odyssomay/sublime-lispindent I hope someone finds this useful. By the way, if someone with a mac could try the keyboard shortcuts that would be great. I don't own a mac so I cannot test them. Sincerely, Jonathan -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: How was the skills matters conference in London (6. Dec)?
'Late Clojurex drinks' was an incredible talk. It changed the very way I view and interact with the the world. On another note all the recordings are now actually up! I particularly enjoyed Sam Aaron's one http://skillsmatter.com/podcast/home-27/live-programming-with-clojure/mh-6189 On Friday, December 7, 2012 12:05:26 PM UTC, john wrote: Thanks a lot Thomas ... I am looking forward to the not yet published podcast 18:00 - late Clojurex drinks Am Freitag, 7. Dezember 2012 12:33:20 UTC+1 schrieb Thomas: It was really great, all the videos are up already: http://skillsmatter.com/event/scala/clojure-exchange-2012 Have a look. Thomas On Friday, December 7, 2012 10:59:32 AM UTC, john wrote: Hi, I couldn't make it to the skill matters conference in London ))-: just being curious,... how was it? Any news? was there a main theme? Many Greetings John -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
let binding for functions
I'm a clojure newbie. Why does the compiler complain about being unable to resolve cube-root-recur on line 17? thanks 1 (def cube-root 2 (fn [n] 3 (let 4 [abs 5(fn [n] 6 (if ( n 0) n (- 0 n))) 7good-enough? 8(fn [guess] 9 ( (abs (- (* guess guess guess) n)) 0.0001)) 10improve-guess 11(fn [guess] 12 (/ (+ (/ n (* guess guess)) (* 2 guess)) 3)) 13cube-root-recur 14(fn [guess] 15 (cond 16(good-enough? guess) guess 17:else (cube-root-recur (improve-guess guess] 18 (cube-root-recur 1 -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
A Practical Optional Type System for Clojure - Final Honours Dissertation
Hi everyone, Very pleased to have handed in my final honours dissertation today. Thanks to all that submitted corrections and offered encouragement. https://github.com/downloads/frenchy64/papers/ambrose-honours.pdf Cheers, Ambrose -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [ANN] New functional programming group (Western Massachusetts)
On Friday, December 7, 2012 at 2:35 PM, Chas Emerick wrote: With help from some friends, I am starting a new FP group here in Western Massachusetts: I just thought that I would mention that I've also recently started a FP group in Providence, Rhode Island; our second meeting is on Jan. 28 at Swipely's offices. http://www.meetup.com/Providence-Functional-Programmers/ - Chris -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: A Practical Optional Type System for Clojure - Final Honours Dissertation
Congrats! :) David On Fri, Dec 7, 2012 at 7:33 AM, Ambrose Bonnaire-Sergeant abonnaireserge...@gmail.com wrote: Hi everyone, Very pleased to have handed in my final honours dissertation today. Thanks to all that submitted corrections and offered encouragement. https://github.com/downloads/frenchy64/papers/ambrose-honours.pdf Cheers, Ambrose -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
abysmal multicore performance, especially on AMD processors
I've been running compute intensive (multi-day), highly parallelizable Clojure processes on high-core-count machines and blithely assuming that since I saw near maximal CPU utilization in top and the like that I was probably getting good speedups. But a colleague recently did some tests and the results are really quite alarming. On intel machines we're seeing speedups but much less than I expected -- about a 2x speedup going from 1 to 8 cores. But on AMD processors we're seeing SLOWDOWNS, with the same tests taking almost twice as long on 8 cores as on 1. I'm baffled, and unhappy that my runs are probably going slower on 48-core and 64-core nodes than on single-core nodes. It's possible that I'm just doing something wrong in the way that I dispatch the tasks, or that I've missed some Clojure or JVM setting... but right now I'm mystified and would really appreciate some help. I'm aware that there's overhead for multicore distribution and that one can expect slowdowns if the computations that are being distributed are fast relative to the dispatch overhead, but this should not be the case here. We're distributing computations that take seconds or minutes, and not huge numbers of them (at least in our tests while trying to figure out what's going on). I'm also aware that the test that produced the data I give below, insofar as it uses pmap to do the distribution, may leave cores idle for a bit if some tasks take a lot longer than others, because of the way that pmap allocates cores to threads. But that also shouldn't be a big issue here because for this test all of the threads are doing the exact same computation. And I also tried using an agent-based dispatch approach that shouldn't have the pmap thread allocation issue, and the results were about the same. Note also that all of the computations in this test are purely functional and independent -- there shouldn't be any resource contention issues. The test: I wrote a time-consuming function that just does a bunch of math and list manipulation (which is what takes a lot of time in my real applications): (defn burn ([] (loop [i 0 value '()] (if (= i 1) (count (last (take 1 (iterate reverse value (recur (inc i) (cons (* (int i) (+ (float i) (- (int i) (/ (float i) (inc (int i)) value) ([_] (burn))) Then I have a main function like this: (defn -main [ args] (time (doall (pmap burn (range 8 (System/exit 0)) We run it with lein run (we've tried both leingingen 1.7.1 and 2.0.0-preview10) with Java 1.7.0_03 Java HotSpot(TM) 64-Bit Server VM. We also tried Java 1.6.0_22. We've tried various JVM memory options (via :jvm-opts with -Xmx and -Xms settings) and also with and without -XX:+UseParallelGC. None of this seems to change the picture substantially. The results that we get generally look like this: - On an Intel Core i7 3770K with 8 cores and 16GB of RAM, running the code above, it takes about 45 seconds (and all cores appear to be fully loaded as it does so). If we change the pmap to just plain map, so that we use only a single core, the time goes up to about 1 minute and 36 seconds. So the speedup for 8 cores is just about 2x, even though there are 8 completely independent tasks. So that's pretty depressing. - But much worse: on a 4 x Opteron 6272 with 48 cores and 32GB of RAM, running the same test (with pmap) takes about 4 minutes and 2 seconds. That's really slow! Changing the pmap to map here produces a runtime of about 2 minutes and 20 seconds. So it's quite a bit faster on one core than on 8! And all of these times are terrible compared to those on the intel. Another strange observation is that we can run multiple instances of the test on the same machine and (up to some limit, presumably) they don't seem to slow each other down, even though just one instance of the test appears to be maxing out all of the CPU according to top. I suppose that means that top isn't telling me what I thought -- my colleague says it can mean that something is blocked in some way with a full instruction queue. But I'm not interested in running multiple instances. I have single computations that involve multiple expensive but independent subcomputations, and I want to farm those subcomputations out to multiple cores -- and get speedups as a result. My subcomputations are so completely independent that I think I should be able to get speedups approaching a factor of n for n cores, but what I see is a factor of only about 2 on intel machines, and a bizarre factor of about 1/2 on AMD machines. Any help would be greatly appreciated! Thanks, -Lee -- Lee Spector, Professor of Computer Science Cognitive Science, Hampshire College 893 West Street, Amherst, MA 01002-3359
Re: abysmal multicore performance, especially on AMD processors
Lee: I'll just give a brief description right now, but one thing I've found in the past on a 2-core machine that was achieving much less than 2x speedup was memory bandwidth being the limiting factor. Not all Clojure code allocates memory, but a lot does. If the hardware in a system can write at rate X from a multicore processor to main memory, and a single-threaded Clojure program writes to memory at rate 0.5*X, then the most speedup you will ever get out of multicore execution of the same code on N cores will be 2x, no matter how large N is. As one way to see if this is the problem, you could try changing your burn function so that instead of doing cons to build up a list result, first allocate a Java mutable array before the loop that is as large as you need it to be at the end, and write values into that. You can convert it to some other Clojure type at the end of the loop if you prefer. I have some C benchmark programs that test memory read and write bandwidth on single and multiple cores you can run on your Intel machine to see if that might be the issue. If this is the issue, I would expect to see at least a little speedup from 1 core to multiple cores, but capped at some maximum speedup that is determined by the memory bandwidth, not the number of cores you run in parallel. I don't currently have any guess about what might be happening with the AMD multicore machine. If you are interested in wild guessing, perhaps there could be some kind of multicore cache coherency protocol that is badly configured, causing cache lines to be frequently invalidated when multiple cores are sharing memory? That would make more sense if multiple cores were reading from and writing to the same cache lines, which doesn't seem terribly likely for a typical Clojure program. Let me know if you are interested and I will find those C programs for you to try out. I got them from somewhere on the Internet and may have tweaked them a little bit. Andy On Dec 7, 2012, at 5:25 PM, Lee Spector wrote: I've been running compute intensive (multi-day), highly parallelizable Clojure processes on high-core-count machines and blithely assuming that since I saw near maximal CPU utilization in top and the like that I was probably getting good speedups. But a colleague recently did some tests and the results are really quite alarming. On intel machines we're seeing speedups but much less than I expected -- about a 2x speedup going from 1 to 8 cores. But on AMD processors we're seeing SLOWDOWNS, with the same tests taking almost twice as long on 8 cores as on 1. I'm baffled, and unhappy that my runs are probably going slower on 48-core and 64-core nodes than on single-core nodes. It's possible that I'm just doing something wrong in the way that I dispatch the tasks, or that I've missed some Clojure or JVM setting... but right now I'm mystified and would really appreciate some help. I'm aware that there's overhead for multicore distribution and that one can expect slowdowns if the computations that are being distributed are fast relative to the dispatch overhead, but this should not be the case here. We're distributing computations that take seconds or minutes, and not huge numbers of them (at least in our tests while trying to figure out what's going on). I'm also aware that the test that produced the data I give below, insofar as it uses pmap to do the distribution, may leave cores idle for a bit if some tasks take a lot longer than others, because of the way that pmap allocates cores to threads. But that also shouldn't be a big issue here because for this test all of the threads are doing the exact same computation. And I also tried using an agent-based dispatch approach that shouldn't have the pmap thread allocation issue, and the results were about the same. Note also that all of the computations in this test are purely functional and independent -- there shouldn't be any resource contention issues. The test: I wrote a time-consuming function that just does a bunch of math and list manipulation (which is what takes a lot of time in my real applications): (defn burn ([] (loop [i 0 value '()] (if (= i 1) (count (last (take 1 (iterate reverse value (recur (inc i) (cons (* (int i) (+ (float i) (- (int i) (/ (float i) (inc (int i)) value) ([_] (burn))) Then I have a main function like this: (defn -main [ args] (time (doall (pmap burn (range 8 (System/exit 0)) We run it with lein run (we've tried both leingingen 1.7.1 and 2.0.0-preview10) with Java 1.7.0_03 Java HotSpot(TM) 64-Bit Server VM. We also tried Java 1.6.0_22. We've tried various JVM memory options (via
Re: abysmal multicore performance, especially on AMD processors
Thanks Andy. My applications definitely allocate a lot of memory, which is reflected in all of that consing in the test I was using. It'd be hard to do what we do in any other way. I can see how a test using a Java mutable array would help to diagnose the problem, but if that IS the problem then it sounds like there'd be no solution short of radically re-engineering our systems. That would be sad! My colleague who's running the tests is out for the weekend but we'll talk early next week and get back to you if we want to try your C programs etc. -Lee On Dec 7, 2012, at 8:41 PM, Andy Fingerhut wrote: Lee: I'll just give a brief description right now, but one thing I've found in the past on a 2-core machine that was achieving much less than 2x speedup was memory bandwidth being the limiting factor. Not all Clojure code allocates memory, but a lot does. If the hardware in a system can write at rate X from a multicore processor to main memory, and a single-threaded Clojure program writes to memory at rate 0.5*X, then the most speedup you will ever get out of multicore execution of the same code on N cores will be 2x, no matter how large N is. As one way to see if this is the problem, you could try changing your burn function so that instead of doing cons to build up a list result, first allocate a Java mutable array before the loop that is as large as you need it to be at the end, and write values into that. You can convert it to some other Clojure type at the end of the loop if you prefer. I have some C benchmark programs that test memory read and write bandwidth on single and multiple cores you can run on your Intel machine to see if that might be the issue. If this is the issue, I would expect to see at least a little speedup from 1 core to multiple cores, but capped at some maximum speedup that is determined by the memory bandwidth, not the number of cores you run in parallel. I don't currently have any guess about what might be happening with the AMD multicore machine. If you are interested in wild guessing, perhaps there could be some kind of multicore cache coherency protocol that is badly configured, causing cache lines to be frequently invalidated when multiple cores are sharing memory? That would make more sense if multiple cores were reading from and writing to the same cache lines, which doesn't seem terribly likely for a typical Clojure program. Let me know if you are interested and I will find those C programs for you to try out. I got them from somewhere on the Internet and may have tweaked them a little bit. Andy -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: abysmal multicore performance, especially on AMD processors
On Dec 7, 2012, at 5:25 PM, Lee Spector wrote: Another strange observation is that we can run multiple instances of the test on the same machine and (up to some limit, presumably) they don't seem to slow each other down, even though just one instance of the test appears to be maxing out all of the CPU according to top. I suppose that means that top isn't telling me what I thought -- my colleague says it can mean that something is blocked in some way with a full instruction queue. But I'm not interested in running multiple instances. I have single computations that involve multiple expensive but independent subcomputations, and I want to farm those subcomputations out to multiple cores -- and get speedups as a result. My subcomputations are so completely independent that I think I should be able to get speedups approaching a factor of n for n cores, but what I see is a factor of only about 2 on intel machines, and a bizarre factor of about 1/2 on AMD machines. Lee: When you say we can run multiple instances of the test on the same machine, do you mean that, for example, on an 8 core machine you run 8 different JVMs in parallel, each doing a single-threaded 'map' in your Clojure code and not a 'pmap'? And what kinds of speedups does that achieve? The results could help indicate whether the problem you are seeing is due to the hardware/OS, or something about multiple threads within a single JVM. Andy -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
