Re: [ANN] 1st public release of thi.ng geometry toolkit (CLJ CLJS)
Hey, Karsten great stuff and congrats! I haven't been following toxic closely for quite some time, sadly but does that mean it's been discontinued completely and you are working in clojure full time? :) What's your impression on working with big code base as above library in clojure vs java? Does clojure scales well? I'm particularly interested in functional vs OO approach and If any performance critical stuff was done in java? If you could share some thoughts would be great. Thanks, Kuba -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Penumbra anyone in clojure 1.5.1
Hi there,
I'm trying to get Penumbra working with clojure 1.5.1 and was wondering if
there is still anyone using that library...?
Also I've just discovered a fork of penumbra in the clojars
(https://clojars.org/prismofeverything/penumbra/versions/0.6.12) which
claims to be compatible with clojure 1.3, adn address some changes to
lein2. Is that version preferable to the original git? So far I have had
not luck with any.
The ('use penumbra.opengl) line gives the following error:
CompilerException java.lang.ClassNotFoundException: penumbra.opengl,
compiling:(NO_SOURCE_PATH:1:1)
Does that mean the native library can not be found or perhaps I'm missing
something in project.clj?
:dependencies [[org.clojure/clojure 1.5.1]
[prismofeverything/penumbra 0.6.12]]
:jvm-opts [-Djava.library.path=./target/native/linux]
Checking ./target/native/linux indicates some native lib do exist there,
but ./target/classes folder is empty. I assume all the jars are loaded now
from ~/.m2/repository/prismofeverything/penumbra/0.6.12 (in my case)
Thanks for help,
kuba
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Penumbra anyone in clojure 1.5.1
ok I think got it working at least black frame shows up.
On Monday, January 27, 2014 12:51:45 PM UTC-8, Kuba Roth wrote:
Hi there,
I'm trying to get Penumbra working with clojure 1.5.1 and was wondering if
there is still anyone using that library...?
Also I've just discovered a fork of penumbra in the clojars (
https://clojars.org/prismofeverything/penumbra/versions/0.6.12) which
claims to be compatible with clojure 1.3, adn address some changes to
lein2. Is that version preferable to the original git? So far I have had
not luck with any.
The ('use penumbra.opengl) line gives the following error:
CompilerException java.lang.ClassNotFoundException: penumbra.opengl,
compiling:(NO_SOURCE_PATH:1:1)
Does that mean the native library can not be found or perhaps I'm missing
something in project.clj?
:dependencies [[org.clojure/clojure 1.5.1]
[prismofeverything/penumbra 0.6.12]]
:jvm-opts [-Djava.library.path=./target/native/linux]
Checking ./target/native/linux indicates some native lib do exist there,
but ./target/classes folder is empty. I assume all the jars are loaded now
from ~/.m2/repository/prismofeverything/penumbra/0.6.12 (in my case)
Thanks for help,
kuba
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
string/replace regex help
Hi, This is more of a regex specific question the clojure, but anyway... I have the a multiline string as follow: (def myText aSTART END ee ff ggg START ii E END ) .. and I need to replace all text between START END 'keywords' blocks. The following regex: (clojure.string/replace myText #START([^END])* ==) does sort of what I'm hoping to get but does not guard me against cases when one of the END (in this case E) characters shows up in the middle of the block to be cut. So my question comes down to what am I missing here? :) Any suggestions are highly appreciated. Thanks, kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: string/replace regex help
Sweet! Thanks so much! On 2014-01-16, at 1:17 PM, John Wiseman jjwise...@gmail.com wrote: (clojure.string/replace myText #(?s)START.*?END ==) ;; a==\nee\nff\nggg\n==\n\n (?s) specifies multi-line mode, *? is the non-greedy form of *. On Thu, Jan 16, 2014 at 1:02 PM, Kuba Roth kuba.r...@gmail.com wrote: Hi, This is more of a regex specific question the clojure, but anyway... I have the a multiline string as follow: (def myText aSTART END ee ff ggg START ii E END ) .. and I need to replace all text between START END 'keywords' blocks. The following regex: (clojure.string/replace myText #START([^END])* ==) does sort of what I'm hoping to get but does not guard me against cases when one of the END (in this case E) characters shows up in the middle of the block to be cut. So my question comes down to what am I missing here? :) Any suggestions are highly appreciated. Thanks, kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/VOz4piaP5mE/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
let bindings in tree-seq funciton
I'm currently working with tree-seq function from clojure.core and there is one thing I'm struggling to understand... Could you please explain why pretty much all of the body of the function sits within 'let binding' and not in the 'let body' where only (walk root) is placed? Has it something to do with tail recursion? Thank you, Kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Usung atom in pmap
I have a few doubts how to approach updating values inside pmap and a general
question if map is ideal to solve the problem. I'd really appreciate to get a
review of the following code.
The file-seqC and file-seqB is slightly modified version of file-seq. The plan
was to spawn subfolders of the root folder into concurrent jobs, count files
and output final total number. In my initial tests (file-seqB) I was using atom
which gets updated as soon as each thread completes. It sort of worked although
was giving me some inconsistent results somewhere around total number but
seemed quite there. This looked to me as the last thread sometimes have not
completed on time
So I've come up with another version (file-seqC). First the pmap returned the
list of total files for each thread and then I reduced that to get the final
result. This approach is solid and I'm quite happy to figure this out but I'm
curious if it's possible to implement files-seqB using Atoms. I'm still not
sure how would I know if all the entries in the pmap completes before the
output is passed further. Shout this used agents instead and completely drop
the idea of using pmap?
In the example below I've included file-seqB function for reference.
Thanks,
Kuba
(ns pdir.core
(:gen-class))
(defn burn []
(dotimes [i 1000] ;; Make it slow
(reduce * (map float (take 1000 (iterate inc i)
)
; comment out due to problems with uberjar
; Unable to resolve symbol: ttt in this context, compiling:(pdir/core.clj:35:6);
;
;(defn file-seqB
; A tree seq on java.io.Files
; {:added 1.0}
; [dir]
;
; ;(burn) ;; SLOW things down for testing
;
;(def total (tree-seq
; (fn [^java.io.File f] (. f (isDirectory)))
; (fn [^java.io.File d] (seq (. d (listFiles
; dir))
;
;; takes the number of the files in the directory and update the atom
;
; (swap! ttt + (count total)) ; this gives sort of predictable results
; ; but still not accurate
;
;
;(println Done: dir (count total) @ttt)
;
;)
(defn file-seqC
A tree seq on java.io.Files
{:added 1.0}
[dir]
(let [total ; Need to wrap output of the tree-seeq into local
variable
(tree-seq ; Otherwise using pmap returns won't complete
before
; (reduce) function kicks in
(fn [^java.io.File f] (. f (isDirectory)))
(fn [^java.io.File d] (seq (. d (listFiles
dir) ]
;(burn) ;; SLOW things down for testing
(println Done: dir (count total))
(count total) ;Returns a list with summed values
)
)
(defn -main [ args]
(def rootPath (nth args 0))
(println Root: rootPath)
(def f (clojure.java.io/file rootPath))
;(println (seq (. f (listFiles
(def subDirs (seq (. f (listFiles ; store the first depth of folders
; and run them concurrently
(- (pmap #(file-seqC %1) subDirs ,,,); returns list of total files in
each folder
(reduce + ,,,); sum them up
(println Total Files: ,,,) ; print
)
;(shutdown-agents) ;terminate JVM which linger in command line
)
(-main /data/temp/kuba/aaa)
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: let bindings in tree-seq funciton
Thanks, very interesting point. It's a shame this is not available in the clojure docs - very useful for newcomers as me. Is it fair to say that let bindings is good/recomended to put all the code and the main body is used primarily for returning results of the function? Cheers, Kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: send agent inside doseq
Yup, I've done OOP in the past but probably even closer procedural programming. Recently I've been working pretty much exclusively in python/C++ which are somewhat at two extremes. I'm hoping to see Clojure to be a blend of these two and a replacement especially in areas where things in python are too slow or impossible (parallel stuff). The functional approach of Clojure is the down side for me at the moment ;). The lack of loops and variables is a bit mind-bending, feels like going back to start with programming in general. kuba On Tue, Aug 20, 2013 at 10:33 PM, Sean Corfield seancorfi...@gmail.comwrote: On Tue, Aug 20, 2013 at 9:31 PM, Kuba Roth kuba.r...@gmail.com wrote: The reason I looked into 'intern' can only be explained by totally lack of experience in Clojure and more general functional programming. Ah, is your background OOP? You'll find the functional world is pretty different. No variables in the traditional sense, no uncontrolled mutation, no loops in the traditional sense, no state in the OOP sense. def and defn only create top-level definitions - bindings of values to symbols - and intern is really an implementation detail that you want to keep away from. As you can see from my example, you can create a sequence of agents easily enough as a data structure, and then you can perform operations on that data structure. Agents provide controlled mutation in that they start out with an initial value and you send them a series of function invocations and they take on new values as each function invocation is applied asynchronously. I wonder what you might be trying to achieve? Or perhaps this is just initial experimentation? One thing you'll find about Clojure is how rarely you need mutation. At World Singles we have about 18,000 lines of Clojure with just two agents (used to provide measured asynchronous DB updates for a couple of very specific situations) and 27 atoms, which are nearly all just caches - only half a dozen of those are really mutable state (and we're consolidating that). Welcome to Clojure! -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/yVOVvgq88hE/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
send agent inside doseq
Hi there, I've got a range of values and I'd like to run agents for each value per thread. For some reason I've got only one agents being updated. Not sure what's wrong here but I suspect must be doing something terrible stupid... Thanks! (doseq [s (range 30 35)] ;(println (format _%s s)) (intern *ns* (symbol (format _%s s) ) (agent s)) ;; set initial value (send @(intern *ns* (symbol (format _%s s))) + 100) ;; send agent and update value (println @(intern *ns* (symbol (format _%s s)) ));; deref ) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: send agent inside doseq
Thanks Sean, your example looks much cleaner and most important works! The reason I looked into 'intern' can only be explained by totally lack of experience in Clojure and more general functional programming. My goal was to dynamically create a var inside the 'doseq' and apparently 'intern' is used in this cases. The 'def' special form didn't work for me... (not sure why). But as I said it was rather my lame attempt to make things work then any well thought out decision :) Thanks, Kuba On Tuesday, August 20, 2013 4:56:30 PM UTC-7, Sean Corfield wrote: Very likely Juan, as seen here: (let [agents (map agent (range 30 35))] (doseq [a agents] (send a + 100) (println @a)) (doseq [a agents] (println @a))) For me that prints: 30 31 32 33 34 130 131 132 133 134 but I suspect that's more luck that anything since there's no reason the agent operations need to have finished before the second println. Kuba, as a separate issue, why are you trying to intern all those symbols? It's not very idiomatic. Sean On Tue, Aug 20, 2013 at 4:49 PM, juan.facorro juan.f...@gmail.comjavascript: wrote: If on the println you don't see the value updated, it's probably because the operation sent to the agent wasn't applied yet. Add a (Thread/sleep 500) in between the send and println expressions and you'll see the expected agents'. Cheers, JF On Wednesday, August 21, 2013 12:21:59 AM UTC+1, Kuba Roth wrote: Hi there, I've got a range of values and I'd like to run agents for each value per thread. For some reason I've got only one agents being updated. Not sure what's wrong here but I suspect must be doing something terrible stupid... Thanks! (doseq [s (range 30 35)] ;(println (format _%s s)) (intern *ns* (symbol (format _%s s) ) (agent s)) ;; set initial value (send @(intern *ns* (symbol (format _%s s))) + 100) ;; send agent and update value (println @(intern *ns* (symbol (format _%s s)) ));; deref ) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.comjavascript: Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com javascript: For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/groups/opt_out. -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
pmap thread oversubscription OSX
Hi there, I'm currently testing a map/pmap performance and run into weird behavior in Clojure running on a OSX (compered to Linux). Running a 'slow' (processor intensive pmap function on OSX suffers from very poor performance whereas the same function on Linux runs just fine. At the first glance looks like there is a oversubsription of threads taking place on OSX and I don't fully understand if the problem is part of my code or is this perhaps a bug on OSX platform. (Clojure 1.4) Here is the slow function, I need to stress it that using Thread/ sleep to create dummy long running job doesn't seem to cause this problem. Realizing long sequence made a trick for me and force all procsc to run on 100%. ; Processor intensive process which will run in parallel ; Notice Thread/sleep is not used - since it doesn't use processor (more appropriate for imitating I/O stuff) (defn myFunc [input] ;(Thread/sleep 500) ;; this doesn't use processor (doseq [n (take 1000 (range))]) ;; some slow dummy process (println (str --- input)) ;; some side effect to track down roughly the progress ) ...and the results: OSX - dual-core: (time (doall (pmap #(myFunc %) (take 4 (range) Elapsed time: 30263.3 msecs (time (doall (map #(myFunc %) (take 4 (range) Elapsed time: 9239.3 msecs Linux 12core - the vector to iterate over has been increased to avoid tha case where there are more cores then tasks: (time (doall (pmap #(myFunc %) (take 100 (range)Elapsed time: 16408.8 msecs (time (doall (map #(myFunc %) (take 100 (range) Elapsed time: 150310.1 msecs I've got more examples for OSX which clearly shows that as soon as the number of tasks exceeds number of cores pmap performance suffers. It seems to me like there is no blocking taking place on threads and all the tasks are started at the same time. I've also run a test using Agents (pardon my code style) but the problem seems to show up there too. All tasks in the range gets submitted instantly causing oversubscribtion, so no task blocking is taking a place. ;forced single threaded (time (doseq [x (take 100 (range))] (send (agent 0) (myFunc x)); still single threaded... - I wonder if there is a better way to do that? )) ; parallel (time (doseq [x (take 100 (range))] ;(send (agent x) myFunc ) ; )) With Agents, on linux parallel was way faster whereas on OSX a magnitude slower: linux parallel thr - 1min 52s linux single thr - 9min 35s Thanks, Kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
pmap oversubscription on OSX
Hi there, I'm a bit nervous, this is my first post here :) I'm currently testing a map/pmap performance and run into weird behavior in Clojure running on a OSX (compered to Linux). Running a 'slow' (processor intensive pmap function on OSX suffers from very poor performance whereas the same function on Linux runs just fine. At the first glance looks like there is a oversubsription of threads taking place on OSX and I don't fully understand if the problem is part of my code or is this perhaps a bug on OSX platform. (Clojure 1.4) Here is the slow function, I need to stress it that using Thread/sleep to create dummy long running job doesn't seem to cause this problem. Realizing long sequence made a trick for me and force all procsc to run on 100%. ; Processor intensive process which will run in parallel ; Notice Thread/sleep is not used - since it doesn't use processor (more appropriate for imitating I/O stuff) (defn myFunc [input] ;(Thread/sleep 500) ;; this doesn't use processor (doseq [n (take 1000 (range))]) ;; some slow dummy process (println (str --- input)) ;; some side effect to track down roughly the progress ) ...and the results: OSX - dual-core: (time (doall (pmap #(myFunc %) (take 4 (range) Elapsed time: 30263.3 msecs (time (doall (map #(myFunc %) (take 4 (range) Elapsed time: 9239.3 msecs Linux 12core - the vector to iterate over has been increased to avoid tha case where there are more cores then tasks: (time (doall (pmap #(myFunc %) (take 100 (range)Elapsed time: 16408.8 msecs (time (doall (map #(myFunc %) (take 100 (range) Elapsed time: 150310.1 msecs I've got more examples for OSX which clearly shows that as soon as the number of tasks exceeds number of cores pmap performance suffers. It seems to me like there is no blocking taking place on threads and all the tasks are started at the same time. I've also run a test using Agents (pardon my code style) but the problem seems to show up there too. All tasks in the range gets submitted instantly causing oversubscribtion, so no task blocking is taking a place. ;forced single threaded (time (doseq [x (take 100 (range))] (send (agent 0) (myFunc x)); still single threaded... - I wonder if there is a better way to do that? )) ; parallel (time (doseq [x (take 100 (range))] ;(send (agent x) myFunc ) ; )) With Agents, on linux parallel was way faster whereas on OSX a magnitude slower: linux parallel thr - 1min 52s linux single thr - 9min 35s Thanks, Kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: pmap thread oversubscription OSX
Thanks all for advise and sorry for the double post. Will check this out tonight and get back. Cheers, kuba On Wednesday, May 1, 2013 11:09:05 AM UTC-7, kuba roth wrote: Hi there, I'm currently testing a map/pmap performance and run into weird behavior in Clojure running on a OSX (compered to Linux). Running a 'slow' (processor intensive pmap function on OSX suffers from very poor performance whereas the same function on Linux runs just fine. At the first glance looks like there is a oversubsription of threads taking place on OSX and I don't fully understand if the problem is part of my code or is this perhaps a bug on OSX platform. (Clojure 1.4) Here is the slow function, I need to stress it that using Thread/ sleep to create dummy long running job doesn't seem to cause this problem. Realizing long sequence made a trick for me and force all procsc to run on 100%. ; Processor intensive process which will run in parallel ; Notice Thread/sleep is not used - since it doesn't use processor (more appropriate for imitating I/O stuff) (defn myFunc [input] ;(Thread/sleep 500) ;; this doesn't use processor (doseq [n (take 1000 (range))]) ;; some slow dummy process (println (str --- input)) ;; some side effect to track down roughly the progress ) ...and the results: OSX - dual-core: (time (doall (pmap #(myFunc %) (take 4 (range) Elapsed time: 30263.3 msecs (time (doall (map #(myFunc %) (take 4 (range) Elapsed time: 9239.3 msecs Linux 12core - the vector to iterate over has been increased to avoid tha case where there are more cores then tasks: (time (doall (pmap #(myFunc %) (take 100 (range)Elapsed time: 16408.8 msecs (time (doall (map #(myFunc %) (take 100 (range) Elapsed time: 150310.1 msecs I've got more examples for OSX which clearly shows that as soon as the number of tasks exceeds number of cores pmap performance suffers. It seems to me like there is no blocking taking place on threads and all the tasks are started at the same time. I've also run a test using Agents (pardon my code style) but the problem seems to show up there too. All tasks in the range gets submitted instantly causing oversubscribtion, so no task blocking is taking a place. ;forced single threaded (time (doseq [x (take 100 (range))] (send (agent 0) (myFunc x)); still single threaded... - I wonder if there is a better way to do that? )) ; parallel (time (doseq [x (take 100 (range))] ;(send (agent x) myFunc ) ; )) With Agents, on linux parallel was way faster whereas on OSX a magnitude slower: linux parallel thr - 1min 52s linux single thr - 9min 35s Thanks, Kuba -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: pmap thread oversubscription OSX
On OSX (.availableProcessors (Runtime/getRuntime)) returns 2 in my case. I've also run into the following thread: https://groups.google.com/forum/#!topic/clojure/AtA-0LKYe9A There are two other implementations of CPU bound pmap: pmapall and pmap-pool. The latter one (by j-g-faustus) actually solved my problem on OSX. Attaching complete example for reference: (import '(java.util.concurrent Executors)) (defn pmap-pool [f coll] (let [queue (ref coll) ;; shared queue of work units nthreads (.availableProcessors (Runtime/getRuntime)) pool (Executors/newFixedThreadPool nthreads) tasks (map (fn [_] (fn [] ; one task per thread (let [local-res (atom [])] ;; collect results per thread to minimize synchronization (while (seq @queue) ;; queue may be emptied between 'while' ;; and 'dosync'. (when-let [wu (dosync ;; grab work unit, update queue (when-let [w (first @queue)] (alter queue next) w))] (swap! local-res conj (f wu local-res))) (range nthreads)) results (doall (map #(deref (.get %)) ;; blocks until completion (.invokeAll pool tasks))) ;; start all tasks results (reduce concat results)] (.shutdown pool) ;; sanity check (when-not (and (empty? @queue) (= (count results) (count coll)) (every? #(= % :done) results)) (println ERROR: queue (count @queue) #results (count results))) results)) (defn myFunc [input] (doseq [n (take 1000 (range))]) ;; some slow dummy process (println (str --- input)) ) (time (last (pmap-pool #(myFunc %) (take 6 (range) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Clojure for Beginners eBook
I would also add - setting up Eclipse with counterclockwise - native java libs setup (lwjgl) for OpenGL folks. Since I don't have java background this took me a while to figure out and Lein (which is a great tool btw) doesn't provide an intuitive support for this sort of stuff out of the box unfortunately. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
