[ANN] electron-app 1.0.0
I’ve just published a clj template for for an Electron application built with deps.edn, Figwheel Main, Reagent, and test integration via cljs-test-display: https://github.com/paulbutcher/electron-app -- paul.butcher->msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/EF92C90B-4E7F-4C5C-944A-728B024451CB%40paulbutcher.com.
Re: Benchmarking structural sharing
On 12 August 2014 at 13:49:42, Linus Ericsson (oscarlinuserics...@gmail.com) wrote: The conclusion of this is I think the easiest way to make this work is to just run the algorithm in both versions and watch the object allocation statistics closely in VisualVM or similar. Yeah, that's exactly what I was hoping I might be able to avoid. Ah well. My intuition is that the there will be a lot of copied arrays Indeed. But intuition often isn't a good guide when it comes to optimisation (hence my wish to measure and make decisions based on data instead of intuition). -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Benchmarking structural sharing
Is there any way to benchmark the degree of structural sharing achieved by a Clojure algorithm? I'm evaluating two different implementations of an algorithm, one which uses zippers and one which uses rrb-vector. It would be great if there were some way to quantify the degree to which they both achieved (or didn't) structural sharing. -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Success Stories w/US Government Projects
Out of interest, which aspect(s) of the EPL are your lawyers objecting to? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con On 9 August 2014 at 01:54:22, rcg (randy.goebb...@gmail.com) wrote: Hello; We are looking for Clojure success stories within the US Government. If you have a success story you'd be willing to share, this may counter or help resolve objections to the EPL license that have been raised by Lawyers. Our development group has had much success with Clojure. Unfortunately, we may be forced away from Clojure. An alternative would be to have the license modified (we understand that is probably a non-starter) or have a dual license setup for Clojure (with acceptable license). However, if there are other groups out there that have survived US Govt Lawyer Scrutiny of the EPL, that may be very useful. Thanks in advance for any help! -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: A few String functions we could implement in Clojure?
I recently hit exactly this question in a ClojureScript app I’m writing. It just so happens that Javascript provides a .indexOf method which is, as near as dammit, the same as the one provided by Java. So in this instance, portability isn’t an issue. But having said that, I would still prefer to see this supported natively in Clojure, even if it’s just a wrapper around the interop. Using interop to do things that are as generic as simple string manipulation just feels messy. -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con On 19 July 2014 at 16:49:33, Andy Fingerhut (andy.finger...@gmail.com) wrote: I would have to defer that question to someone who makes decisions regarding what goes into Clojure/ClojureScript, and what does not. Of course, anyone else is free to create libraries that try to make portability between those two platforms easier. Perhaps someone has already taken a go at creating such a thing? I haven't used ClojureScript myself yet, so haven't looked for anything in that area. Andy On Sat, Jul 19, 2014 at 8:23 AM, Bruce Durling b...@otfrom.com wrote: Andy, How much of this reasoning do you think changes when we starting thinking about being hosted on multiple platforms (I'm thinking specifically clojure/clojurescript and cljx)? cheers, Bruce On Sat, Jul 19, 2014 at 4:17 PM, Andy Fingerhut andy.finger...@gmail.com wrote: Pierre: I maintain the cheatsheet, and I put .indexOf and .lastIndexOf on there since they are probably the most common thing I saw asked about that is in the Java API but not the Clojure API, for strings. There are also links to whole Java classes and their entire API, e.g. for file I/O, for which there is no Clojure equivalent, since file I/O is a common need. Clojure is meant to be a hosted language, not hiding its host platform, but making it easily callable. If there are entire Java classes that meet very common needs that aren't mentioned on the cheatsheet, I would consider adding links to their documentation pages. I don't want to fill up the cheatsheet with many individual Java methods, though. As for why there are not Clojure equivalents of particular Java API methods, I think the reasoning might be similar (it has likely been discussed publicly, but I don't have a link handy) -- don't create a large number of Clojure functions that do nothing more than what the equivalent Java APIs do. Andy On Sat, Jul 19, 2014 at 3:58 AM, Pierre Masci mas...@gmail.com wrote: Hi all, just nit picking about Clojure's String API. I've been comparing it with Java's, and I noticed that (not surprisingly) they are very similar. There are just 2-3 functions that exist in Java but don't have an equivalent in Clojure. I was wondering if they could be worth adding to clojure.string : (.indexOf s c) and (.lastIndexOf c) (.startsWith s danc) and (.endsWith s ing) (.charAt s 5) same as (get s 5) but expresses a clearer intent. It's less general than (get) though as it only applies to Strings, so that might be unnecessary sugar. .indexOf and .lastIndexOf are indicated in the Clojure Cheatsheet, maybe .startsWith and .endsWith also deserve to be mentioned there? I've been wondering why some functions have been ported, like (lower-case) for (.toLowerCase), but not the ones mentioned above. I told you it was nit picking (^c^) Clojure's API is awesome as it is. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to
Re: Leiningen profile problem
Ah! Thanks Curtis - much appreciated.
Any insight as to why that’s not the default configuration?
--
paul.butcher-msgCount++
Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher
Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con
On 29 July 2014 at 01:19:17, Curtis Summers (curtis.summ...@gmail.com) wrote:
Paul,
Leiningen can isolate the target-paths by profile if you use :target-path in
your project.clj with the %s in it.
Per the sample project.clj file
(https://github.com/technomancy/leiningen/blob/master/sample.project.clj):
;; All generated files will be placed in :target-path. In order to avoid
;; cross-profile contamination (for instance, uberjar classes interfering
;; with development), it's recommended to include %s in in your custom
;; :target-path, which will splice in names of the currently active profiles.
:target-path target/%s/
--Curtis
On Monday, July 28, 2014 12:40:45 PM UTC-5, Paul Butcher wrote:
Ah! I thought that Leiningen put class files in different places for different
profiles, but it looks like I was mistaken.
Thanks for the EDN hint - that sounds like a good way to go.
Cheers!
--
paul.butcher-msgCount++
Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher
Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con
On 28 July 2014 at 18:01:16, Thomas Heller (th.h...@gmail.com) wrote:
Your issue is probably due to AOT compilation, uberjar aot compiles everything
and leaves alot of class files in your target directory (aka classpath). When
you run your app afterwards the class file is used instead of the clj file
since the .class file is newer. When you run lein clean those .class files
are removed and the classloader then looks for the clj file.
I'd recommend keeping a config file (can be EDN) and just having a development
and production version instead of using different code. You can either use a
command line argument to choose which config to load or use leinigen profiles
and resource paths so only include the one you want.
HTH,
/thomas
On Monday, July 28, 2014 3:45:51 PM UTC+2, Paul Butcher wrote:
Oops - I originally sent this to the ClojureScript group, which probably wasn’t
the best place. Apologies to those who subscribe to both lists for the spam:
I’m clearly misunderstanding something fundamental about how Leiningen profiles
work. I’d appreciate help fixing my understanding.
I’m trying to create a Ring server that behaves differently when in production
or development mode. I’ve checked a minimal cut-down version of my server into
GitHub here:
https://github.com/paulbutcher/profile-problem
There are two different versions of config.clj. The production version looks
like this:
(ns problem.config)
(def production true)
And the development version like this:
(ns problem.config)
(def production false)
The profiles section of project.clj adds the appropriate one of these to
source-paths depending on which profile is active:
:profiles {:production {:source-paths [production/src]}
:dev {:source-paths [dev/src]}
:uberjar [:production {:aot :all}]}
Finally, the server prints a message during startup:
(println Starting
in (if config/production production development) mode))
This all works fine - if I run “lein ring server-headless” or “lein
with-profile dev server-headless”, I see “Starting in development mode” and if
I run “lein with-profile production server-headless”, I see “Starting in
production mode”.
So far so good.
The problem arises after I run “lein ring uberjar”. The uberjar works exactly
as I would expect (it runs in production mode). But if I then run “lein ring
server-headless” or “lein with-profile dev server-headless” in my project
directory, instead of seeing “Starting in development mode”, I see “Starting in
production mode”.
Running “lein clean” sorts the problem out, but I worry that I’m missing
something fundamental that’s going to bite me at some point…
Thanks in advance...
--
paul.butcher-msgCount++
Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher
Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit
Leiningen profile problem
Oops - I originally sent this to the ClojureScript group, which probably wasn’t
the best place. Apologies to those who subscribe to both lists for the spam:
I’m clearly misunderstanding something fundamental about how Leiningen profiles
work. I’d appreciate help fixing my understanding.
I’m trying to create a Ring server that behaves differently when in production
or development mode. I’ve checked a minimal cut-down version of my server into
GitHub here:
https://github.com/paulbutcher/profile-problem
There are two different versions of config.clj. The production version looks
like this:
(ns problem.config)
(def production true)
And the development version like this:
(ns problem.config)
(def production false)
The profiles section of project.clj adds the appropriate one of these to
source-paths depending on which profile is active:
:profiles {:production {:source-paths [production/src]}
:dev {:source-paths [dev/src]}
:uberjar [:production {:aot :all}]}
Finally, the server prints a message during startup:
(println Starting in (if config/production production development)
mode))
This all works fine - if I run “lein ring server-headless” or “lein
with-profile dev server-headless”, I see “Starting in development mode” and if
I run “lein with-profile production server-headless”, I see “Starting in
production mode”.
So far so good.
The problem arises after I run “lein ring uberjar”. The uberjar works exactly
as I would expect (it runs in production mode). But if I then run “lein ring
server-headless” or “lein with-profile dev server-headless” in my project
directory, instead of seeing “Starting in development mode”, I see “Starting in
production mode”.
Running “lein clean” sorts the problem out, but I worry that I’m missing
something fundamental that’s going to bite me at some point…
Thanks in advance...
--
paul.butcher-msgCount++
Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher
Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Leiningen profile problem
Ah! I thought that Leiningen put class files in different places for different
profiles, but it looks like I was mistaken.
Thanks for the EDN hint - that sounds like a good way to go.
Cheers!
--
paul.butcher-msgCount++
Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher
Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con
On 28 July 2014 at 18:01:16, Thomas Heller (th.hel...@gmail.com) wrote:
Your issue is probably due to AOT compilation, uberjar aot compiles everything
and leaves alot of class files in your target directory (aka classpath). When
you run your app afterwards the class file is used instead of the clj file
since the .class file is newer. When you run lein clean those .class files
are removed and the classloader then looks for the clj file.
I'd recommend keeping a config file (can be EDN) and just having a development
and production version instead of using different code. You can either use a
command line argument to choose which config to load or use leinigen profiles
and resource paths so only include the one you want.
HTH,
/thomas
On Monday, July 28, 2014 3:45:51 PM UTC+2, Paul Butcher wrote:
Oops - I originally sent this to the ClojureScript group, which probably wasn’t
the best place. Apologies to those who subscribe to both lists for the spam:
I’m clearly misunderstanding something fundamental about how Leiningen profiles
work. I’d appreciate help fixing my understanding.
I’m trying to create a Ring server that behaves differently when in production
or development mode. I’ve checked a minimal cut-down version of my server into
GitHub here:
https://github.com/paulbutcher/profile-problem
There are two different versions of config.clj. The production version looks
like this:
(ns problem.config)
(def production true)
And the development version like this:
(ns problem.config)
(def production false)
The profiles section of project.clj adds the appropriate one of these to
source-paths depending on which profile is active:
:profiles {:production {:source-paths [production/src]}
:dev {:source-paths [dev/src]}
:uberjar [:production {:aot :all}]}
Finally, the server prints a message during startup:
(println Starting
in (if config/production production development) mode))
This all works fine - if I run “lein ring server-headless” or “lein
with-profile dev server-headless”, I see “Starting in development mode” and if
I run “lein with-profile production server-headless”, I see “Starting in
production mode”.
So far so good.
The problem arises after I run “lein ring uberjar”. The uberjar works exactly
as I would expect (it runs in production mode). But if I then run “lein ring
server-headless” or “lein with-profile dev server-headless” in my project
directory, instead of seeing “Starting in development mode”, I see “Starting in
production mode”.
Running “lein clean” sorts the problem out, but I worry that I’m missing
something fundamental that’s going to bite me at some point…
Thanks in advance...
--
paul.butcher-msgCount++
Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher
Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel
http://pragprog.com/book/pb7con
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
London Clojure/ClojureScript job
I’m looking to recruit a senior software engineer in London to work with Clojure, ClojureScript, React/Reagent, Docker, and a bunch of other equally interesting technologies. We’re not necessarily looking for Clojure/ClojureScript experience, so if you’ve been champing at the bit to get into this kind of thing, but have been unable to do so in your current role, we’d definitely be interested in talking to you. https://www.linkedin.com/jobs2/view/11032601 -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ClojureScript] London Clojure/ClojureScript job
On 24 July 2014 at 11:32:53, Bruce Durling (b...@otfrom.com) wrote: Paul, You might also want to post this on London Clojurians Jobs https://groups.google.com/forum/#!forum/london-clojurian-jobs Ah! I’ve just posted it to London Clojurians - I wasn’t aware of the separate jobs list (blush). Thanks! -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Is Korma still a good current choice for DB backend?
I wasn’t aware of yesql - thanks for the pointer. My concern with “write your queries in pure SQL” is increased vulnerability to SQL injection. From a quick glance at yesql, it seems likely that it does provide protection against SQL injection, but there’s nothing in the documentation (that I can find at first glance, at least) that explicitly says so. Can anyone who understands it confirm or deny? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con On 22 July 2014 at 14:08:20, Timothy Baldridge (tbaldri...@gmail.com) wrote: Also, read the rationale behint yesql: https://github.com/krisajenkins/yesql IMO, it hits the nail on the head. ORMs are both crappy object systems and crappy DB DSLs. With a library like yesql you write your queries in pure SQL and get pure data back. Now you can fully leverage both SQL and Clojure. Timothy On Tue, Jul 22, 2014 at 6:45 AM, Jonathon McKitrick jmckitr...@gmail.com wrote: I'll keep that in mind when I decide to learn MongoDB. :-) On Tuesday, July 22, 2014 8:25:16 AM UTC-4, Robin Heggelund Hansen wrote: Don't know about SQL-based solutions, but Monger (MongoDB bindings) just released a 2.0 update and is great! kl. 14:10:16 UTC+2 tirsdag 22. juli 2014 skrev Jonathon McKitrick følgende: Development and support seem to have slowed down. Are there newer or better choices out there with momentum right now? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Are zippers the right choice for sequence-like trees?
Thanks Jason - that’s helpful. -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con On 4 June 2014 at 14:33:27, Jason Felice (jason.m.fel...@gmail.com) wrote: In general, I've found that zippers make complicated edits super-easy, while a recursive phrasing of the same algorithm - if it exists and isn't super complicated to write - performs better and gives more control over structural sharing. I might prove out an algorithm with zippers, but when the pedal hits the road for performance, I'll try to rephrase it. If you can phrase things in terms of (catvec (subvec ...) (subvec ...)), then I'll bet (but you should check) that rrb-vector will perform better and do more structural sharing. On Wed, Jun 4, 2014 at 8:29 AM, Paul Butcher p...@paulbutcher.com wrote: I am working with “sequence like” trees - by which I mean that they’re very broad (typically the root node will have several thousand children) and shallow (no more than 2 levels deep). I’m often dealing with a degenerate tree that’s really just a sequence (all the nodes in the tree are children of the root). A typical edit is either changing the value of a single element, inserting a single element, or deleting a small set of contiguous elements. I’m currently using clojure.zip to edit the tree, and it’s working just fine. But I’m concerned that it won’t be giving me much value in terms of structure sharing for this type of tree and this type of edit. So I have two questions: 1) What would be a good way for me to instrument my code to determine what level of structure sharing I am achieving? 2) Should I consider switching to something based on (say?) rrb-vector? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Are zippers the right choice for sequence-like trees?
I am working with “sequence like” trees - by which I mean that they’re very broad (typically the root node will have several thousand children) and shallow (no more than 2 levels deep). I’m often dealing with a degenerate tree that’s really just a sequence (all the nodes in the tree are children of the root). A typical edit is either changing the value of a single element, inserting a single element, or deleting a small set of contiguous elements. I’m currently using clojure.zip to edit the tree, and it’s working just fine. But I’m concerned that it won’t be giving me much value in terms of structure sharing for this type of tree and this type of edit. So I have two questions: 1) What would be a good way for me to instrument my code to determine what level of structure sharing I am achieving? 2) Should I consider switching to something based on (say?) rrb-vector? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel http://pragprog.com/book/pb7con -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: core.async - extracting code from a go block
Thanks all - I appreciate the thoughts. -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher On 1 Jan 2014, at 05:46, Timothy Baldridge tbaldri...@gmail.com wrote: A few other things that might help: 1) you can use (put! c val) from inside any function. Sometimes if you want to fire and forget a send, that might be the best option 2) you can wrap the code inside the sub fns inside another go. This isn't as slow as you might think 3) think about using core.async's higher order ops like map, filter, mux, pub, mix, etc. Often I've found that by rethinking the problem a bit in therms of these ideas, it's possible to simplify the go blocks a bit. Timothy On Tue, Dec 31, 2013 at 7:14 PM, Cedric Greevey cgree...@gmail.com wrote: It should work if it's inlined or a macro. It won't shrink foo's generated code size any if bar is a macro, but it will split up the source code into smaller pieces if that's all you're concerned about. On Tue, Dec 31, 2013 at 8:50 PM, Paul Butcher p...@paulbutcher.com wrote: I recently discovered that parking calls only work if they're directly contained within a go block. So this works fine: (defn foo [ch] (go (! ch))) But this: (defn bar [ch] (! ch)) (defn foo [ch] (go (bar ch))) Results in Assert failed: ! used not in (go ...) block Is there any way around this? If I have a go block that's getting too big and want to extract some portion of it into a separate function, what's the best approach? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: case/java interop weirdness
On 30 Dec 2013, at 16:34, Cedric Greevey cgree...@gmail.com wrote: To do it with case, you'd need to wrap case in a macro that expanded to `(case ~thingy ~(eval case1) ...) or something along those lines Thanks, but I suspect that that might be another way of saying use condp :-) Cheers, -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
core.async - extracting code from a go block
I recently discovered that parking calls only work if they're directly contained within a go block. So this works fine: (defn foo [ch] (go (! ch))) But this: (defn bar [ch] (! ch)) (defn foo [ch] (go (bar ch))) Results in Assert failed: ! used not in (go ...) block Is there any way around this? If I have a go block that's getting too big and want to extract some portion of it into a separate function, what's the best approach? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
case/java interop weirdness
I'm sure I'm missing something very simple here, but I'm damned if I can see it. I'm trying to use the Java Rome RSS/Atom utility library from Clojure. The Rome Fetcher library defines a FeedEvent class, which among other things defines a number of static final string members: https://github.com/rometools/rome-fetcher/blob/master/src/main/java/org/rometools/fetcher/FetcherEvent.java#L17 Using the following simple project.clj: (defproject rsstest 1.0 :dependencies [[org.clojure/clojure 1.5.1] [org.rometools/rome-fetcher 1.2]]) I see the following: user= (import 'org.rometools.fetcher.FetcherEvent) org.rometools.fetcher.FetcherEvent user= FetcherEvent/EVENT_TYPE_FEED_POLLED FEED_POLLED user= (case FetcherEvent/EVENT_TYPE_FEED_POLLED #_= FetcherEvent/EVENT_TYPE_FEED_POLLED :ok) IllegalArgumentException No matching clause: FEED_POLLED user/eval1132 (NO_SOURCE_FILE:1) For some reason, case doesn't seem to be able to match on a static final String member of a Java class? Everything works fine if I use the literal string value though: user= (case FetcherEvent/EVENT_TYPE_FEED_POLLED #_= FEED_POLLED :ok) :ok I'd be very grateful for any guidance as to what's going on here? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: case/java interop weirdness
On 30 Dec 2013, at 13:57, Nicola Mometto brobro...@gmail.com wrote: The test clauses of case expressions are not evaluated, so that case is trying to match the symbol 'FetcherEvent/EVENT_TYPE_FEED_POLLED, not the value of FetcherEvent/EVENT_TYPE_FEED_POLLED. Ah - I was aware of the requirement for the constants to be compile-time literals, but assumed that give that they were final static Strings, that would work. Obviously not. Is there any way to achieve the effect I'm after with case? Or do I need to switch to condp? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: core.async/thread failing silently
On 8 Dec 2013, at 05:04, Alex Miller a...@puredanger.com wrote: Errors are most likely occurring on a thread different than main. Assuming you're using nrepl, have you looked at the nrepl-error buffer? Thanks. I wasn't aware of nrepl-error. Some quick googling has turned up a few articles about how to see nrepl-error when using emacs, but I'm afraid that I'm not - I'm starting nREPL with lein repl at the bash prompt. I'd be grateful for any pointers to how to view nrepl-error in this scenario. Thanks, -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: core.async/thread failing silently
On 8 Dec 2013, at 14:21, Alex Miller a...@puredanger.com wrote: If you're starting with lein repl then I would expect errors to be printed at the console - that's the actual process running the code. It's also possible that the thread's untaught exception handler doesn't print. I know that the expectation for async go blocks is that you should catch and handle any error. Have tried catching Throwable in your thread block and printing? Yup - that works fine (see below). The implication, I guess, is that nrepl doesn't set the default uncaught exception handler? I find that surprising? user= (require '[clojure.core.async :refer [thread !!]]) nil user= (defn thread-add [x y] #_= (thread #_= (try #_= (if (zero? y) #_= x #_= (let [t (thread-add (inc x) (dec y))] #_= (!! t))) #_= (catch Throwable e (println e) #'user/thread-add user= (!! (thread-add 10 10)) 20 user= (!! (thread-add 10 2000)) Exception in thread async-thread-macro-1986 java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:992) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher On 8 Dec 2013, at 14:21, Alex Miller a...@puredanger.com wrote: If you're starting with lein repl then I would expect errors to be printed at the console - that's the actual process running the code. It's also possible that the thread's untaught exception handler doesn't print. I know that the expectation for async go blocks is that you should catch and handle any error. Have tried catching Throwable in your thread block and printing? -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
core.async/thread failing silently
Consider the following function that adds two numbers exceptionally inefficiently by creating lots of threads: user= (require '[clojure.core.async :refer [thread !!]]) nil user= (defn thread-add [x y] #_= (thread #_= (if (zero? y) #_= x #_= (let [t (thread-add (inc x) (dec y))] #_= (!! t) #'user/thread-add This works for small numbers: user= (!! (thread-add 10 10)) 20 user= (!! (thread-add 10 1000)) 1010 Unsurprisingly, it fails after a certain point, because the JVM simply can't create enough threads. But what I find surprising is that it fails completely silently - no error message, just a nil return: user= (!! (thread-add 10 2000)) nil Is this what I should expect? Is there any way to persuade the REPL to show me errors in this kind of situation? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
tails function?
I fear that I'm missing something obvious here, so I'm getting ready to kick myself. I'm looking for an equivalent of Scala's tails method: scala List(1, 2, 3, 4).tails.toList res0: List[List[Int]] = List(List(1, 2, 3, 4), List(2, 3, 4), List(3, 4), List(4), List()) But I'm damned if I can find anything in the standard library. Clearly I can define it myself: user= (defn tails [coll] #_= (lazy-seq #_= (when-let [s (seq coll)] #_= (cons s (tails (rest s)) #'user/tails user= (tails [1 2 3 4]) ((1 2 3 4) (2 3 4) (3 4) (4)) But I can't believe that an equivalent isn't already in there somewhere? Thanks in advance to anyone who points me in the right direction. -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: tails function?
On 30 Nov 2013, at 01:33, Mark Engelberg mark.engelb...@gmail.com wrote: (take-while seq (iterate rest [1 2 3 4])) D'oh! Told you I would kick myself. Thanks Mark. -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Do (should?) these core.async utilities exist?
I've been playing with core.async, and have come across a couple of things that it seemed would probably be common use cases, but can't find anything in the library that addresses them. I'd be grateful for pointers if any of these do exist and I'm just missing them, or suggestions for reasons why I don't really want them and should be tackling the problem in a different way: A way to convert a channel to a lazy sequence (i.e. build the sequence by repeatedly reading from a channel until it's closed). It seems odd that core.async provides a means to turn a lazy sequence into a channel (to-chan) but not the inverse? An equivalent of doseq for a channel, which repeatedly reads from the channel and calls its body with the result of doing so, terminating when the channel is closed. Of course, both of these are easy enough to write, but I'm wondering whether the fact that they aren't provided as standard is telling me something? -- paul.butcher-msgCount++ Silverstone, Brands Hatch, Donington Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 24 Oct 2013, at 01:46, Stuart Halloway stuart.hallo...@gmail.com wrote: Is the Scala for lazy or eager? If the latter, you are not comparing apples to apples (separate from the other differences David already pointed out.) Oh, it's eager. Bear in mind that my purpose wasn't really to directly compare Scala and Clojure - in fact, I was originally planning to knock up a fully lazy solution in Clojure to demonstrate that the memory-hungry eager Scala solution could be trivially made to run in much less memory in Clojure. Unfortunately I got mired in this performance issue before I could get to that point. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 24 Oct 2013, at 11:34, Phillip Lord phillip.l...@newcastle.ac.uk wrote: What does Scala do? I mean, given that it doesn't have the same problem, perhaps it has a solution? By default, Scala uses a tree-based set, not a hash-based set. So the fact that it doesn't run into hashing issues isn't surprising (and is yet another example of how this isn't a perfect apples-to-apples comparison). However, I've just tried rerunning the Scala using a hash-based set implementation, and that takes the runtime down from around 2.5 minutes to just over 2. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 18:37, David Nolen dnolen.li...@gmail.com wrote:
The problem here is that you're not following your Scala solution closely
enough. I suspect if you used defrecords to represent the pieces the way that
you used a class in Scala you can avoid the number of collisions for larger
problems.
Not tested much but I tried something like the following and I no longer see
hash collisions dominating in YourKit:
It would be lovely if it was that easy, David. However, I still see lots of
hash collisions when using records. Here's a portion of the output from an
implementation that's instrumented to print hash codes, similar to what Andy
did with the vector-based implementation:
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 2]}
#chess_clojure.core.Piece{:name :B, :pos [1 5]} #chess_clojure.core.Piece{:name
:N, :pos [0 1]} #chess_clojure.core.Piece{:name :R, :pos [8 4]}}
842565000: #{#chess_clojure.core.Piece{:name :N, :pos [0 4]}
#chess_clojure.core.Piece{:name :Q, :pos [3 3]} #chess_clojure.core.Piece{:name
:B, :pos [1 0]} #chess_clojure.core.Piece{:name :R, :pos [8 5]}}
842565000: #{#chess_clojure.core.Piece{:name :N, :pos [0 4]}
#chess_clojure.core.Piece{:name :Q, :pos [3 3]} #chess_clojure.core.Piece{:name
:B, :pos [1 4]} #chess_clojure.core.Piece{:name :R, :pos [8 1]}}
842565000: #{#chess_clojure.core.Piece{:name :N, :pos [0 3]}
#chess_clojure.core.Piece{:name :Q, :pos [3 4]} #chess_clojure.core.Piece{:name
:B, :pos [1 0]} #chess_clojure.core.Piece{:name :R, :pos [8 5]}}
842565000: #{#chess_clojure.core.Piece{:name :N, :pos [0 3]}
#chess_clojure.core.Piece{:name :Q, :pos [3 5]} #chess_clojure.core.Piece{:name
:B, :pos [1 0]} #chess_clojure.core.Piece{:name :R, :pos [8 4]}}
842565000: #{#chess_clojure.core.Piece{:name :R, :pos [0 3]}
#chess_clojure.core.Piece{:name :Q, :pos [4 2]} #chess_clojure.core.Piece{:name
:B, :pos [1 0]} #chess_clojure.core.Piece{:name :N, :pos [8 0]}}
842565000: #{#chess_clojure.core.Piece{:name :R, :pos [0 3]}
#chess_clojure.core.Piece{:name :Q, :pos [3 4]} #chess_clojure.core.Piece{:name
:B, :pos [1 5]} #chess_clojure.core.Piece{:name :N, :pos [8 0]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 2]}
#chess_clojure.core.Piece{:name :B, :pos [1 5]} #chess_clojure.core.Piece{:name
:R, :pos [0 0]} #chess_clojure.core.Piece{:name :N, :pos [8 5]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [4 2]}
#chess_clojure.core.Piece{:name :R, :pos [1 1]} #chess_clojure.core.Piece{:name
:B, :pos [0 5]} #chess_clojure.core.Piece{:name :N, :pos [8 5]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 2]}
#chess_clojure.core.Piece{:name :R, :pos [1 1]} #chess_clojure.core.Piece{:name
:B, :pos [0 4]} #chess_clojure.core.Piece{:name :N, :pos [8 5]}}
842565000: #{#chess_clojure.core.Piece{:name :R, :pos [0 3]}
#chess_clojure.core.Piece{:name :Q, :pos [3 4]} #chess_clojure.core.Piece{:name
:B, :pos [1 0]} #chess_clojure.core.Piece{:name :N, :pos [8 5]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 4]}
#chess_clojure.core.Piece{:name :R, :pos [1 1]} #chess_clojure.core.Piece{:name
:B, :pos [0 5]} #chess_clojure.core.Piece{:name :N, :pos [8 2]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 5]}
#chess_clojure.core.Piece{:name :R, :pos [1 1]} #chess_clojure.core.Piece{:name
:B, :pos [0 4]} #chess_clojure.core.Piece{:name :N, :pos [8 2]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 4]}
#chess_clojure.core.Piece{:name :B, :pos [1 5]} #chess_clojure.core.Piece{:name
:R, :pos [0 2]} #chess_clojure.core.Piece{:name :N, :pos [8 1]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 5]}
#chess_clojure.core.Piece{:name :B, :pos [0 4]} #chess_clojure.core.Piece{:name
:R, :pos [1 2]} #chess_clojure.core.Piece{:name :N, :pos [8 1]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [3 4]}
#chess_clojure.core.Piece{:name :B, :pos [1 5]} #chess_clojure.core.Piece{:name
:R, :pos [7 2]} #chess_clojure.core.Piece{:name :N, :pos [0 0]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [4 2]}
#chess_clojure.core.Piece{:name :B, :pos [0 4]} #chess_clojure.core.Piece{:name
:N, :pos [6 5]} #chess_clojure.core.Piece{:name :R, :pos [1 0]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [4 0]}
#chess_clojure.core.Piece{:name :R, :pos [0 2]} #chess_clojure.core.Piece{:name
:N, :pos [6 5]} #chess_clojure.core.Piece{:name :B, :pos [1 4]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [4 2]}
#chess_clojure.core.Piece{:name :R, :pos [0 0]} #chess_clojure.core.Piece{:name
:N, :pos [6 5]} #chess_clojure.core.Piece{:name :B, :pos [1 4]}}
842565000: #{#chess_clojure.core.Piece{:name :Q, :pos [4 2]}
#chess_clojure.core.Piece{:name :R, :pos [1 1]} #chess_clojure.core.Piece{:name
:N, :pos [6 5]} #chess_clojure.core.Piece{:name :B, :pos [0 3]}}
842565000: #{#chess_clojure.core.Piece{:name :R, :pos [1 4]}
#chess_clojure.core.Piece{:name :Q, :pos [3 3]}
Re: Request for help optimising a Clojure program
On 22 Oct 2013, at 20:20, David Nolen dnolen.li...@gmail.com wrote: On Tue, Oct 22, 2013 at 3:11 PM, Paul Butcher p...@paulbutcher.com wrote: Yeah - I have tried giving it more RAM without any effect on the timing whatsoever. And I couldn't see the point of stopping people with less RAM than that from being able to run it :-) But without enough RAM most JVMs will thrash in GC when running that code. Yes, of course. But as I said, I tried giving it more RAM (the same 12G as the Scala solution) and it made no difference whatsoever. In fact, YourKit shows me that, even after running for an hour (i.e. more than 10x longer than the Scala version takes to return the full solution) the Clojure version is still using less than 2G. Whatever is going on, it's not a result of a lack of RAM. Saying both run in less than a second is not particularly informative for the small problems :) Does Scala take 4ms and Clojure takes 400ms on the small problems? That's a big difference. Yes, of course. But my experience with small problems has been that differences between execution time, especially when talking about runtimes as dissimilar as the Clojure and Scala runtimes, are meaningless. Those differences tend to have more to do with startup costs and the vagaries of when hotspot happens to optimise things than anything fundamental. But you asked, so here it is: For Clojure, the 3x3 problem takes 7ms and the 4x4 problem takes 78ms. For Scala, it's 15ms and 252ms respectively. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 22 Oct 2013, at 20:20, David Nolen dnolen.li...@gmail.com wrote: On Tue, Oct 22, 2013 at 3:11 PM, Paul Butcher p...@paulbutcher.com wrote: Yeah - I have tried giving it more RAM without any effect on the timing whatsoever. And I couldn't see the point of stopping people with less RAM than that from being able to run it :-) But without enough RAM most JVMs will thrash in GC when running that code. Saying both run in less than a second is not particularly informative for the small problems :) Does Scala take 4ms and Clojure takes 400ms on the small problems? That's a big difference. David -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 14:18, David Nolen dnolen.li...@gmail.com wrote: If set construction was 1000X worse why don't the smaller problem sizes exhibit exactly the same issue? If the Scala version requires 12G why is the Clojure version steady at 300M? Both excellent questions. Aren't Scala for comprehensions optimized now into lower level loops? Clojure for comprehensions generate lazy sequences and all the allocation and forcing of thunks that implies. I don't think so - AFAIK Scala's for statement is rewritten into flatMap and map under the hood. Happy to be corrected though. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 14:19, David Nolen dnolen.li...@gmail.com wrote: The Scala code is using immutable sets. Correct - the Scala is immutable throughout. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 15:32, David Powell djpow...@djpowell.net wrote: When you say it is spending 99% of its time in PersistentHashSet.cons, is that the time spent in just that method, or the time spent in that method and the methods that it calls? The latter. Given that (set ...) is one of the first things called by (solve..), and that its contents are produced lazily by a for comprehension, if you are looking at the total time spent in PersistentHashSet.assoc, then maybe that will be particularly slow. Not that that solves the problem, but I was wondering if PersistentHashSet.assoc might be a red herring. Ah - that's entirely plausible. I have basically no experience of profiling Clojure, or understanding the way in which laziness affects doing so (I suspect that it complicates things somewhat?). Also, out of interest, are you using a sampling profiler, or an instrumenting one? Sampling. Sometimes the built-in hprof profiler is good for this sort of thing: java -Xrunhprof:cpu=times,depth=4 -Xmx2g -jar target\chess-clojure-0.1.0-SNAPSHOT-standalone.jar - you can use either cpu=times for instrumenting, or cpu=samples for sampling. It is nice because the depth parameter lets you find hotspots including a few steps up the stacktrace (4 in this case). And you can use Ctrl-\ / Ctrl-Break / SIG_QUIT to get it to dump the current stats while the program is still running to see if some methods are getting worse over time. I've not used hprof before - I'll see if I can divine anything useful from it. Thanks. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 17:06, Andy Fingerhut andy.finger...@gmail.com wrote: I have instrumented a copy of Paul's Clojure program to print the hash code of all of the solutions in the set returned by solve, and there are *many* pairs of solutions that have identical hash values Aha! The smoking gun :-) Very many thanks, Andy. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 17:43, Andy Fingerhut andy.finger...@gmail.com wrote:
Paul, your function solve returns a set of solutions, but there is nothing on
the program that seems to rely upon being able to quickly test whether a
particular solution is in such a set. Returning a sequence from solve is
much faster, since it avoids the PersistentHashSet hash collision issue
entirely.
Just replace #{#{}} with [#{}], and remove the call to 'set' in solve.
The problem with that, Andy, is that it will result in a lot of duplicates. For
the 4x4 problem, for example, that approach raises the size of the returned
collection from 8 to 384.
My guess is that for the 6x9 problem it will just remove the set handling
problem and replace it with a memory/time issue as the search space grows (and
I only have 16G in my MacBook Pro :-)
I don't need to get this working - I only knocked it together as an interesting
exercise. And if it's helped to track down an issue in the Clojure runtime,
it's already achieved far more than I expected it to :-)
Thanks again,
--
paul.butcher-msgCount++
Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: p...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 23 Oct 2013, at 18:15, Andy Fingerhut andy.finger...@gmail.com wrote: If we had a 'universal comparator', i.e. a comparison function that provided a total order on any pair of values that anyone would ever want to put into a set or use as a map key, then instead of having linked lists for values that collide, we could have trees like those in the implementations of sorted-maps and sorted-sets today. Wouldn't it be better to improve the way that hashes are calculated for vectors? A good hash function should make it unlikely that similar values have the same hash. The current algorithm seems to make that more likely than it should? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Request for help optimising a Clojure program
I've been playing around with a generalised version of the N-Queens problem that handles other chess pieces. I have a Scala implementation that uses an eager depth-first search. Given enough RAM (around 12G - it's very memory hungry!) it can solve a 6x9 board with 6 pieces in around 2.5 minutes on my MacBook Pro. I then created a Clojure version of the same algorithm with the intention of (eventually) using laziness to avoid the memory issue. However, the performance of the Clojure version is several orders of magnitude worse than the Scala version (I've left it running overnight on the problem that the Scala version solves in 2.5 minutes and it still hasn't completed). I'm struggling to see why the performance of the Clojure version is so much worse than the Scala. Profiling in YourKit suggests that it's spending 99% of its time in PersistentHashSet.cons, but I'm at a loss to explain why. I would be grateful for any insight. The code for the two different versions is on GitHub: https://github.com/paulbutcher/chess-scala https://github.com/paulbutcher/chess-clojure Notes: - I know that an exhaustive depth-first search isn't a great way to tackle this problem. But I'd like to understand why I see such a dramatic difference between the performance of the Scala and Clojure versions. - I believe that I've implemented the same algorithm in both Scala and Clojure - certainly they both generate the same results for small problems (there are 3x3 and 4x4 problems commented out in the source). But clearly I can't rule out the possibility that I've made a mistake in the Clojure implementation. If anyone can spot my stupid mistake, I'd greatly appreciate it. Thanks in advance for any insight that anyone can offer. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 22 Oct 2013, at 18:45, Mark Engelberg mark.engelb...@gmail.com wrote: I looked briefly at the code and can confirm that to my eye, the two implementations appear to be implementing the same algorithm. Thanks - always good to have a second pair of eyes :-) My first guess would be that the performance difference comes from Clojure's use of boxed numbers That was my first thought too - but I thought that I'd look at a profile first :-) -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Request for help optimising a Clojure program
On 22 Oct 2013, at 19:55, David Nolen dnolen.li...@gmail.com wrote: I note that the Clojure version isn't being given 12gigs of RAM, is this something you're giving to the JVM after when you run a AOTed version of the Clojure code. Yeah - I have tried giving it more RAM without any effect on the timing whatsoever. And I couldn't see the point of stopping people with less RAM than that from being able to run it :-) You also haven't said whether the timings for the smaller problems are significantly different between Scala and Clojure. They both run in less than a second in both Clojure and Scala, so no significant difference. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: How to go about 'proving' why dynamically typed languages are better.
On 6 Oct 2013, at 04:35, zcaudate z...@caudate.me wrote: I'm a little bit miffed over this current craze of `types` and `correctness` of programs. It smells to me of the whole `object` craze of the last two decades. This debate is as old as the hills (it certainly predates object-oriented programming). There's no question that there is a valid debate to be had about static versus dynamic typing, but most such debates are hamstrung by basic misunderstandings. The best explanation of these misunderstandings I've come across is What to Know Before Debating Type Systems: http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ In particular it asserts (correctly in my view) that what static type system proponents mean by type and what dynamic type system proponents mean by type are very different things. Most debates founder on a failure to recognise that difference. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
Alan, Apologies for the delayed reply - I remember Iota well (there was some cross-fertilisation between it and foldable-seq a few months back IIRC :-) Having said that, I don't think that Iota will help in my particular situation (although I'd be delighted to be proven wrong)? Given that the file I'm processing is an XML file, and will therefore have to pass through an XML parser, unless I write an XML parser on top of the reducers framework, I'm stuck with dealing with sequences at some point along the way? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 30 Sep 2013, at 11:15, Alan Busby thebu...@gmail.com wrote: Sorry to jump in, but I thought it worthwhile to add a couple points; (sorry for being brief) 1. Reducers work fine with data much larger than memory, you just need to mmap() the data you're working with so Clojure thinks everything is in memory when it isn't. Reducer access is fairly sequential, not random, so spinning disks work great here. 2. A 40GB XML file is very often many many smaller XML documents aggregated together. It's often faster to separate each document into it's own line (via various UNIX tools) and parse each line separately. I typically do something like $ zcat bigxml.gz | tr '\n' ' ' | sed 's/foo/\nfoo/' | grep '^foo' records.xml . 3. Check out the Iota library, https://github.com/thebusby/iota/ . I often use for reducing over 100's of GB's worth of text data. It does what Jozef suggests, and makes a text file a foldable collection. 4. While pmap is great for advertising the power of Clojure, it's likely safe to say that it should be ignored if you're actually looking for performance. Hope this helps, Alan Busby -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
On 29 Sep 2013, at 22:58, Paul Mooser taron...@gmail.com wrote: Paul, is there any easy way to get the (small) dataset you're working with, so we can run your actual code against the same data? The dataset I'm using is a Wikipedia dump, which hardly counts as small :-) Having said that, the first couple of million lines is all you need to reproduce the results I'm getting, which you can download with: curl http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 | bunzip2 | head -n 200 enwiki-short.xml -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
On 28 Sep 2013, at 00:27, Stuart Halloway stuart.hallo...@gmail.com wrote: I have posted an example that shows partition-then-fold at https://github.com/stuarthalloway/exploring-clojure/blob/master/examples/exploring/reducing_apple_pie.clj. I would be curious to know how this approach performs with your data. With the generated data I used, the partition+fold and partition+pmap approaches both used most of my cores and had similar perf. Hey Stuart, Thanks for getting back to me. I've updated the code for my word count example on GitHub with (I believe) something that works along the lines you suggest: https://github.com/paulbutcher/parallel-word-count Here are some sample runs on my machine (a 4-core retina MacBook Pro). Each of these runs counts the words in the first 1 pages of Wikipedia: $ lein run 1 ~/enwiki.xml sequential Elapsed time: 23630.443 msecs $ lein run 1 ~/enwiki.xml fold Elapsed time: 8697.79 msecs $ lein run 1 ~/enwiki.xml pmap 1 Elapsed time: 27393.703 msecs $ lein run 1 ~/enwiki.xml pthenf 1 Elapsed time: 37263.193 msecs As you can see, the the foldable-seq version gives an almost 3x speedup relative to the sequential version, and both the partition-then-pmap and partition-then-fold versions are significantly slower. The last argument for the partition-then-pmap and partition-then-fold versions is a partition size. I've tried various different sizes with no material effect: $ lein run 1 ~/enwiki.xml pthenf 1000 Elapsed time: 43187.385 msecs $ lein run 1 ~/enwiki.xml pthenf 10 Elapsed time: 35702.651 msecs $ lein run 1 ~/enwiki.xml pmap 1000 Elapsed time: 34087.314 msecs $ lein run 1 ~/enwiki.xml pmap 10 Elapsed time: 47340.969 msecs The performance of the partition-then-pmap version is actually much worse than the numbers above suggest. There's something very weird going on with (I guess) garbage collection - it takes a *long* time to finish after printing the elapsed time and the performance is completely pathological with larger page counts. Bottom line: the only way I've been able to obtain any kind of speedup remains foldable-seq. I'd be very grateful indeed if you could take a look at how I've implemented partition-then-fold to make sure that I've correctly captured your intent. Or if you have any suggestions for anything else that might work, or to explain the poor performance of partition-then-pmap and partition-then-fold. My guess is that the problem with partition-then-fold is the copying that's going on during the (into [] %). I can see that it is operating in parallel because the number of cores in use goes up, but the net effect is an overall slowdown rather than a speedup. That it performs worse than foldable-seq isn't surprising to me, given that it introduces an unnecessary copy. I still think that it's a crying shame to disallow folding over sequences - as the above shows, the gains both in performance and programming ease are significant, and it would only take a small modification to the reducers API to fix the holding-onto-head problem. What would be the downside of making this modification and allowing foldable sequences? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 28 Sep 2013, at 00:27, Stuart Halloway stuart.hallo...@gmail.com wrote: Hi Paul, I have posted an example that shows partition-then-fold at https://github.com/stuarthalloway/exploring-clojure/blob/master/examples/exploring/reducing_apple_pie.clj. I would be curious to know how this approach performs with your data. With the generated data I used, the partition+fold and partition+pmap approaches both used most of my cores and had similar perf. Enjoying your book! Stu On Sat, May 25, 2013 at 12:34 PM, Paul Butcher p...@paulbutcher.com wrote: I'm currently working on a book on concurrent/parallel development for The Pragmatic Programmers. One of the subjects I'm covering is parallel programming in Clojure, but I've hit a roadblock with one of the examples. I'm hoping that I can get some help to work through it here. The example counts the words contained within a Wikipedia dump. It should respond well to parallelisation (I have Java and Scala versions that perform excellently) but I've been incapable of coming up with a nice solution in Clojure. The code I'm working with is checked into GitHub: The basic sequential algorithm is: (frequencies (mapcat get-words pages)) If I run that on the first 10k pages in Wikipedia dump, it takes ~21s on my MacBook Pro. One way to parallelise it is to create a parallel version of frequencies that uses reducers: (defn frequencies-fold [words] (r/fold (partial merge-with +) (fn
Re: Parallelising over a lazy sequence - request for help
On 28 Sep 2013, at 01:22, Rich Morin r...@cfcl.com wrote: On Sat, May 25, 2013 at 12:34 PM, Paul Butcher p...@paulbutcher.com wrote: I'm currently working on a book on concurrent/parallel development for The Pragmatic Programmers. ... Ordered; PDF just arrived (:-). Cool - very interested to hear your feedback once you've had a chance to read it. I don't know yet whether the book has anything like this, but I'd like to see a table that shows which concurrency and parallelism approaches are supported (and to what extent) by various languages. So, actors, agents, queues, reducers (etc) would be on one axis; Clojure, Erlang, Go, and Ruby (etc) would be on the other. I'd expect Clojure to have pretty complete coverage; Ruby, not so much. That might make a good appendix or section in the (as yet unwritten) wrap-up chapter - thanks for the suggestion. And yes, I suspect that you're right that Clojure will fare well in the comparison :-) -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 28 Sep 2013, at 01:22, Rich Morin r...@cfcl.com wrote: On Sat, May 25, 2013 at 12:34 PM, Paul Butcher p...@paulbutcher.com wrote: I'm currently working on a book on concurrent/parallel development for The Pragmatic Programmers. ... Ordered; PDF just arrived (:-). I don't know yet whether the book has anything like this, but I'd like to see a table that shows which concurrency and parallelism approaches are supported (and to what extent) by various languages. So, actors, agents, queues, reducers (etc) would be on one axis; Clojure, Erlang, Go, and Ruby (etc) would be on the other. I'd expect Clojure to have pretty complete coverage; Ruby, not so much. -r -- http://www.cfcl.com/rdmRich Morin http://www.cfcl.com/rdm/resume r...@cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Software system design, development, and documentation -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
Ah - one mystery down. Thanks Andy! -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 28 Sep 2013, at 16:26, Andy Fingerhut andy.finger...@gmail.com wrote: I do not know about the most important parts of your performance difficulties, but on a more trivial point I might be able to shed some light. See the ClojureDocs page for pmap, which refers to the page for future, linked below. If you call (shutdown-agents) the 60-second wait to exit should go away. http://clojuredocs.org/clojure_core/clojure.core/future Andy Sent from my iPhone On Sep 28, 2013, at 1:41 AM, Paul Butcher p...@paulbutcher.com wrote: On 28 Sep 2013, at 00:27, Stuart Halloway stuart.hallo...@gmail.com wrote: I have posted an example that shows partition-then-fold at https://github.com/stuarthalloway/exploring-clojure/blob/master/examples/exploring/reducing_apple_pie.clj. I would be curious to know how this approach performs with your data. With the generated data I used, the partition+fold and partition+pmap approaches both used most of my cores and had similar perf. Hey Stuart, Thanks for getting back to me. I've updated the code for my word count example on GitHub with (I believe) something that works along the lines you suggest: https://github.com/paulbutcher/parallel-word-count Here are some sample runs on my machine (a 4-core retina MacBook Pro). Each of these runs counts the words in the first 1 pages of Wikipedia: $ lein run 1 ~/enwiki.xml sequential Elapsed time: 23630.443 msecs $ lein run 1 ~/enwiki.xml fold Elapsed time: 8697.79 msecs $ lein run 1 ~/enwiki.xml pmap 1 Elapsed time: 27393.703 msecs $ lein run 1 ~/enwiki.xml pthenf 1 Elapsed time: 37263.193 msecs As you can see, the the foldable-seq version gives an almost 3x speedup relative to the sequential version, and both the partition-then-pmap and partition-then-fold versions are significantly slower. The last argument for the partition-then-pmap and partition-then-fold versions is a partition size. I've tried various different sizes with no material effect: $ lein run 1 ~/enwiki.xml pthenf 1000 Elapsed time: 43187.385 msecs $ lein run 1 ~/enwiki.xml pthenf 10 Elapsed time: 35702.651 msecs $ lein run 1 ~/enwiki.xml pmap 1000 Elapsed time: 34087.314 msecs $ lein run 1 ~/enwiki.xml pmap 10 Elapsed time: 47340.969 msecs The performance of the partition-then-pmap version is actually much worse than the numbers above suggest. There's something very weird going on with (I guess) garbage collection - it takes a *long* time to finish after printing the elapsed time and the performance is completely pathological with larger page counts. Bottom line: the only way I've been able to obtain any kind of speedup remains foldable-seq. I'd be very grateful indeed if you could take a look at how I've implemented partition-then-fold to make sure that I've correctly captured your intent. Or if you have any suggestions for anything else that might work, or to explain the poor performance of partition-then-pmap and partition-then-fold. My guess is that the problem with partition-then-fold is the copying that's going on during the (into [] %). I can see that it is operating in parallel because the number of cores in use goes up, but the net effect is an overall slowdown rather than a speedup. That it performs worse than foldable-seq isn't surprising to me, given that it introduces an unnecessary copy. I still think that it's a crying shame to disallow folding over sequences - as the above shows, the gains both in performance and programming ease are significant, and it would only take a small modification to the reducers API to fix the holding-onto-head problem. What would be the downside of making this modification and allowing foldable sequences? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 28 Sep 2013, at 00:27, Stuart Halloway stuart.hallo...@gmail.com wrote: Hi Paul, I have posted an example that shows partition-then-fold at https://github.com/stuarthalloway/exploring-clojure/blob/master/examples/exploring/reducing_apple_pie.clj. I would be curious to know how this approach performs with your data. With the generated data I used, the partition+fold and partition+pmap approaches both used most of my cores and had similar perf. Enjoying your book! Stu On Sat, May 25, 2013 at 12:34 PM, Paul Butcher p...@paulbutcher.com wrote: I'm currently working
Re: Parallelising over a lazy sequence - request for help
On 28 Sep 2013, at 17:14, Jozef Wagner jozef.wag...@gmail.com wrote: I would go a bit more further and suggest that you do not use sequences at all and work only with reducible/foldable collections. Make an input reader which returns a foldable collection and you will have the most performant solution. The thing about holding into the head is being worked on right now, see http://dev.clojure.org/jira/browse/CLJ-1250 That's fantastic news Jozef - is there any idea when that might be fixed? I can see that it's been triaged, but I'm not sure what exactly that means when it comes to the Clojure dev process? Could you expand on exactly what you mean when you say an input reader which returns a foldable collection? Bear in mind that the problem I'm trying to solve involves a 40GB Wikipedia dump, which clearly can't all be in RAM at one time. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
On 28 Sep 2013, at 17:42, Jozef Wagner jozef.wag...@gmail.com wrote: I mean that you should forgot about lazy sequences and sequences in general, if you want to have a cutting edge performance with reducers. Example of reducible slurp, https://gist.github.com/wagjo/6743885 , does not hold into the head. OK - I buy that logic. But I'm unsure whether it can be applied to the case that I'm interested in. Firstly, I'm dealing with an XML file, so I need to parse it. Right now I'm using data.xml/parse to do so, which returns a lazy sequence. So, short of writing my own XML parser, I have to go via a lazy sequence at some point along the way? Secondly, the primary point of this is to exploit parallelism, not reducers per-se. So supporting CollReduce isn't enough, I also need to support CollFold. Which is exactly what foldable-seq does - I'm not sure how that differs from what you're proposing? Am I missing something? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
On 28 Sep 2013, at 19:51, Jozef Wagner jozef.wag...@gmail.com wrote: Anyway, I think the bottleneck in your code is at https://github.com/paulbutcher/parallel-word-count/blob/master/src/wordcount/core.clj#L9 Instead of creating new persistent map for each word, you should use a transient here. I would love it if that were true. However the code you reference is from the algorithm that uses foldable-seq and runs quickly (i.e. not the one that has the performance problem). -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
Thanks Alex - I've made both of these changes. The shutdown-agents did get rid of the pause at the end of the pmap solution, and the -server argument made a very slight across-the-board performance improvement. But neither of them fundamentally change the basic result (that the implementation that uses foldable-seq gives a 3x speedup and the other implementations are slower than the sequential version). -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 28 Sep 2013, at 21:47, Alex Miller a...@puredanger.com wrote: For your timings, I would also strongly recommend altering your project.clj to force the -server hotspot: :jvm-opts ^:replace [-Xmx1g -server ... and whatever else you want here ... ] By default lein will use tiered compilation to optimize repl startup, which is not what you want for timing. And I second Andy's (shutdown-agents) advice. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
On 28 Sep 2013, at 22:00, Alex Miller a...@puredanger.com wrote: Reducers (and fork/join in general) are best suited for fine-grained computational parallelism on in-memory data. The problem in question involves processing more data than will fit in memory. So the question is then what is the best way to parallelize computation over the stream. There are many ways to do so - pmap is one easy mechanism to get parallelism over a lazy sequence but the chunking behavior of pmap can easily be non-optimal for particular use cases. Stu's other solution (prepare-with-partition-then-fold) reads chunks of data into memory, then uses f/j over the top of those chunks. Yes - I'm aware that that's how Stu's solution works. However when I use that in my example, the performance sucks. As I said in my earlier message, I believe that the reason for this is the additional copying that it involves. To date, I have exactly one implementation that has good parallel performance - the one that uses foldable-seq (which is 3x faster than the sequential version). Everything else I've tried performs worse than the sequential version. Yet another possible solution is to use a pool of compute threads and to read from the stream, give those compute tasks to the pool to execute in a background thread, then reassemble the results at the end (or in a separate thread). The balance in this approach is to make the tasks the right chunkiness. Using a buffered queue is one technique to avoid having your input reader overload the capacity of the processing system. That's basically *exactly* what the foldable-seq implementation does. I would also mention that using transients as you build input collections will be a big win. I'm sure that that's true - but the foldable-seq implementation is the one that would gain most from that, and that's the one that already runs 3x faster than the sequential implementation. But it's also the one that Stu objects to. What I'm trying to find is an implementation that Stu doesn't object to, but is faster than the sequential version of the algorithm. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 28 Sep 2013, at 22:00, Alex Miller a...@puredanger.com wrote: Reducers (and fork/join in general) are best suited for fine-grained computational parallelism on in-memory data. The problem in question involves processing more data than will fit in memory. So the question is then what is the best way to parallelize computation over the stream. There are many ways to do so - pmap is one easy mechanism to get parallelism over a lazy sequence but the chunking behavior of pmap can easily be non-optimal for particular use cases. Stu's other solution (prepare-with-partition-then-fold) reads chunks of data into memory, then uses f/j over the top of those chunks. Yet another possible solution is to use a pool of compute threads and to read from the stream, give those compute tasks to the pool to execute in a background thread, then reassemble the results at the end (or in a separate thread). The balance in this approach is to make the tasks the right chunkiness. Using a buffered queue is one technique to avoid having your input reader overload the capacity of the processing system. I would also mention that using transients as you build input collections will be a big win. Alex On Saturday, September 28, 2013 11:14:41 AM UTC-5, Jozef Wagner wrote: I would go a bit more further and suggest that you do not use sequences at all and work only with reducible/foldable collections. Make an input reader which returns a foldable collection and you will have the most performant solution. The thing about holding into the head is being worked on right now, see http://dev.clojure.org/jira/browse/CLJ-1250 JW On Saturday, September 28, 2013 10:41:20 AM UTC+2, Paul Butcher wrote: On 28 Sep 2013, at 00:27, Stuart Halloway stuart@gmail.com wrote: I have posted an example that shows partition-then-fold at https://github.com/stuarthalloway/exploring-clojure/blob/master/examples/exploring/reducing_apple_pie.clj. I would be curious to know how this approach performs with your data. With the generated data I used, the partition+fold and partition+pmap approaches both used most of my cores and had similar perf. Hey Stuart, Thanks for getting back to me. I've updated the code for my word count example on GitHub with (I believe) something that works along the lines you suggest: https://github.com/paulbutcher/parallel-word-count Here are some sample runs on my machine (a 4-core retina MacBook Pro). Each of these runs counts the words in the first 1 pages of Wikipedia: $ lein run 1 ~/enwiki.xml
Parallelising over a lazy sequence - request for help
I'm currently working on a book on concurrent/parallel development for The Pragmatic Programmers. One of the subjects I'm covering is parallel programming in Clojure, but I've hit a roadblock with one of the examples. I'm hoping that I can get some help to work through it here. The example counts the words contained within a Wikipedia dump. It should respond well to parallelisation (I have Java and Scala versions that perform excellently) but I've been incapable of coming up with a nice solution in Clojure. The code I'm working with is checked into GitHub: The basic sequential algorithm is: (frequencies (mapcat get-words pages)) If I run that on the first 10k pages in Wikipedia dump, it takes ~21s on my MacBook Pro. One way to parallelise it is to create a parallel version of frequencies that uses reducers: (defn frequencies-fold [words] (r/fold (partial merge-with +) (fn [counts word] (assoc counts word (inc (get counts word 0 words)) And sure enough, if I use that, along with use the foldable-seq utility I posted about here are while ago it runs in ~8s, almost a 3x speedup, not bad given that the parallel version is unable to use transients. Unfortunately, as they currently stand, reducers have a fatal flaw that means that, even with foldable-seq, they're basically useless with lazy sequences. Reducers always hold onto the head of the sequence they're given, so there's no way to use this approach for a complete Wikipedia dump (which runs to around 40GiB). So the other approach I've tried is to use pmap: (defn frequencies-pmap [words] (reduce (partial merge-with +) (pmap frequencies (partition-all 1 words But, for reasons I don't understand, this performs dreadfully - taking ~26s, i.e. significantly slower than the sequential version. I've tried playing with different partition sizes without materially affecting the result. So, what I'm looking for is either: a) An explanation for why the pmap-based solution performs so badly b) A way to fix the holding onto head problem that's inherent within reducers. With the last of these in mind, it strikes me that the problem fundamentally arises from the desire for reducers to follow the same basic API as normal code. So: (reduce (filter ... (map ... coll))) becomes: (r/fold (r/filter ... (r/map ... coll))) A very small change to the reducers API - passing the collection to the reduce and/or fold - would avoid the problem: (r/fold (r/filter ... (r/map ...)) coll) Anyway - I'd be very grateful for any insight into either of the above questions. Or for suggestions for an alternative approach that might be more fruitful. Many thanks in advance, -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: What guarantees (if any) does Clojure make about laziness?
On 15 Mar 2013, at 07:04, Meikel Brandmeyer (kotarak) m...@kotka.de wrote: this highly depends on the sequence function at hand. Usually they are guaranteed to be as lazy as possible. But the are two aspects: a) sometimes you need to look ahead to actually perform the action (eg. take-while or drop-while) and b) sometimes there might be a bug in the implementation and it is not as lazy as it could be. You can't do anything about a). And b) is very unlikely but also happened in the past. So the word I'm interested in in that sentence is guaranteed. I can see, from reading the source, what's guaranteed by a particular lazy structure and/or function. But that's just a statement about the current implementation, which might change in a future version of the language/library. Are these guarantees stated anywhere, or is my only recourse reading the source and/or experimenting with examples? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: What guarantees (if any) does Clojure make about laziness?
On 15 Mar 2013, at 08:28, Marko Topolnik marko.topol...@gmail.com wrote: To the best of my knowledge the only guarantee you get is the existence of an upper bound on the size of the eagerly fetched chunk, so a potentially infinite lazy sequence will not result in an endless loop/OOME. The whole mechanism is based on a best-effort policy. That sounds very plausible to me. Is there an official statement to that end anywhere? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: What guarantees (if any) does Clojure make about laziness?
On 15 Mar 2013, at 09:23, Marko Topolnik marko.topol...@gmail.com wrote: This is not about bureaucracy --- it's about API contract, Quite. you must preserve laziness whenever applicable, but you may not take advantage of it by assuming any guarantees. Erm - I certainly hope that this isn't true. Otherwise, I wouldn't be able to write: (take 10 (repeatedly 0)) Because, without *some* guarantees, it might never terminate and/or consume all memory. Of course, the only guarantee that this requires is the one that you mentioned in your earlier mail, that there will be *some* upper bound to how much of the sequence is realised. But it would be nice if there was an official statement to that end somewhere. And if there are other guarantees that can be relied upon, it would be nice to know what they were... -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Using transients within fold
I've been experimenting with reducers using a small example that counts the words in Wikipedia pages by parsing the Wikipedia XML dump. The basic structure of the code is: (frequencies (flatten (map get-words (get-pages where get-pages returns a lazy sequence of pages from the XML dump and get-words takes a page and returns a sequence of the words on that page. The above code takes ~40s to count the words on the first 1 pages. If I convert that code to use reducers, it runs in ~22s (yay!). If I convert it to use fold and therefore run in parallel, it runs in ~13s on my 4-core MacBook Pro. So it's faster (yay!) but nowhere near 4x faster (boo). The primary reason for this is that, in order to be able to use fold, I've had to write my own version of frequencies: (defn frequencies-parallel [words] (r/fold (partial merge-with +) (fn [counts x] (assoc counts x (inc (get counts x 0 words)) And, unlike the version in core, this doesn't use transients. If I replace the fold with reduce (i.e. make it run sequentially) it runs in ~43s. So, I *am* getting close to a 4x speedup from parallelising the code, but unfortunately I'm also seeing a 2x slowdown because I can't use transients. Can anyone think of any way that it would be possible to modify this code to use transients? Or any way to modify reducers to allow transients to be used? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Using transients within fold
On 14 Mar 2013, at 11:49, Meikel Brandmeyer (kotarak) m...@kotka.de wrote: that's not really possible at the moment. cf. https://groups.google.com/d/topic/clojure-dev/UbJlMO9XYjo/discussion and https://github.com/cgrand/clojure/commit/65e1acef03362a76f7043ebf3fe2fa277c581912 Dang. At least other people have the same problem, so perhaps there's a chance it'll be addressed :-) -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Using transients within fold
On 14 Mar 2013, at 13:13, David Powell djpow...@djpowell.net wrote: As a temporary hack, perhaps you could implement a deftype ReduceToTransient wrapper that implements CollReduce by calling reduce on the parameter, and then calling persistent! on the return value of reduce. You'd also need to implement CollFold so that the partitioning function produces wrapped results. Would that work? Hmm. Possibly. I'll have a think about it... -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
What guarantees (if any) does Clojure make about laziness?
Clojure's sequences are lazy - but is there anything that guarantees *how* lazy they are? To give a concrete example - given an infinite lazy sequence of promises: (def promises (repeatedly promise)) If, in one thread I do: (doseq [p (map deref promises)] (println p)) And in another thread, I call deliver on various elements of promises, will promises always be printed as soon as they're available? Or can things block for longer than they theoretically might because a later (undelivered) promise in the sequence is being realized? I've been doing some experiments, and in practice the above works just fine - promises are printed as soon as they theoretically might be. If, however, I put a flatten in place of the map then I seem to have to deliver one more promise before the first is printed. What I'm interested in here aren't practicalities. I'm interested in what Clojure and its libraries guarantee (if anything) about laziness. Is there an upper bound on how much of a sequence will be realized? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
Thanks Stuart - my Contributor Agreement is on its way. In the meantime, I've published foldable-seq as a library: https://clojars.org/foldable-seq I'd be very interested in any feedback on the code or how it works. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 12 Mar 2013, at 21:48, Stuart Sierra m...@stuartsierra.com wrote: See clojure.org/contributing it's all there. On Tuesday, March 12, 2013, Paul Butcher wrote: On 12 Mar 2013, at 18:26, Stuart Sierra the.stuart.sie...@gmail.com wrote: This might be an interesting contribution to clojure.core.reducers. I haven't looked at your code in detail, so I can't say for sure, but being able to do parallel fold over semi-lazy sequences would be very useful. I'd be delighted if this (or something like it) could make it into clojure.core.reducers, Stuart. What would I need to do to make this happen? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/8RKCjF00ukQ/unsubscribe?hl=en. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 13 Mar 2013, at 14:05, Jim foo.bar jimpil1...@gmail.com wrote: how come your project depends on the problematic version 1.5.0? 1.5.0 is problematic? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
Ah - sorry, missed that. I've just released version 0.2, which depends on 1.5.1. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 13 Mar 2013, at 14:13, Jim foo.bar jimpil1...@gmail.com wrote: there was a memory leak hence the 1.5.1 release the next day... Jim On 13/03/13 14:12, Paul Butcher wrote: On 13 Mar 2013, at 14:05, Jim foo.bar jimpil1...@gmail.com wrote: how come your project depends on the problematic version 1.5.0? 1.5.0 is problematic? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
So this turned out to be pretty easy. I've implemented a function called foldable-seq that takes a lazy sequence and turns it into something that can be folded in parallel. I've checked an example program that uses it to count words in a Wikipedia XML dump into GitHub: https://github.com/paulbutcher/foldable-seq The code for foldable-seq is here: https://github.com/paulbutcher/foldable-seq/blob/master/src/wordcount/reducers.clj#L60 On my 4-core MacBook Pro, I see a 40 second runtime without parallel-seq, 13 seconds with. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 11 Mar 2013, at 13:38, Paul Butcher p...@paulbutcher.com wrote: On 11 Mar 2013, at 11:00, Marko Topolnik marko.topol...@gmail.com wrote: The idea is to transform into a lazy sequence of eager chunks. That approach should work. Exactly. Right - I guess I should put my money where my mouth is and see if I can get it working... -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 12 Mar 2013, at 13:45, Marko Topolnik marko.topol...@gmail.com wrote: Nice going :) Is it really impossible to somehow do this from the outside, through the public API? I think that it *does* do it from the outside through the public API :-) I'm just reifying the (public) CollFold protocol. I do copy a bunch of helper functions related to fork/join which are private within the reducers namespace (which isn't particularly nice). But I guess that this will go away when (if?) Clojure has a proper API to support fork/join. But I thought it better to do that than reinvent the wheel. Or am I missing something? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 12 Mar 2013, at 13:52, Marko Topolnik marko.topol...@gmail.com wrote: That's what I meant, succeed by relying on the way f/j is used by the reducers public API, without copy-pasting the internals and using them directly. So I guess the answer is no. I don't believe that I could - the CollFold implementation effectively does a binary chop on the sequence which would force the whole sequence to be realised if I did the same. It's a sensible strategy for a fully realised data structure that already exists in memory, but not for a lazy sequence. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 12 Mar 2013, at 13:49, Adam Clements adam.cleme...@gmail.com wrote: How would feeding a line-seq into this compare to iota? And how would that compare to a version of iota tweaked to work in a slightly less eager fashion? It'll not suffer from the problem of having to drag the whole file into memory, but will incur the overhead of turning everything into JVM data structures that iota avoids. I don't imagine that it would be hard to modify iota to use a similar approach though. Although I imagine that Alan's better placed to have an opinion on that :-) -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 12 Mar 2013, at 15:55, Alan Busby thebu...@thebusby.com wrote: If Paul wouldn't mind I'd like to add a a similar seq function to Iota that would allow for index-less processing like he did in foldable-seq. Paul would be delighted :-) -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 12 Mar 2013, at 18:26, Stuart Sierra the.stuart.sie...@gmail.com wrote: This might be an interesting contribution to clojure.core.reducers. I haven't looked at your code in detail, so I can't say for sure, but being able to do parallel fold over semi-lazy sequences would be very useful. I'd be delighted if this (or something like it) could make it into clojure.core.reducers, Stuart. What would I need to do to make this happen? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 11 Mar 2013, at 10:40, Jim foo.bar jimpil1...@gmail.com wrote: why can't you 'vec' the result of xml/parse and then use fold on that? Is it a massive seq? In my case, it's the Wikipedia XML dump, so around 40GiB (so no, that wouldn't work :-) -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On 11 Mar 2013, at 11:00, Marko Topolnik marko.topol...@gmail.com wrote: The idea is to transform into a lazy sequence of eager chunks. That approach should work. Exactly. Right - I guess I should put my money where my mouth is and see if I can get it working... -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
fold over a sequence
As things currently stand, fold can be used on a sequence-based reducible collection, but won't be parallel. I'm currently working on code that processes XML generated by clojure.data.xml/parse, and would love to do so in parallel. I can't immediately see any reason why it wouldn't be possible to create a version of CollFold that takes a sequence and chunks it so that it can be folded in parallel. Has anyone tried this yet? Is there some gotcha lurking to catch me out? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: STM - a request for war stories
On 14 Dec 2012, at 13:52, Rich Hickey richhic...@gmail.com wrote: On Dec 14, 2012, at 12:55 AM, Paul Butcher wrote: Rich - what is the soundbite description of Clojure's concurrency model you're happiest with? Ah, soundbites, the foundation of modern programmer education :) Maybe I should have said least unhappy? :-) Certainly, pigeonholing Clojure as STM-based is way off the mark. Agreed 100%. Also, a chapter on STM that doesn't distinguish Clojure and Haskell's functional STM approach from the ordinary wrap your old imperative code in transactions approach is going to miss the biggest point. Also agreed 100%. That's a (very) large part of why I've chosen Clojure as the language for this chapter. I guess this would be my alternative chapter title proposal: Functional Programming + Reference Types I can see that logic. Unfortunately, as well as this chapter, I also plan to have a chapter on functional programming using Haskell's Par and Eval monads, so I need some kind of title that draws the distinction. I know that there are similarities between the approaches, but they also have very distinct flavours (as you say in your The Database as a Value talk, it's thoroughly unclear how to represent an immutable value that performs IO as a monad). For the avoidance of doubt, I don't for one second disagree that Clojure is more than STM. Nor do I disagree that it's important that I cover the separation of values from identity, atoms and agents as well as refs/STM. I'm just trying to come up with a pithy chapter title. How about transactional state? Immutable state? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 14 Dec 2012, at 13:52, Rich Hickey richhic...@gmail.com wrote: On Dec 14, 2012, at 12:55 AM, Paul Butcher wrote: On 14 Dec 2012, at 00:30, kovas boguta kovas.bog...@gmail.com wrote: My recommendation is either Persistent Datastructures or Database as a Value Interesting. I'd be interested to hear others thoughts on this. In particular Rich's Rich - what is the soundbite description of Clojure's concurrency model you're happiest with? Ah, soundbites, the foundation of modern programmer education :) How about this: Clojure doesn't need a concurrency model. It has a state model that can be realized in multiple concurrency-safe ways. For the state model: Separate identities and values The best thing about Clojure's reference types is that they exist. The best thing about Clojure is that you rarely need them. Certainly, pigeonholing Clojure as STM-based is way off the mark. Also, a chapter on STM that doesn't distinguish Clojure and Haskell's functional STM approach from the ordinary wrap your old imperative code in transactions approach is going to miss the biggest point. People reading a chapter on the 'STM model' independent of the functional approach of the languages in which it has succeeded are bound to be disappointed and ill-informed. I'd argue that the 'concurrency model' of Clojure and Haskell is 'functional programming' + reference types. Their STMs are a subset of that. As Kovas has pointed out, the symbiosis of persistent data structures and this reference approach is fundamental. I guess this would be my alternative chapter title proposal: Functional Programming + Reference Types Rich -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: STM - a request for war stories
On 14 Dec 2012, at 00:22, Patrick Logan patrickdlo...@gmail.com wrote: Another concurrency model I've used a great deal is the tuplespace model, specifically javaspaces. This is an often forgotten model that has a lot to offer with a high expressiveness to complexity ratio. Ah! That brings back memories :-) I wrote my PhD thesis on Linda back in the early 90s. I agree that it's a cute model (I thought that it was cute enough to spend 3 years of my life on it!) but I'm not aware of anyone using it in anger now? I'd be very interested to hear about live projects. I know about Javaspaces/JINI/River, but I've not really heard about anyone using them? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: STM - a request for war stories
On 14 Dec 2012, at 00:30, kovas boguta kovas.bog...@gmail.com wrote: My recommendation is either Persistent Datastructures or Database as a Value Interesting. I'd be interested to hear others thoughts on this. In particular Rich's Rich - what is the soundbite description of Clojure's concurrency model you're happiest with? If you can't incorporate novelty without cloning the entire datastructure, thats not that useful. I'm not 100% sure what you mean by this - can you expand? -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: STM - a request for war stories
Hey Stuart, Thanks for the response. What I'm trying to do is keep each chapter focussed on an approach, rather than a language. For example, in the chapter on Actors, I'll be showing examples in Scala, but the discussion won't be (I hope!) particularly Scala-specific. I hope to leave the reader with general lessons which could be applied to Scala, Erlang, or any other language with Actor support. Similarly, when talking about Threads and Locks, I'll be showing examples in Java, but the lessons should be equally applicable to C/C++, etc. I completely take your point about Clojure's approach being a great deal more than STM. I guess that I chose STM as the title because it's got visibility - people are talking about it, and there will be an expectation on the part of the reader that any book that covers concurrency will spend some time talking about it. I'd be very interested to hear any suggestions for an alternative chapter title. I guess what best sums up Clojure's approach is that it separates state from identity - but Separating State from Identity isn't exactly pithy, and I fear won't mean much at first glance to most readers. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 11 Dec 2012, at 19:41, Stuart Halloway stuart.hallo...@gmail.com wrote: Hi Paul, If it isn't too late to change your chapter title, I would encourage emphasizing Clojure's model of references and values in general, and the option of implementing a variety of different reference semantics that all conform to the same basic API shape. That general approach has been game-changing for me, and the STM occupies a rather small niche in the overall space. Datomic stores the entire database in an atom (not an STM ref), and updates it with a call to swap! It is literally no more complex than a trivial hackneyed book example. :-) Cheers, Stu On Sun, Dec 2, 2012 at 11:03 AM, Paul Butcher p...@paulbutcher.com wrote: All, I have a request which I hope the members of this group are uniquely positioned to help with. I have recently started working on a new book for The Pragmatic Programmers with the working title Seven Concurrency Models in Seven Weeks (it follows on from their existing Seven Languages and Seven Databases titles). One of the approaches that I'll be covering is STM, and I'll be presenting it in Clojure. What I'd like to solicit are war stories about problems you've solved using STM, which demonstrate the strengths of the technique over and above (say) threads and locks. I'm looking for real-world examples instead of presenting yet another hackneyed atomically-make-a-bank-account-withdrawal :-) Very many thanks in advance for your help! -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: STM - a request for war stories
On 10 Dec 2012, at 12:56, Chas Emerick c...@cemerick.com wrote: I'd be surprised if Paul doesn't hear from people directly I wish that that were true, but no, I've not had anyone get in touch off-list. Many thanks, Marko, for resurrecting the thread - I'm still definitely keen to hear of first-hand experiences! -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: STM - a request for war stories
On 10 Dec 2012, at 13:37, Marko Topolnik marko.topol...@gmail.com wrote: But concurrency is all about performance and throughput. So where is the benefit of using correct, slow concurrent mutation? I guess in a write-seldom, read-often scenario. I'm not at all sure that that's true. There are plenty of occasions where concurrency is about being able to do more than one thing at a time, and not necessarily about making something faster. For example, your mobile 'phone is concurrent because, while it's playing music to you, it also wants to notice when you poke the screen and listen for incoming calls/messages from the network. And your IDE is concurrent so that it can check the syntax of your code in the background while the UI remains responsive. I'm not, of course, saying that performance isn't important - even in cases such as the above. It would be a major problem if everything was an order of magnitude slower just because I tried to do two things at the same time. But there are certainly plenty of occasions where we might choose to write concurrent code without our focus being on performance per-se. -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher On 10 Dec 2012, at 13:37, Marko Topolnik marko.topol...@gmail.com wrote: On Monday, December 10, 2012 1:56:08 PM UTC+1, Chas Emerick wrote: On Dec 10, 2012, at 5:39 AM, Marko Topolnik wrote: I personally have never used STM in nontrivial ways (AFAIC), but that's due more to the demands of the problems I run into more than anything else. On the other hand, I have used, abused, and benefitted from agents in umpteen ways. Actually, I have often done things using agents that might otherwise been done using STM or other similar approaches, simply to ensure that: (a) the processing involved can be readily parallelized, and (b) if necessary, the system can be partitioned/distributed with minimal impact to the architecture, since — if you're careful about things — it doesn't matter whether a send is evaluated in an in-process agent or one housed in a different server/VM/whatever The argument (b) is an even better fit (or, should we say, perfect fit) for Actors, as implemented in Erlang. It's true that STM is all or nothing, but it is so over the scope of refs you choose. If there's some side-effecting bit you need to do somewhere, then clearly that's not going to fit within a transaction…but that bit will often fit just fine in a send-off to an agent provoked _by_ a transaction. send-off fails to be useful whenever you need the results within the transaction (quite often, that is). My guess is, if your task is something purely computational and amenable to massive parallelization, you may have a go with STM; if it's just about business logic accessible concurrently by many clients, you won't find it workable. If your task is purely computational and amenable to massive parallelization, you _should_ use agents whenever possible. STM provides for coordination in order to enforce consistency; unless all of your operations are commutative (in which case, you should probably be using agents anyway), a program using STM _will_ provoke retries and other means to route around ref contention. This is acceptable because STM is all about maintaining correctness in the face of concurrent mutation, and not necessarily about performance, aggregate throughput, and so on. But concurrency is all about performance and throughput. So where is the benefit of using correct, slow concurrent mutation? I guess in a write-seldom, read-often scenario. On the other hand, ref readers are _never_ blocked (regardless of what's going on on the write side), so the data in such refs is always accessible. This sounds like an ideal combination for business logic (as nebulous a term as that is) to me. Business logic almost always involves communication with outside systems (since it's usually about integration of many existing systems). Even if not, a scalable solution must be stateless (a prerequisite for cluster deployment) and any durable state must go into a single datasource common to all cluster nodes. Again, these datasources don't participate in an STM transaction. Maybe this would be a major route of improvement: integrate the STM with external datasource transactions. But this is still quite removed from the present. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com
STM - a request for war stories
All, I have a request which I hope the members of this group are uniquely positioned to help with. I have recently started working on a new book for The Pragmatic Programmers with the working title Seven Concurrency Models in Seven Weeks (it follows on from their existing Seven Languages and Seven Databases titles). One of the approaches that I'll be covering is STM, and I'll be presenting it in Clojure. What I'd like to solicit are war stories about problems you've solved using STM, which demonstrate the strengths of the technique over and above (say) threads and locks. I'm looking for real-world examples instead of presenting yet another hackneyed atomically-make-a-bank-account-withdrawal :-) Very many thanks in advance for your help! -- paul.butcher-msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: p...@paulbutcher.com AIM: paulrabutcher Skype: paulrabutcher -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
