Re: non-lazy clojure?
A followup on these issues, after experimenting a bit with reducers, just for anyone who may be interested: - Naive use of reducers in place of seq functions, throughout my program, got messy and led to some new problems with memory management, I think (or at least to jerkier execution over time). I realized that reducers probably shouldn't be used so indiscriminately. - With more moderate use of reducers, and using fold, I do indeed see performance improvements relative to sequential code and also relative to the previous version of my code that used agents for concurrency. I don't actually seem to be getting much benefit from the way that reducers avoid building intermediate results -- all of the benefit appears to be due to using a combination of r/fold and one r/map to process collection elements concurrently. I've only done small numbers of tests and they all involve randomness, but I do see improvements on the order of 1/3. - The best behaving/looking approach for me seems to be to stick with lazy sequence operations and aggressive de-lazifying for the bulk of my code (possibly cleaning some things up by defining my own forv, repeatv, etc., or possibly rewriting some of the core lazy sequence functions to not be lazy in the first place), but to use r/fold and r/map for concurrency for my top-level tasks. I'm getting the best results by using reducers only in one place, in my definition of a concurrent mapping function, which I then use only for top-level functions (never nested): (defn pmapall [f coll] (if single-thread-mode (doall (map f coll)) (r/fold 1 r/cat r/append! (r/map f coll This replaces my previous agent-based version: (defn pmapall [f coll] (if single-thread-mode (doall (map f coll)) (let [agents (map #(agent % :error-handler (fn [agnt except] (clojure.repl/pst except 1000) (System/exit 0))) coll)] (dorun (map #(send % f) agents)) (apply await agents) (doall (map deref agents) The one that uses reducers is neater and seems to make my program run faster, although I'm not really sure why. In any event, I'm using reducers now only as an alternative concurrency mechanism and solving my original problems by de-lazification. -Lee On Jun 4, 2014, at 7:21 PM, Lee Spector lspec...@hampshire.edu wrote: On Jun 4, 2014, at 1:29 PM, Timothy Baldridge tbaldri...@gmail.com wrote: Although your original complaint was about clojure seqs being lazy. It should be noted that reducers are also lazy down to the point of a fold or reduce, so I'm not sure what you're really getting there. It wouldn't be hard at all to write map, filter, remove, etc. in terms of list operations. Perhaps that's what you're looking for? If I understand your original complaint, you want map, filter, etc, to be eager. True, my original concern was (and my main concern still is) to avoid crashes (in this case, stack overflow errors) that stem from unexpected (to me) consequences of laziness. Those problems could indeed be solved by rewriting map and filter etc, or (easier) by wringing out the laziness wherever it arises, e.g. by forcing everything to be a vector (or a list, as you suggest). It's a little cumbersome to do this everywhere, though, and I was wondering if there was an existing library or approach for this issue. Then, however, it was suggested that I could have my cake and another kind of cake too, by using reducers. The suggestion was that this could banish the laziness-related issues while also providing significant performance improvements. Sounds good to me! I'm still working on getting my current project to behave well using reducers, but it seems promising. On the other hand, time spent learning how lazy seqs work, and the caveats involved will make it easier for you to understand the code produced by the rest of the community. So perhaps that's the better option. I agree that it's important to understand lazy sequences, and I do think I understand the core concepts reasonably well (and I've implemented lazy evaluation in other languages, etc.). But I've also traced some really annoying and hard to find bugs to unexpected (and sometimes never fully explained) consequences of laziness, so I'd like to find the best ways to avoid it when I don't really want it. If that best way turns out to make things run faster too then that'd be fantastic. -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you
Re: non-lazy clojure?
Oops -- something was wrong with my benchmarks, and my improvements on the order of 1/3 was wrong. I still see improvements with r/fold as compared to my agent-based approach, but the difference now appears to be only something like 1/20. -Lee On Jun 5, 2014, at 7:19 PM, Lee Spector lspec...@hampshire.edu wrote: A followup on these issues, after experimenting a bit with reducers, just for anyone who may be interested: - Naive use of reducers in place of seq functions, throughout my program, got messy and led to some new problems with memory management, I think (or at least to jerkier execution over time). I realized that reducers probably shouldn't be used so indiscriminately. - With more moderate use of reducers, and using fold, I do indeed see performance improvements relative to sequential code and also relative to the previous version of my code that used agents for concurrency. I don't actually seem to be getting much benefit from the way that reducers avoid building intermediate results -- all of the benefit appears to be due to using a combination of r/fold and one r/map to process collection elements concurrently. I've only done small numbers of tests and they all involve randomness, but I do see improvements on the order of 1/3. - The best behaving/looking approach for me seems to be to stick with lazy sequence operations and aggressive de-lazifying for the bulk of my code (possibly cleaning some things up by defining my own forv, repeatv, etc., or possibly rewriting some of the core lazy sequence functions to not be lazy in the first place), but to use r/fold and r/map for concurrency for my top-level tasks. I'm getting the best results by using reducers only in one place, in my definition of a concurrent mapping function, which I then use only for top-level functions (never nested): (defn pmapall [f coll] (if single-thread-mode (doall (map f coll)) (r/fold 1 r/cat r/append! (r/map f coll This replaces my previous agent-based version: (defn pmapall [f coll] (if single-thread-mode (doall (map f coll)) (let [agents (map #(agent % :error-handler (fn [agnt except] (clojure.repl/pst except 1000) (System/exit 0))) coll)] (dorun (map #(send % f) agents)) (apply await agents) (doall (map deref agents) The one that uses reducers is neater and seems to make my program run faster, although I'm not really sure why. In any event, I'm using reducers now only as an alternative concurrency mechanism and solving my original problems by de-lazification. -Lee On Jun 4, 2014, at 7:21 PM, Lee Spector lspec...@hampshire.edu wrote: On Jun 4, 2014, at 1:29 PM, Timothy Baldridge tbaldri...@gmail.com wrote: Although your original complaint was about clojure seqs being lazy. It should be noted that reducers are also lazy down to the point of a fold or reduce, so I'm not sure what you're really getting there. It wouldn't be hard at all to write map, filter, remove, etc. in terms of list operations. Perhaps that's what you're looking for? If I understand your original complaint, you want map, filter, etc, to be eager. True, my original concern was (and my main concern still is) to avoid crashes (in this case, stack overflow errors) that stem from unexpected (to me) consequences of laziness. Those problems could indeed be solved by rewriting map and filter etc, or (easier) by wringing out the laziness wherever it arises, e.g. by forcing everything to be a vector (or a list, as you suggest). It's a little cumbersome to do this everywhere, though, and I was wondering if there was an existing library or approach for this issue. Then, however, it was suggested that I could have my cake and another kind of cake too, by using reducers. The suggestion was that this could banish the laziness-related issues while also providing significant performance improvements. Sounds good to me! I'm still working on getting my current project to behave well using reducers, but it seems promising. On the other hand, time spent learning how lazy seqs work, and the caveats involved will make it easier for you to understand the code produced by the rest of the community. So perhaps that's the better option. I agree that it's important to understand lazy sequences, and I do think I understand the core concepts reasonably well (and I've implemented lazy evaluation in other languages, etc.). But I've also traced some really annoying and hard to find bugs to unexpected (and sometimes never fully explained) consequences of laziness, so I'd like to find the best ways to avoid it when I don't really want it. If that best way turns out to make things run faster too then that'd be fantastic. -Lee -- Lee Spector, Professor of Computer Science Cognitive Science, Hampshire College 893
Re: non-lazy clojure?
Fair enough. Fortunately, Clojure provides so many different tools to select from in creating you perfect recipe. ;-) I'm glad to hear that reducers ultimately provided you with some benefits over your previous concurrency approach. The one thing that seems rather odd to me though is that your group-size is 1. I'm presuming that the function you're r/mapping must take a substantial amount of time and resources for that to be efficient. Have you experimented with larger group sizes to avoid too much thread swapping? ~Gary -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Jun 5, 2014, at 8:51 PM, Gary Johnson gwjoh...@uvm.edu wrote: Fair enough. Fortunately, Clojure provides so many different tools to select from in creating you perfect recipe. ;-) I'm glad to hear that reducers ultimately provided you with some benefits over your previous concurrency approach. The one thing that seems rather odd to me though is that your group-size is 1. I'm presuming that the function you're r/mapping must take a substantial amount of time and resources for that to be efficient. Have you experimented with larger group sizes to avoid too much thread swapping? Yes, I've tried other group sizes and I get nearly no speedup over single-threaded code with the default group size (512). Group size 1 seems to be about as good as any other value i've tried. -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Jun 2, 2014, at 7:14 PM, Gary Johnson gwjoh...@uvm.edu wrote: Hey Lee, I would second Jozef's suggestion that you look into using the reducers library when you need non-lazy sequence operations. [etc] On Jun 2, 2014, at 10:38 PM, Lee Spector lspec...@hampshire.edu wrote: Gary: That's compelling indeed, and I will look into it more! Some quick notes and a question from my first look into this: - I watched a Rich Hickey reducers video, was riveted, and see that they are beautiful. I particularly appreciated his brief aside about how lazy seqs have independent utility. - In practice, for the little program I posted about previously, switching to reducers involved a couple of initially unexpected complications, some of now make sense to me but others of which don't fully: results of reducers sometimes have to be explicitly converted, e.g. with into []; r/map doesn't take multiple collections; I don't immediately see elegant reducer-based approaches to uses of for or repeat, etc. - My initial swapping of clojure.core.reducers functions for lazy seq (and agent-based parallel computing) functions seems to make my performance worse rather than better. I realize that there are several possible explanations for this and I have to look at my usage more carefully. It's definitely possible that I'm doing more than one thing wrong, but one question that this leads to is: - If I operate on a vector with a sequence of r/map and r/filter operations and finally with into [] to get back a vector, then I think that fold will be called within the call to into, and that parallelism should happen there... right? But if that's right, how do I control the group size that the call to fold uses in this case? I see how to specify the group size for a direct call to fold, but not for other function in the library. Thanks, -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
Hey Lee,
(vec ...) is NOT the same as (into [] ...) in this case.
Whenever you use a reducing function, like r/map, r/filter, r/mapcat, and
so on, you are not, in fact, performing any computations on the collection
to which you apply it. These functions simply wrap the collection with a
kind of delayed operation that will later be triggered by applying reduce
to it.
So, if I have a collection called samples, and I run (r/map inc samples),
I just get a reducible object back. Same thing with a chain of these
functions as in (r/filter odd? (r/map inc samples)). I just get back a
reducible object (which itself is wrapping another reducible object). When
you run reduce on this whole thing, each of the delayed operations are
combined together into one giant reduce function, as if you had written
something like this:
(reduce + (r/filter odd? (r/map inc samples)))
= (reduce (fn [sum sample] (let [sample' (inc sample)] (if (odd?
sample') (+ sample' sum) sum))) 0 samples)
This is the reason that you need to use (into [] ...) rather than (vec
...). Running vec on a reducible object will just throw a RuntimeException.
When you use into, you will be applying reduce (since that's how it is
implemented), and as long as you use into with a vector, map, or set, it
will be run using transients for efficiency.
Uses transients:
(into [] (r/map inc (range 10)))
(into {} (r/map #(vector % 0) (range 10)))
(into #{}| (r/map inc (range 10)))
Doesn't use transients (but still works). Note that the list will have
been reversed because of the nature of list consing:
(into () (r/map inc (range 10)))
Alright, hopefully that's enough from me for now. Good luck with your
program.
~Gary
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Jun 4, 2014, at 12:59 PM, Gary Johnson gwjoh...@uvm.edu wrote: Hey Lee, (vec ...) is NOT the same as (into [] ...) in this case. [etc] Thanks Gary -- very clear and helpful. -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
Hey Lee, answers below. Also make sure to read my other post at 12:59pm today regarding the behavior of vec vs. into for reducible collections. On Wednesday, June 4, 2014 12:51:45 PM UTC-4, Lee wrote: Some quick notes and a question from my first look into this: - I watched a Rich Hickey reducers video, was riveted, and see that they are beautiful. I particularly appreciated his brief aside about how lazy seqs have independent utility. - In practice, for the little program I posted about previously, switching to reducers involved a couple of initially unexpected complications, some of now make sense to me but others of which don't fully: results of reducers sometimes have to be explicitly converted, e.g. with into []; r/map doesn't take multiple collections; I don't immediately see elegant reducer-based approaches to uses of for or repeat, etc. It's true that the library does have these limitations. You could, of course, just implement a for-like macro that uses the reducing operations under the hood. Easier (but less readable) is to just convert your for into the equivalent r/map and r/filter operations. When using repeat, make sure the result sequence will be finite and call vec on it directly. Then you can apply the reducing functions to it and still have a foldable collection. - My initial swapping of clojure.core.reducers functions for lazy seq (and agent-based parallel computing) functions seems to make my performance worse rather than better. I realize that there are several possible explanations for this and I have to look at my usage more carefully. It's definitely possible that I'm doing more than one thing wrong, but one question that this leads to is: Hard to say why this would be without seeing your code, but the first thing that comes to mind is that if you aren't calling fold or foldcat anywhere, then you won't be getting any parallelization from using reducers. So maybe your performance decrease is because you're now running single-threaded. - If I operate on a vector with a sequence of r/map and r/filter operations and finally with into [] to get back a vector, then I think that fold will be called within the call to into, and that parallelism should happen there... right? But if that's right, how do I control the group size that the call to fold uses in this case? I see how to specify the group size for a direct call to fold, but not for other function in the library. No, fold will not be called in into. The definition of into uses reduce, which is single-threaded. What you want is one of the following two formulations: (fold group-size cat append! foldable-collection) ;; if you want to specify the group size (foldcat foldable-collection) ;; if you are happy with the default 512 group size In both cases, the foldable-collection is just one of your (r/map f (r/filter p? initial-collection)) forms. For it to be foldable though, initial-collection needs to be a vector, map, or set (i.e., a tree-like collection). I hope that helps. ~Gary Thanks, -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
Although your original complaint was about clojure seqs being lazy. It should be noted that reducers are also lazy down to the point of a fold or reduce, so I'm not sure what you're really getting there. It wouldn't be hard at all to write map, filter, remove, etc. in terms of list operations. Perhaps that's what you're looking for? If I understand your original complaint, you want map, filter, etc, to be eager. On the other hand, time spent learning how lazy seqs work, and the caveats involved will make it easier for you to understand the code produced by the rest of the community. So perhaps that's the better option. Timothy On Wed, Jun 4, 2014 at 11:20 AM, Gary Johnson gwjoh...@uvm.edu wrote: Hey Lee, answers below. Also make sure to read my other post at 12:59pm today regarding the behavior of vec vs. into for reducible collections. On Wednesday, June 4, 2014 12:51:45 PM UTC-4, Lee wrote: Some quick notes and a question from my first look into this: - I watched a Rich Hickey reducers video, was riveted, and see that they are beautiful. I particularly appreciated his brief aside about how lazy seqs have independent utility. - In practice, for the little program I posted about previously, switching to reducers involved a couple of initially unexpected complications, some of now make sense to me but others of which don't fully: results of reducers sometimes have to be explicitly converted, e.g. with into []; r/map doesn't take multiple collections; I don't immediately see elegant reducer-based approaches to uses of for or repeat, etc. It's true that the library does have these limitations. You could, of course, just implement a for-like macro that uses the reducing operations under the hood. Easier (but less readable) is to just convert your for into the equivalent r/map and r/filter operations. When using repeat, make sure the result sequence will be finite and call vec on it directly. Then you can apply the reducing functions to it and still have a foldable collection. - My initial swapping of clojure.core.reducers functions for lazy seq (and agent-based parallel computing) functions seems to make my performance worse rather than better. I realize that there are several possible explanations for this and I have to look at my usage more carefully. It's definitely possible that I'm doing more than one thing wrong, but one question that this leads to is: Hard to say why this would be without seeing your code, but the first thing that comes to mind is that if you aren't calling fold or foldcat anywhere, then you won't be getting any parallelization from using reducers. So maybe your performance decrease is because you're now running single-threaded. - If I operate on a vector with a sequence of r/map and r/filter operations and finally with into [] to get back a vector, then I think that fold will be called within the call to into, and that parallelism should happen there... right? But if that's right, how do I control the group size that the call to fold uses in this case? I see how to specify the group size for a direct call to fold, but not for other function in the library. No, fold will not be called in into. The definition of into uses reduce, which is single-threaded. What you want is one of the following two formulations: (fold group-size cat append! foldable-collection) ;; if you want to specify the group size (foldcat foldable-collection) ;; if you are happy with the default 512 group size In both cases, the foldable-collection is just one of your (r/map f (r/filter p? initial-collection)) forms. For it to be foldable though, initial-collection needs to be a vector, map, or set (i.e., a tree-like collection). I hope that helps. ~Gary Thanks, -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to
Re: non-lazy clojure?
On Jun 4, 2014, at 1:20 PM, Gary Johnson gwjoh...@uvm.edu wrote: - If I operate on a vector with a sequence of r/map and r/filter operations and finally with into [] to get back a vector, then I think that fold will be called within the call to into, and that parallelism should happen there... right? But if that's right, how do I control the group size that the call to fold uses in this case? I see how to specify the group size for a direct call to fold, but not for other function in the library. No, fold will not be called in into. The definition of into uses reduce, which is single-threaded. What you want is one of the following two formulations: (fold group-size cat append! foldable-collection) ;; if you want to specify the group size (foldcat foldable-collection) ;; if you are happy with the default 512 group size In both cases, the foldable-collection is just one of your (r/map f (r/filter p? initial-collection)) forms. For it to be foldable though, initial-collection needs to be a vector, map, or set (i.e., a tree-like collection). I hope that helps. Definitely helpful. With the fold version, and messing with the group size, I'm seeing much higher multicore utilization and I think I'm seeing speedups too (although it's a little hard to measure because of randomness). I'm also having new problems (pauses that get worse over time... GC?) but I'll experiment more before I bother anyone else with those. Thanks so much Gary, -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Jun 4, 2014, at 1:29 PM, Timothy Baldridge tbaldri...@gmail.com wrote: Although your original complaint was about clojure seqs being lazy. It should be noted that reducers are also lazy down to the point of a fold or reduce, so I'm not sure what you're really getting there. It wouldn't be hard at all to write map, filter, remove, etc. in terms of list operations. Perhaps that's what you're looking for? If I understand your original complaint, you want map, filter, etc, to be eager. True, my original concern was (and my main concern still is) to avoid crashes (in this case, stack overflow errors) that stem from unexpected (to me) consequences of laziness. Those problems could indeed be solved by rewriting map and filter etc, or (easier) by wringing out the laziness wherever it arises, e.g. by forcing everything to be a vector (or a list, as you suggest). It's a little cumbersome to do this everywhere, though, and I was wondering if there was an existing library or approach for this issue. Then, however, it was suggested that I could have my cake and another kind of cake too, by using reducers. The suggestion was that this could banish the laziness-related issues while also providing significant performance improvements. Sounds good to me! I'm still working on getting my current project to behave well using reducers, but it seems promising. On the other hand, time spent learning how lazy seqs work, and the caveats involved will make it easier for you to understand the code produced by the rest of the community. So perhaps that's the better option. I agree that it's important to understand lazy sequences, and I do think I understand the core concepts reasonably well (and I've implemented lazy evaluation in other languages, etc.). But I've also traced some really annoying and hard to find bugs to unexpected (and sometimes never fully explained) consequences of laziness, so I'd like to find the best ways to avoid it when I don't really want it. If that best way turns out to make things run faster too then that'd be fantastic. -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Mon 2 Jun 2014 at 10:38:23PM -0400, Lee Spector wrote: PS would a call to vec do the same thing as into [] here? IIRC vec and into [] are equivalent unless the source collection implements IEditableCollection, in which case transients are used for a significant performance boost. guns pgpL2Biq3LnWB.pgp Description: PGP signature
Re: non-lazy clojure?
On Monday, June 2, 2014 3:32:59 PM UTC-5, Lee wrote: I've generally liked Clojure's pervasive laziness. It's cute and it sometimes permits lovely, elegant approaches to particular programming problems. After worrying about some bad potential problems with mutation of data structures (well all have to side-effect sometimes) that require some care with laziness--and not just my newbie oh darn I forgot that line was lazy problems--I had also come to feel that laziness wasn't worth its benefits. Cute, yeah, but the fact that I can do (take n (generate-infinite-seq)) rather than(generate-finite-seq n) is not helpful, and if I don't need to keep the whole thing in memory, then I can write a loop that doesn't. But so much of what's convenient about Clojure is lazy, so it's a hard thing to avoid. I've begun to see that laziness can really provide a modularity benefit. My main project is an application that generates subsequent states of a simulation using iterate, which generates a lazy sequence. I can do things like this: (nth (map write-summary-data-about-each-state-to-file (map display-some-data-for-debugging-about-each-state (generate-infinite-sequence-of-states initial-state))) index-of-last-state-I-care-about) I typically don't need all information about every subsequent state; I just need to know a few things, and that's what those functions that are mapped do. The important point is that those functions I'm mapping over the sequence of states are optional. I need different ones in different situations. In the first, Common Lisp version of this application, I embedded call to every such function into my main loop, and wrapped each one in an if test on a separate Boolean variable. So I had to embed every call that I might possibly want to use into the main loop. Now my main loop, i.e. the function that iterate calls, is only a few lines long, and I can add arbitrary reporting functions whenever I want using map. None of that is a solution to the problems that laziness brings, but I now think it's possible that it's worth dealing with laziness's drawbacks. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
non-lazy clojure?
I've generally liked Clojure's pervasive laziness. It's cute and it sometimes permits lovely, elegant approaches to particular programming problems. And although I've long known that it can get you into trouble in a few unusual cases -- I think I recall seeing a nice blog post on issues related to binding and laziness by Chas Emerick several years ago -- it has generally seemed safe enough for most purposes. I figured that while it may not always provide benefits it should at least be mostly harmless. However, after spending a couple of days tracking down a particularly confusing bug, and finding once again (this has happened to me at least a few times) that the bug was due to laziness that I never really wanted in the first place, I'm reconsidering. I don't have a pared-down version of the bug that I was just dealing with (see below for a pointer to the non-pared-down version), but it goes away when I wring out all of the laziness. In this particular case I do so by calling vec or doall on the results of all of my calls to map and other functions function that produce lazy sequences (none of which I actually want to be lazy in this application). I was getting StackOverflowErrors and I have a half-baked half-theory that it was due to some bad interaction between laziness and garbage collection, with the JVM not realizing that it could reclaim lazy things that weren't yet fully evaluated. That doesn't make complete sense to me, and maybe that kind of bad interaction is not even possible -- I don't know. But I do know that all is well when I wring out all of the laziness. One possible lesson from this experience (which was particularly frustrating, BTW, because the stack backtraces were particularly uninformative [1]) is that I should just be careful to surround any call to any function that produces a lazy sequence with something like vec that ensures that the laziness will be eliminated as soon as it is created -- except in those relatively rare cases where I really want the laziness. But this would be a pain, and ugly. I rely heavily on Clojure's sequence processing functions and I would hate to clutter up every call to every one of them or to have to reimplement my own non-lazy versions of everything. Is there a more elegant way? Maybe somebody has already made non-lazy clones of all of the core functions that normally produce lazy sequences? And if so, maybe there's some clean way to cause my code to use the non-lazy versions unless specifically directed to use the lazy versions? I realize that this is fighting with a core feature of Clojure, and that the idea will probably rub lots of folks the wrong way. But I've now been down this road a couple of times and I'm beginning to think that I'd spend less time tracking down mysterious bugs if I could avoid laziness more easily. In case anybody is motivated to look into the specific bug that I was just dealing with, I've put the project at http://hampshire.edu/lspector/temp/world2D.zip. If you do lein run in its present state it will run fine, making little balls bounce around. (This is a very early version of some code that I plan to use both for teaching an AI class and for some ALife experiments...) But if you edit src/world2D/vec2D.clj, commenting out the definitions of *v, +v, and -v, and uncommenting the alternatives (which are the same except that they lack the calls to vec) then after running for a few seconds or a minute or so you'll get the stack overflow. Thanks for any suggestions, -Lee [1] The stack traces I was getting provide no information about where/what the offending lazy sequences are, since they don't show locals etc. I see no references to any of my own code, just a repeating sequence that starts: StackOverflowError clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) [etc] -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
mapv a bit shorter :) Luc P. I've generally liked Clojure's pervasive laziness. It's cute and it sometimes permits lovely, elegant approaches to particular programming problems. And although I've long known that it can get you into trouble in a few unusual cases -- I think I recall seeing a nice blog post on issues related to binding and laziness by Chas Emerick several years ago -- it has generally seemed safe enough for most purposes. I figured that while it may not always provide benefits it should at least be mostly harmless. However, after spending a couple of days tracking down a particularly confusing bug, and finding once again (this has happened to me at least a few times) that the bug was due to laziness that I never really wanted in the first place, I'm reconsidering. I don't have a pared-down version of the bug that I was just dealing with (see below for a pointer to the non-pared-down version), but it goes away when I wring out all of the laziness. In this particular case I do so by calling vec or doall on the results of all of my calls to map and other functions function that produce lazy sequences (none of which I actually want to be lazy in this application). I was getting StackOverflowErrors and I have a half-baked half-theory that it was due to some bad interaction between laziness and garbage collection, with the JVM not realizing that it could reclaim lazy things that weren't yet fully evaluated. That doesn't make complete sense to me, and maybe that kind of bad interaction is not even possible -- I don't know. But I do know that all is well when I wring out all of the laziness. One possible lesson from this experience (which was particularly frustrating, BTW, because the stack backtraces were particularly uninformative [1]) is that I should just be careful to surround any call to any function that produces a lazy sequence with something like vec that ensures that the laziness will be eliminated as soon as it is created -- except in those relatively rare cases where I really want the laziness. But this would be a pain, and ugly. I rely heavily on Clojure's sequence processing functions and I would hate to clutter up every call to every one of them or to have to reimplement my own non-lazy versions of everything. Is there a more elegant way? Maybe somebody has already made non-lazy clones of all of the core functions that normally produce lazy sequences? And if so, maybe there's some clean way to cause my code to use the non-lazy versions unless specifically directed to use the lazy versions? I realize that this is fighting with a core feature of Clojure, and that the idea will probably rub lots of folks the wrong way. But I've now been down this road a couple of times and I'm beginning to think that I'd spend less time tracking down mysterious bugs if I could avoid laziness more easily. In case anybody is motivated to look into the specific bug that I was just dealing with, I've put the project at http://hampshire.edu/lspector/temp/world2D.zip. If you do lein run in its present state it will run fine, making little balls bounce around. (This is a very early version of some code that I plan to use both for teaching an AI class and for some ALife experiments...) But if you edit src/world2D/vec2D.clj, commenting out the definitions of *v, +v, and -v, and uncommenting the alternatives (which are the same except that they lack the calls to vec) then after running for a few seconds or a minute or so you'll get the stack overflow. Thanks for any suggestions, -Lee [1] The stack traces I was getting provide no information about where/what the offending lazy sequences are, since they don't show locals etc. I see no references to any of my own code, just a repeating sequence that starts: StackOverflowError clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) [etc] -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to
RE: non-lazy clojure?
Funny, I posted an article of my Clojure gotcha's today (http://www.russet.org.uk/blog/2991), and this is one of them. I've also had very nasty bugs, in addition to the general hassle of wrapping a Java API which works through side-effects. I created a few functions covering the most common ones (domap, dofor and so on); having a comprehensive list somewhere in it's own namespace would be quite useful. Phil From: clojure@googlegroups.com [clojure@googlegroups.com] on behalf of Lee Spector [lspec...@hampshire.edu] Sent: 02 June 2014 21:32 To: clojure@googlegroups.com Subject: non-lazy clojure? I've generally liked Clojure's pervasive laziness. It's cute and it sometimes permits lovely, elegant approaches to particular programming problems. And although I've long known that it can get you into trouble in a few unusual cases -- I think I recall seeing a nice blog post on issues related to binding and laziness by Chas Emerick several years ago -- it has generally seemed safe enough for most purposes. I figured that while it may not always provide benefits it should at least be mostly harmless. However, after spending a couple of days tracking down a particularly confusing bug, and finding once again (this has happened to me at least a few times) that the bug was due to laziness that I never really wanted in the first place, I'm reconsidering. I don't have a pared-down version of the bug that I was just dealing with (see below for a pointer to the non-pared-down version), but it goes away when I wring out all of the laziness. In this particular case I do so by calling vec or doall on the results of all of my calls to map and other functions function that produce lazy sequences (none of which I actually want to be lazy in this application). I was getting StackOverflowErrors and I have a half-baked half-theory that it was due to some bad interaction between laziness and garbage collection, with the JVM not realizing that it could reclaim lazy things that weren't yet fully evaluated. That doesn't make complete sense to me, and maybe that kind of bad interaction is not even possible -- I don't know. But I do know that all is well when I wring out all of the laziness. One possible lesson from this experience (which was particularly frustrating, BTW, because the stack backtraces were particularly uninformative [1]) is that I should just be careful to surround any call to any function that produces a lazy sequence with something like vec that ensures that the laziness will be eliminated as soon as it is created -- except in those relatively rare cases where I really want the laziness. But this would be a pain, and ugly. I rely heavily on Clojure's sequence processing functions and I would hate to clutter up every call to every one of them or to have to reimplement my own non-lazy versions of everything. Is there a more elegant way? Maybe somebody has already made non-lazy clones of all of the core functions that normally produce lazy sequences? And if so, maybe there's some clean way to cause my code to use the non-lazy versions unless specifically directed to use the lazy versions? I realize that this is fighting with a core feature of Clojure, and that the idea will probably rub lots of folks the wrong way. But I've now been down this road a couple of times and I'm beginning to think that I'd spend less time tracking down mysterious bugs if I could avoid laziness more easily. In case anybody is motivated to look into the specific bug that I was just dealing with, I've put the project at http://hampshire.edu/lspector/temp/world2D.zip. If you do lein run in its present state it will run fine, making little balls bounce around. (This is a very early version of some code that I plan to use both for teaching an AI class and for some ALife experiments...) But if you edit src/world2D/vec2D.clj, commenting out the definitions of *v, +v, and -v, and uncommenting the alternatives (which are the same except that they lack the calls to vec) then after running for a few seconds or a minute or so you'll get the stack overflow. Thanks for any suggestions, -Lee [1] The stack traces I was getting provide no information about where/what the offending lazy sequences are, since they don't show locals etc. I see no references to any of my own code, just a repeating sequence that starts: StackOverflowError clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) [etc] -- You received this message because you are subscribed
Re: non-lazy clojure?
Reducers [1] provide eager variants of some core seq functions (map, filter, etc.). Note that they do not cache the result, so they recompute it every time you use their result. [1] http://clojure.org/reducers On Mon, Jun 2, 2014 at 10:32 PM, Lee Spector lspec...@hampshire.edu wrote: I've generally liked Clojure's pervasive laziness. It's cute and it sometimes permits lovely, elegant approaches to particular programming problems. And although I've long known that it can get you into trouble in a few unusual cases -- I think I recall seeing a nice blog post on issues related to binding and laziness by Chas Emerick several years ago -- it has generally seemed safe enough for most purposes. I figured that while it may not always provide benefits it should at least be mostly harmless. However, after spending a couple of days tracking down a particularly confusing bug, and finding once again (this has happened to me at least a few times) that the bug was due to laziness that I never really wanted in the first place, I'm reconsidering. I don't have a pared-down version of the bug that I was just dealing with (see below for a pointer to the non-pared-down version), but it goes away when I wring out all of the laziness. In this particular case I do so by calling vec or doall on the results of all of my calls to map and other functions function that produce lazy sequences (none of which I actually want to be lazy in this application). I was getting StackOverflowErrors and I have a half-baked half-theory that it was due to some bad interaction between laziness and garbage collection, with the JVM not realizing that it could reclaim lazy things that weren't yet fully evaluated. That doesn't make complete sense to me, and maybe that kind of bad interaction is not even possible -- I don't know. But I do know that all is well when I wring out all of the laziness. One possible lesson from this experience (which was particularly frustrating, BTW, because the stack backtraces were particularly uninformative [1]) is that I should just be careful to surround any call to any function that produces a lazy sequence with something like vec that ensures that the laziness will be eliminated as soon as it is created -- except in those relatively rare cases where I really want the laziness. But this would be a pain, and ugly. I rely heavily on Clojure's sequence processing functions and I would hate to clutter up every call to every one of them or to have to reimplement my own non-lazy versions of everything. Is there a more elegant way? Maybe somebody has already made non-lazy clones of all of the core functions that normally produce lazy sequences? And if so, maybe there's some clean way to cause my code to use the non-lazy versions unless specifically directed to use the lazy versions? I realize that this is fighting with a core feature of Clojure, and that the idea will probably rub lots of folks the wrong way. But I've now been down this road a couple of times and I'm beginning to think that I'd spend less time tracking down mysterious bugs if I could avoid laziness more easily. In case anybody is motivated to look into the specific bug that I was just dealing with, I've put the project at http://hampshire.edu/lspector/temp/world2D.zip. If you do lein run in its present state it will run fine, making little balls bounce around. (This is a very early version of some code that I plan to use both for teaching an AI class and for some ALife experiments...) But if you edit src/world2D/vec2D.clj, commenting out the definitions of *v, +v, and -v, and uncommenting the alternatives (which are the same except that they lack the calls to vec) then after running for a few seconds or a minute or so you'll get the stack overflow. Thanks for any suggestions, -Lee [1] The stack traces I was getting provide no information about where/what the offending lazy sequences are, since they don't show locals etc. I see no references to any of my own code, just a repeating sequence that starts: StackOverflowError clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) clojure.core/seq (core.clj:133) clojure.core/map/fn--4211 (core.clj:2490) clojure.lang.LazySeq.sval (LazySeq.java:42) clojure.lang.LazySeq.seq (LazySeq.java:60) clojure.lang.RT.seq (RT.java:484) [etc] -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at
Re: non-lazy clojure?
On Jun 2, 2014, at 4:51 PM, Softaddicts lprefonta...@softaddicts.ca wrote: mapv a bit shorter :) Luc P. Thanks Luc. I have indeed migrated to mapv for many cases and I could have for the specific final fix for the example I posted. But there are lots of other fixes that I made earlier in the same project, some of which may also have been necessary. I also used filterv in some cases... but other places it was messier, e.g. with for. -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Jun 2, 2014, at 4:52 PM, Phillip Lord phillip.l...@newcastle.ac.uk wrote: Funny, I posted an article of my Clojure gotcha's today (http://www.russet.org.uk/blog/2991), and this is one of them. I've also had very nasty bugs, in addition to the general hassle of wrapping a Java API which works through side-effects. I created a few functions covering the most common ones (domap, dofor and so on); having a comprehensive list somewhere in it's own namespace would be quite useful. Phil It'd be nice to have a solid library of non-lazy alternatives to the core sequence functions. FWIW while I've often dealt with this issue by using vectors, either with an explicit vec or with mapv or filterv, it's often not necessary or helpful for me that my sequences are actually vectors. That is, they could often be lists or some other kind of sequence instead. It's hard for me to know for sure, but I think that in many cases a doall solves the same problem that a vec would. This seems to be the case, for example, for the code/fix that I posted. -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
On Jun 2, 2014, at 4:52 PM, Jozef Wagner jozef.wag...@gmail.com wrote: Reducers [1] provide eager variants of some core seq functions (map, filter, etc.). Note that they do not cache the result, so they recompute it every time you use their result. [1] http://clojure.org/reducers Thanks Josef. I haven't yet really looked into reducers, but it seems on first glance like it may be overkill for just dealing with this issue -- using fork/join and getting unneeded parallelism for routine sequence processing, just to ensure that one doesn't have laziness-related bugs. The recomputation also sounds potentially problematic. But I do want to look into reducers for other purposes, so thanks for the pointer! -Lee -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
Hey Lee, I would second Jozef's suggestion that you look into using the reducers library when you need non-lazy sequence operations. Although a major motivation of Rich's work was clearly to enable easy parallel folding via fork/join, the fold function is only one of many in this library. Think instead that the main (philosophical) purpose of reducers is to decomplect the reducing operation from the data representation it is acting on. And of course, since reduce can be used to implement (virtually) any non-lazy sequence operation, it stands to reason that reducers should be fully capable of providing new implementations of many of these functions on top of reduce (which it does). Importantly, whenever you will be chaining sequence operations together, reducers should be more efficient than both the lazy sequence functions (e.g., map, filter) and the eager vector-returning functions (e.g., mapv, filterv). This is because a chain of reducing functions generate no intermediate representations. obligatory contrived example For example, let's say I wanted to sum the squares of all the even numbers in a sequence called samples. Using lazy functions: (reduce + (map #(* % %) (filter even? samples))) Using non-lazy functions (reduce + (mapv #(* % %) (filterv even? samples))) Using reducers (aliased as r): (reduce + (r/map #(* % %) (r/filter even? samples))) /obligatory contrived example If you need to collect the results of a sequence operation in a data structure rather than reducing them to an atomic value, simply use into rather than reduce (since into uses reduce under the hood). So to collect the squares of all the even numbers in the samples sequence, just do this: (into [] (r/map #(* % %) (r/filter even? samples))) As just one sample point, when I updated a statistical fire analysis algorithm that I wrote from using the lazy sequence functions to using the reducers library, I experience a full order of magnitude speedup. This sped up my runtime from ~6 hours to around 20 minutes. So please do yourself a favor and give this library a close look. It has made worlds of difference for some of my work. Good luck, ~Gary -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: non-lazy clojure?
Gary: That's compelling indeed, and I will look into it more! Thanks, -Lee PS would a call to vec do the same thing as into [] here? On Jun 2, 2014, at 7:14 PM, Gary Johnson gwjoh...@uvm.edu wrote: Hey Lee, I would second Jozef's suggestion that you look into using the reducers library when you need non-lazy sequence operations. Although a major motivation of Rich's work was clearly to enable easy parallel folding via fork/join, the fold function is only one of many in this library. Think instead that the main (philosophical) purpose of reducers is to decomplect the reducing operation from the data representation it is acting on. And of course, since reduce can be used to implement (virtually) any non-lazy sequence operation, it stands to reason that reducers should be fully capable of providing new implementations of many of these functions on top of reduce (which it does). Importantly, whenever you will be chaining sequence operations together, reducers should be more efficient than both the lazy sequence functions (e.g., map, filter) and the eager vector-returning functions (e.g., mapv, filterv). This is because a chain of reducing functions generate no intermediate representations. obligatory contrived example For example, let's say I wanted to sum the squares of all the even numbers in a sequence called samples. Using lazy functions: (reduce + (map #(* % %) (filter even? samples))) Using non-lazy functions (reduce + (mapv #(* % %) (filterv even? samples))) Using reducers (aliased as r): (reduce + (r/map #(* % %) (r/filter even? samples))) /obligatory contrived example If you need to collect the results of a sequence operation in a data structure rather than reducing them to an atomic value, simply use into rather than reduce (since into uses reduce under the hood). So to collect the squares of all the even numbers in the samples sequence, just do this: (into [] (r/map #(* % %) (r/filter even? samples))) As just one sample point, when I updated a statistical fire analysis algorithm that I wrote from using the lazy sequence functions to using the reducers library, I experience a full order of magnitude speedup. This sped up my runtime from ~6 hours to around 20 minutes. So please do yourself a favor and give this library a close look. It has made worlds of difference for some of my work. Good luck, ~Gary -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
