Re: apply a function to every item in a sequence without realizing the sequence
On 2 May 2012 14:44, Baishampayan Ghose b.gh...@gmail.com wrote: You can't use `map` because `map` will return a sequence of the same size and that can blow your heap. Isn't `map` lazy too? Regards, Stuart -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
I don't think so. On Wed, May 2, 2012 at 1:22 AM, Stuart Campbell stu...@harto.org wrote: On 2 May 2012 14:44, Baishampayan Ghose b.gh...@gmail.com wrote: You can't use `map` because `map` will return a sequence of the same size and that can blow your heap. Isn't `map` lazy too? Regards, Stuart -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
Map IS lazy but it still returns the entire realized sequence, as expected. On May 2, 2012 8:31 AM, Sean Neilan s...@seanneilan.com wrote: I don't think so. On Wed, May 2, 2012 at 1:22 AM, Stuart Campbell stu...@harto.org wrote: On 2 May 2012 14:44, Baishampayan Ghose b.gh...@gmail.com wrote: You can't use `map` because `map` will return a sequence of the same size and that can blow your heap. Isn't `map` lazy too? Regards, Stuart -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
On Wed, May 2, 2012 at 12:01 PM, Sean Neilan s...@seanneilan.com wrote: I don't think so. Of course it is. The problem is not in laziness, but in holding on to the head. Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
My example included a use of `map`. It is lazy and will work but you have to be sure that you aren't using it in a way that would hold onto the head of the sequence. When experimenting in a repl it might not seem that it is lazy since the repl will attempt to print the result of calling map when that is the only expression. AJ On Wed, May 2, 2012 at 4:06 AM, Baishampayan Ghose b.gh...@gmail.com wrote: On Wed, May 2, 2012 at 12:01 PM, Sean Neilan s...@seanneilan.com wrote: I don't think so. Of course it is. The problem is not in laziness, but in holding on to the head. Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
Note also that in the REPL the last three values returned are kept available under *1, *2 and *3. M. On 2 May 2012 15:40, Allen Johnson akjohnso...@gmail.com wrote: My example included a use of `map`. It is lazy and will work but you have to be sure that you aren't using it in a way that would hold onto the head of the sequence. When experimenting in a repl it might not seem that it is lazy since the repl will attempt to print the result of calling map when that is the only expression. AJ On Wed, May 2, 2012 at 4:06 AM, Baishampayan Ghose b.gh...@gmail.com wrote: On Wed, May 2, 2012 at 12:01 PM, Sean Neilan s...@seanneilan.com wrote: I don't think so. Of course it is. The problem is not in laziness, but in holding on to the head. Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
apply a function to every item in a sequence without realizing the sequence
Hi, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problems 1. I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. 2. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. 3. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. 4. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) 5. I can't use refs because even though they provide state, they can't save the state of the sequence without realizing the sequence. My question *Should I try the new stream library?* * * Thank you for your time. -Sean -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
I forgot to mention: (nth (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) 20) works great because nth doesn't realize the sequence! For now, I'll look at nth's source code to see how it iterates. On Tue, May 1, 2012 at 11:24 PM, Sean Neilan sneil...@gmail.com wrote: Hi, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problems 1. I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. 2. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. 3. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. 4. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) 5. I can't use refs because even though they provide state, they can't save the state of the sequence without realizing the sequence. My question *Should I try the new stream library?* * * Thank you for your time. -Sean -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
On Wednesday, May 2, 2012 7:24:04 AM UTC+3, Sean Neilan wrote: Hi, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problems 1. I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. 2. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. Try doseq instead: (doseq [x (files)] (println x)) 1. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. 2. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) 3. I can't use refs because even though they provide state, they can't save the state of the sequence without realizing the sequence. My question *Should I try the new stream library?* * * Thank you for your time. -Sean -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
Sean, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problem is not in realizing the sequence, but in holding on to the head. Naming the sequence using `def` or `'let` will indeed hold on to the head, but I don't see why loop/recur can't solve your problem. Something like this should work - (loop [fs (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/))] (println (first fs)) (recur (rest fs))) Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
Do you have example code that is failing? You should be able to use some of the items you listed as problems. Try something like this: (- (file-seq (io/file /some/dir)) (map println) (dorun)) AJ On Wed, May 2, 2012 at 12:26 AM, Sean Neilan s...@seanneilan.com wrote: I forgot to mention: (nth (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) 20) works great because nth doesn't realize the sequence! For now, I'll look at nth's source code to see how it iterates. On Tue, May 1, 2012 at 11:24 PM, Sean Neilan sneil...@gmail.com wrote: Hi, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problems I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) I can't use refs because even though they provide state, they can't save the state of the sequence without realizing the sequence. My question Should I try the new stream library? Thank you for your time. -Sean -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
You can also use doseq afaik, altough every element must realized at least once, you just have to make sure you don't hold onto the head of the sequence as you proceed. It is not immediately apparent to me why that doesn't happen with your loop-recur solution On May 2, 2012 6:27 AM, Sean Neilan s...@seanneilan.com wrote: I forgot to mention: (nth (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) 20) works great because nth doesn't realize the sequence! For now, I'll look at nth's source code to see how it iterates. On Tue, May 1, 2012 at 11:24 PM, Sean Neilan sneil...@gmail.com wrote: Hi, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problems 1. I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. 2. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. 3. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. 4. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) 5. I can't use refs because even though they provide state, they can't save the state of the sequence without realizing the sequence. My question *Should I try the new stream library?* * * Thank you for your time. -Sean -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
Wow lot of active people in the early morning, all typing faster than me on my phone... :-) On May 2, 2012 6:36 AM, László Török ltoro...@gmail.com wrote: You can also use doseq afaik, altough every element must realized at least once, you just have to make sure you don't hold onto the head of the sequence as you proceed. It is not immediately apparent to me why that doesn't happen with your loop-recur solution On May 2, 2012 6:27 AM, Sean Neilan s...@seanneilan.com wrote: I forgot to mention: (nth (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) 20) works great because nth doesn't realize the sequence! For now, I'll look at nth's source code to see how it iterates. On Tue, May 1, 2012 at 11:24 PM, Sean Neilan sneil...@gmail.com wrote: Hi, I'm sure this has been discussed to death but I can't figure it out. I've got a file-seq sequence from (file-seq (java.io.File. /DirectoryWithMillionsOfFiles/)) that will cause an out of memory error if realized. I want to call a function such as println on every element in the sequence. I understand that naming the sequence will cause it to be realized. The problems 1. I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. 2. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. 3. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. 4. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) 5. I can't use refs because even though they provide state, they can't save the state of the sequence without realizing the sequence. My question *Should I try the new stream library?* * * Thank you for your time. -Sean -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
The problems I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. You can't use `map` because `map` will return a sequence of the same size and that can blow your heap. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) Going by the examples here, it seems you are holding on the head by binding the while file-seq to a var called `files`. That is the problem. Take a look at my loop-recur example. You'll see that I create a temporary binding that gets reduced in size every time I recur. So in that example, I am not holding on to the head anywhere and the files that I have printed will get GCd quickly without blowing up. To repeat, the problem is in holding on to the head (ie. the whole sequence) and if you process every item and throw away items from the seq there shouldn't be any problem. Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: apply a function to every item in a sequence without realizing the sequence
Wow!! Lots of responses! The problem was that I was running the code inside of a jark repl. The jark repl apparently causes head holding of some sort. All the examples you guys provided (@Allen, @Jonas, @Baishampayan) work in a regular project from lein run. They also work from the leiningen repl. Thank you guys very much! I got this working and my problem was the jark repl. On Tue, May 1, 2012 at 11:44 PM, Baishampayan Ghose b.gh...@gmail.comwrote: The problems I can't use map as in (map println (file-seq (java.io.File. /DirectoryWithMillionsOfFiles))). Map realizes the sequence. You can't use `map` because `map` will return a sequence of the same size and that can blow your heap. Right. I can't use for as in (for [x (files)] (println x)). For realizes the sequence. I can't use dorun because even though dorun doesn't realize the sequence, it can't execute a function on every element. I can't use loop recur because it also realizes the sequence: (loop [a (files) b (first a)] (println b) (recur (rest a) (first a))) Going by the examples here, it seems you are holding on the head by binding the while file-seq to a var called `files`. That is the problem. Take a look at my loop-recur example. You'll see that I create a temporary binding that gets reduced in size every time I recur. So in that example, I am not holding on to the head anywhere and the files that I have printed will get GCd quickly without blowing up. To repeat, the problem is in holding on to the head (ie. the whole sequence) and if you process every item and throw away items from the seq there shouldn't be any problem. Ok. And rest can do that because rest returns a lazy sequence. Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
