Re: Accidentally Retaining Head?
By the way, you seem to misunderstand some of the workings of GC. In the kind of generational scheme used in virtually all modern algorithms, a collection of the nursery (where allocations are first hatched) will only be forced when the memory assigned to the nursery is exhausted--not the memory of any higher generations. -Per On Tue, Mar 23, 2010 at 9:49 PM, aria42 wrote: > Whoops duh. That was silly, far to early on the west coast. > > On Mar 23, 7:46 am, Per Vognsen wrote: >> It doesn't seem very accidental: the namespace binding for 'trees' is >> retaining the head. You probably want to wrap it in a function: >> >> (defn trees [] >> ...) >> >> -Per >> >> On Tue, Mar 23, 2010 at 9:40 PM, aria42 wrote: >> > Hi, >> > I was experimenting with some code and I had an largish sequence I >> > did a doseq over. The memory hit the ceiling, which you expect since >> > even though the head isn't retained GC doesn't happen until you hit >> > your memory limit (is there a way to change that). Once it hit the >> > memory limit, 1.2 gigs in this case, the doseq slows to a halt. >> >> > On the other hand if rather than make a long lazy seq, I do it >> > implicitly in the doseq itself (see FAST snippet below), the >> > performance is great. Is there anyway to get good performance using a >> > single lazy seq? Relevant snippets below. >> >> > Thanks, Aria >> >> > (defn lines >> > "get lines from gz file" >> > [#^String path] >> > (-> path >> > java.io.FileInputStream. >> > java.util.zip.GZIPInputStream. >> > java.io.InputStreamReader. >> > java.io.BufferedReader. >> > line-seq)) >> >> > ; the tree-from-str does some processing, but doesn't have state >> > (def trees (for [l (lines "/usr/local/corpora//NANC/003.gz") >> > :when (not (empty? l)) >> > :let [[t _] (tree/tree-from-str l)]] >> > t)) >> >> > ; SLOW: hits memory limit and becomes slow b/c of constant >> > ; GC hits >> > (doseq [t trees] >> > (println (str t))) >> >> > ; FAST: low memory >> > (doseq [l (lines "/usr/local/corpora//NANC/003.gz") >> > :when (not (empty? l)) >> > :let [[t _] (tree/tree-from-str l)]] >> > (println (str t))) >> >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Clojure" group. >> > To post to this group, send email to clojure@googlegroups.com >> > Note that posts from new members are moderated - please be patient with >> > your first post. >> > To unsubscribe from this group, send email to >> > clojure+unsubscr...@googlegroups.com >> > For more options, visit this group at >> >http://groups.google.com/group/clojure?hl=en >> >> > To unsubscribe from this group, send email to >> > clojure+unsubscribegooglegroups.com or reply to this email with the words >> > "REMOVE ME" as the subject. >> >> > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > > To unsubscribe from this group, send email to > clojure+unsubscribegooglegroups.com or reply to this email with the words > "REMOVE ME" as the subject. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
Re: Accidentally Retaining Head?
Whoops duh. That was silly, far to early on the west coast. On Mar 23, 7:46 am, Per Vognsen wrote: > It doesn't seem very accidental: the namespace binding for 'trees' is > retaining the head. You probably want to wrap it in a function: > > (defn trees [] > ...) > > -Per > > On Tue, Mar 23, 2010 at 9:40 PM, aria42 wrote: > > Hi, > > I was experimenting with some code and I had an largish sequence I > > did a doseq over. The memory hit the ceiling, which you expect since > > even though the head isn't retained GC doesn't happen until you hit > > your memory limit (is there a way to change that). Once it hit the > > memory limit, 1.2 gigs in this case, the doseq slows to a halt. > > > On the other hand if rather than make a long lazy seq, I do it > > implicitly in the doseq itself (see FAST snippet below), the > > performance is great. Is there anyway to get good performance using a > > single lazy seq? Relevant snippets below. > > > Thanks, Aria > > > (defn lines > > "get lines from gz file" > > [#^String path] > > (-> path > > java.io.FileInputStream. > > java.util.zip.GZIPInputStream. > > java.io.InputStreamReader. > > java.io.BufferedReader. > > line-seq)) > > > ; the tree-from-str does some processing, but doesn't have state > > (def trees (for [l (lines "/usr/local/corpora//NANC/003.gz") > > :when (not (empty? l)) > > :let [[t _] (tree/tree-from-str l)]] > > t)) > > > ; SLOW: hits memory limit and becomes slow b/c of constant > > ; GC hits > > (doseq [t trees] > > (println (str t))) > > > ; FAST: low memory > > (doseq [l (lines "/usr/local/corpora//NANC/003.gz") > > :when (not (empty? l)) > > :let [[t _] (tree/tree-from-str l)]] > > (println (str t))) > > > -- > > You received this message because you are subscribed to the Google > > Groups "Clojure" group. > > To post to this group, send email to clojure@googlegroups.com > > Note that posts from new members are moderated - please be patient with > > your first post. > > To unsubscribe from this group, send email to > > clojure+unsubscr...@googlegroups.com > > For more options, visit this group at > >http://groups.google.com/group/clojure?hl=en > > > To unsubscribe from this group, send email to > > clojure+unsubscribegooglegroups.com or reply to this email with the words > > "REMOVE ME" as the subject. > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
Re: Accidentally Retaining Head?
It doesn't seem very accidental: the namespace binding for 'trees' is retaining the head. You probably want to wrap it in a function: (defn trees [] ...) -Per On Tue, Mar 23, 2010 at 9:40 PM, aria42 wrote: > Hi, > I was experimenting with some code and I had an largish sequence I > did a doseq over. The memory hit the ceiling, which you expect since > even though the head isn't retained GC doesn't happen until you hit > your memory limit (is there a way to change that). Once it hit the > memory limit, 1.2 gigs in this case, the doseq slows to a halt. > > On the other hand if rather than make a long lazy seq, I do it > implicitly in the doseq itself (see FAST snippet below), the > performance is great. Is there anyway to get good performance using a > single lazy seq? Relevant snippets below. > > Thanks, Aria > > (defn lines > "get lines from gz file" > [#^String path] > (-> path > java.io.FileInputStream. > java.util.zip.GZIPInputStream. > java.io.InputStreamReader. > java.io.BufferedReader. > line-seq)) > > ; the tree-from-str does some processing, but doesn't have state > (def trees (for [l (lines "/usr/local/corpora//NANC/003.gz") > :when (not (empty? l)) > :let [[t _] (tree/tree-from-str l)]] > t)) > > ; SLOW: hits memory limit and becomes slow b/c of constant > ; GC hits > (doseq [t trees] > (println (str t))) > > ; FAST: low memory > (doseq [l (lines "/usr/local/corpora//NANC/003.gz") > :when (not (empty? l)) > :let [[t _] (tree/tree-from-str l)]] > (println (str t))) > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > > To unsubscribe from this group, send email to > clojure+unsubscribegooglegroups.com or reply to this email with the words > "REMOVE ME" as the subject. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
Accidentally Retaining Head?
Hi, I was experimenting with some code and I had an largish sequence I did a doseq over. The memory hit the ceiling, which you expect since even though the head isn't retained GC doesn't happen until you hit your memory limit (is there a way to change that). Once it hit the memory limit, 1.2 gigs in this case, the doseq slows to a halt. On the other hand if rather than make a long lazy seq, I do it implicitly in the doseq itself (see FAST snippet below), the performance is great. Is there anyway to get good performance using a single lazy seq? Relevant snippets below. Thanks, Aria (defn lines "get lines from gz file" [#^String path] (-> path java.io.FileInputStream. java.util.zip.GZIPInputStream. java.io.InputStreamReader. java.io.BufferedReader. line-seq)) ; the tree-from-str does some processing, but doesn't have state (def trees (for [l (lines "/usr/local/corpora//NANC/003.gz") :when (not (empty? l)) :let [[t _] (tree/tree-from-str l)]] t)) ; SLOW: hits memory limit and becomes slow b/c of constant ; GC hits (doseq [t trees] (println (str t))) ; FAST: low memory (doseq [l (lines "/usr/local/corpora//NANC/003.gz") :when (not (empty? l)) :let [[t _] (tree/tree-from-str l)]] (println (str t))) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
