Re: java.lang.OutOfMemoryError: PermGen space
I received a message from Moritz that everyone is on openJDK 1.7. I was curious and looked into this a little bit. I found one older thread from this forum [1] that mentioned PermGen and OutOfMemory. The following are some notes that will hopefully help here. Regarding the exception: Caused by: java.lang.OutOfMemoryError: PermGen space, compiling:(cljs/core/async/impl/ioc_macros.clj:348:1) that line of ioc_macros.clj uses a defrecord. That'd make sense that that would trigger the OutOfMemory as that would generate classes to go into PermGen. Now, the problem in analyzing this is that it might be one part of the system is holding references to older classes and surviving reloads, but other parts are loading/unloading just fine. If the one part is creeping up in size, you might get the PermGen OOME because something triggered it, but what triggered it might not be the root cause. Also, it might be something in one of the libraries you're using, but it could also depend on what you have evaluated in your REPL name space. It might also happen if a thread is still alive through a reload, as mentioned in [2]. One other question: Are you using core.async on the server-side of your application? I wonder if there is a possible situation here where parked IOC threads from a previous app load are being kept around. I don't know core.async well enough to know if/when parked threads go out of scope, but I thought this could be a possibility. [1] - https://groups.google.com/forum/#!topic/clojure/LGZlC5BX92A [2] - http://java.dzone.com/articles/what-permgen-leak On Wednesday, December 17, 2014 7:05:36 PM UTC-5, Steven Yi wrote: Out of curiosity, are you colleagues perhaps using Java 8 JDK's? I saw your message mentions JDK 7 and PermGen was removed in Java 8[1]. [1] - http://www.infoq.com/articles/Java-PERMGEN-Removed On Wednesday, December 17, 2014 12:41:29 PM UTC-5, Moritz Ulrich wrote: Hello, I'm getting the following error while working on a Clojure(Script) REPL on a middle-sized project after some time. None of my colleagues seem to be able to reproduce it. I'm not able to reproduce it on other projects either. Caused by: java.lang.OutOfMemoryError: PermGen space, compiling:(cljs/core/async/impl/ioc_macros.clj:348:1) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6651) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6100) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5217) at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3846) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6642) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.eval(Compiler.java:6700) at clojure.lang.Compiler.load(Compiler.java:7130) at clojure.lang.RT.loadResourceScript(RT.java:370) at clojure.lang.RT.loadResourceScript(RT.java:361) at clojure.lang.RT.load(RT.java:440) at clojure.lang.RT.load(RT.java:411) at clojure.core$load$fn__5066.invoke(core.clj:5641) at clojure.core$load.doInvoke(core.clj:5640) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invoke(core.clj:5446) at clojure.core$load_lib$fn__5015.invoke(core.clj:5486) at clojure.core$load_lib.doInvoke(core.clj:5485) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invoke(core.clj:626) at clojure.core$load_libs.doInvoke(core.clj:5524) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invoke(core.clj:626) at clojure.core$require.doInvoke(core.clj:5607) at clojure.lang.RestFn.invoke(RestFn.java:408) at cljs.analyzer$eval21264$fn__21266.invoke(analyzer.clj:1190) at clojure.lang.MultiFn.invoke(MultiFn.java:249) at cljs.analyzer$analyze_seq.invoke(analyzer.clj:1490) I'm using lein-repl in combination with weasel to work on the server- and client-side (cljs) of this project. The server-side uses the component-design by stuartsierra, though the permgen issues don't seem to correlate to calls to (reload). $ lein version Leiningen 2.5.0 on Java 1.7.0-u65 OpenJDK 64-Bit Server VM Some of the dependencies (I suspect some of them are responsible): [org.clojure/clojure 1.6.0] [org.clojure/core.async 0.1.346.0-17112a-alpha] [com.stuartsierra/component 0.2.2]
Re: java.lang.OutOfMemoryError: PermGen space
Just to follow up, I did some tests with a simple project that had core.async, component, and tools.namespace. I ran a REPL and in it had a doseq loop calling (refresh) over and over. In another terminal, I had a watch -n 1 touch src/test/core.clj repeating to update the file. The core file created some core.async channel and defined a component using defrecord. I then opened JVisualVM and watched the PermGen space and Class graph over time. At least with this simple setup, the PermGen space was freeing nicely and the number of loaded classes would go down as well upon the frees of PermGen space. I had Oracle JDK 1.7 on this test VM, so I don't know if that's a factor. Either way, I suspect it's an interaction of parts of your system with something being cached along the way that might be causing the OOME. On Thursday, December 18, 2014 1:07:54 PM UTC-5, Steven Yi wrote: I received a message from Moritz that everyone is on openJDK 1.7. I was curious and looked into this a little bit. I found one older thread from this forum [1] that mentioned PermGen and OutOfMemory. The following are some notes that will hopefully help here. Regarding the exception: Caused by: java.lang.OutOfMemoryError: PermGen space, compiling:(cljs/core/async/impl/ioc_macros.clj:348:1) that line of ioc_macros.clj uses a defrecord. That'd make sense that that would trigger the OutOfMemory as that would generate classes to go into PermGen. Now, the problem in analyzing this is that it might be one part of the system is holding references to older classes and surviving reloads, but other parts are loading/unloading just fine. If the one part is creeping up in size, you might get the PermGen OOME because something triggered it, but what triggered it might not be the root cause. Also, it might be something in one of the libraries you're using, but it could also depend on what you have evaluated in your REPL name space. It might also happen if a thread is still alive through a reload, as mentioned in [2]. One other question: Are you using core.async on the server-side of your application? I wonder if there is a possible situation here where parked IOC threads from a previous app load are being kept around. I don't know core.async well enough to know if/when parked threads go out of scope, but I thought this could be a possibility. [1] - https://groups.google.com/forum/#!topic/clojure/LGZlC5BX92A [2] - http://java.dzone.com/articles/what-permgen-leak On Wednesday, December 17, 2014 7:05:36 PM UTC-5, Steven Yi wrote: Out of curiosity, are you colleagues perhaps using Java 8 JDK's? I saw your message mentions JDK 7 and PermGen was removed in Java 8[1]. [1] - http://www.infoq.com/articles/Java-PERMGEN-Removed On Wednesday, December 17, 2014 12:41:29 PM UTC-5, Moritz Ulrich wrote: Hello, I'm getting the following error while working on a Clojure(Script) REPL on a middle-sized project after some time. None of my colleagues seem to be able to reproduce it. I'm not able to reproduce it on other projects either. Caused by: java.lang.OutOfMemoryError: PermGen space, compiling:(cljs/core/async/impl/ioc_macros.clj:348:1) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6651) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6100) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5217) at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3846) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6642) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.eval(Compiler.java:6700) at clojure.lang.Compiler.load(Compiler.java:7130) at clojure.lang.RT.loadResourceScript(RT.java:370) at clojure.lang.RT.loadResourceScript(RT.java:361) at clojure.lang.RT.load(RT.java:440) at clojure.lang.RT.load(RT.java:411) at clojure.core$load$fn__5066.invoke(core.clj:5641) at clojure.core$load.doInvoke(core.clj:5640) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invoke(core.clj:5446) at clojure.core$load_lib$fn__5015.invoke(core.clj:5486) at clojure.core$load_lib.doInvoke(core.clj:5485) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invoke(core.clj:626) at clojure.core$load_libs.doInvoke(core.clj:5524) at
Re: java.lang.OutOfMemoryError: PermGen space
You might try experimenting with JVM options to increase PermGen: https://github.com/boot-clj/boot/wiki/JVM-Options#permgen-errors On Wednesday, December 17, 2014 12:41:29 PM UTC-5, Moritz Ulrich wrote: Hello, I'm getting the following error while working on a Clojure(Script) REPL on a middle-sized project after some time. None of my colleagues seem to be able to reproduce it. I'm not able to reproduce it on other projects either. Caused by: java.lang.OutOfMemoryError: PermGen space, compiling:(cljs/core/async/impl/ioc_macros.clj:348:1) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6651) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6100) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5217) at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3846) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6642) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.eval(Compiler.java:6700) at clojure.lang.Compiler.load(Compiler.java:7130) at clojure.lang.RT.loadResourceScript(RT.java:370) at clojure.lang.RT.loadResourceScript(RT.java:361) at clojure.lang.RT.load(RT.java:440) at clojure.lang.RT.load(RT.java:411) at clojure.core$load$fn__5066.invoke(core.clj:5641) at clojure.core$load.doInvoke(core.clj:5640) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invoke(core.clj:5446) at clojure.core$load_lib$fn__5015.invoke(core.clj:5486) at clojure.core$load_lib.doInvoke(core.clj:5485) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invoke(core.clj:626) at clojure.core$load_libs.doInvoke(core.clj:5524) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invoke(core.clj:626) at clojure.core$require.doInvoke(core.clj:5607) at clojure.lang.RestFn.invoke(RestFn.java:408) at cljs.analyzer$eval21264$fn__21266.invoke(analyzer.clj:1190) at clojure.lang.MultiFn.invoke(MultiFn.java:249) at cljs.analyzer$analyze_seq.invoke(analyzer.clj:1490) I'm using lein-repl in combination with weasel to work on the server- and client-side (cljs) of this project. The server-side uses the component-design by stuartsierra, though the permgen issues don't seem to correlate to calls to (reload). $ lein version Leiningen 2.5.0 on Java 1.7.0-u65 OpenJDK 64-Bit Server VM Some of the dependencies (I suspect some of them are responsible): [org.clojure/clojure 1.6.0] [org.clojure/core.async 0.1.346.0-17112a-alpha] [com.stuartsierra/component 0.2.2] [org.clojure/tools.namespace 0.2.7] [lein-cljsbuild 1.0.3] [cider/cider-nrepl 0.8.1] -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: java.lang.OutOfMemoryError: PermGen space
Out of curiosity, are you colleagues perhaps using Java 8 JDK's? I saw your message mentions JDK 7 and PermGen was removed in Java 8[1]. [1] - http://www.infoq.com/articles/Java-PERMGEN-Removed On Wednesday, December 17, 2014 12:41:29 PM UTC-5, Moritz Ulrich wrote: Hello, I'm getting the following error while working on a Clojure(Script) REPL on a middle-sized project after some time. None of my colleagues seem to be able to reproduce it. I'm not able to reproduce it on other projects either. Caused by: java.lang.OutOfMemoryError: PermGen space, compiling:(cljs/core/async/impl/ioc_macros.clj:348:1) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6651) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6100) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.analyze(Compiler.java:6406) at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782) at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5217) at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3846) at clojure.lang.Compiler.analyzeSeq(Compiler.java:6642) at clojure.lang.Compiler.analyze(Compiler.java:6445) at clojure.lang.Compiler.eval(Compiler.java:6700) at clojure.lang.Compiler.load(Compiler.java:7130) at clojure.lang.RT.loadResourceScript(RT.java:370) at clojure.lang.RT.loadResourceScript(RT.java:361) at clojure.lang.RT.load(RT.java:440) at clojure.lang.RT.load(RT.java:411) at clojure.core$load$fn__5066.invoke(core.clj:5641) at clojure.core$load.doInvoke(core.clj:5640) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invoke(core.clj:5446) at clojure.core$load_lib$fn__5015.invoke(core.clj:5486) at clojure.core$load_lib.doInvoke(core.clj:5485) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invoke(core.clj:626) at clojure.core$load_libs.doInvoke(core.clj:5524) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invoke(core.clj:626) at clojure.core$require.doInvoke(core.clj:5607) at clojure.lang.RestFn.invoke(RestFn.java:408) at cljs.analyzer$eval21264$fn__21266.invoke(analyzer.clj:1190) at clojure.lang.MultiFn.invoke(MultiFn.java:249) at cljs.analyzer$analyze_seq.invoke(analyzer.clj:1490) I'm using lein-repl in combination with weasel to work on the server- and client-side (cljs) of this project. The server-side uses the component-design by stuartsierra, though the permgen issues don't seem to correlate to calls to (reload). $ lein version Leiningen 2.5.0 on Java 1.7.0-u65 OpenJDK 64-Bit Server VM Some of the dependencies (I suspect some of them are responsible): [org.clojure/clojure 1.6.0] [org.clojure/core.async 0.1.346.0-17112a-alpha] [com.stuartsierra/component 0.2.2] [org.clojure/tools.namespace 0.2.7] [lein-cljsbuild 1.0.3] [cider/cider-nrepl 0.8.1] -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: java.lang.OutOfMemoryError when slurping lots of files?
Thank you very much, that worked splendidly.
On Friday, April 5, 2013 5:14:30 PM UTC+2, Alex Nixon wrote:
Java substrings prevent the original string from being garbage collected;
perhaps this also happens with regex matches?
You can test the theory by surrounding the values in your map with
(String. ) and seeing if the problem goes away.
On 5 April 2013 15:57, Adrian Muresan muresan@gmail.com javascript:
wrote:
Hello everyone,
I'm trying to parse a large number of small reports for some data and I'm
doing this by repeatedly calling the following function (with a for) on
each of the files:
(defn get-rep [file]
(let [report (with-open [rdr (io/reader file)](*slurp rdr*))
reg #;\s+(\d+\.\d+) \s+; (\d+\.\d+) \s+;
match (re-matcher reg report)
found (re-find match)
fmax-85c (second found)
rfmax-85c (nth found 2)
found2 (re-find match)
fmax-0c (second found2)
rfmax-0c (nth found2 2)]
{:fmax-85c fmax-85c
:rfmax-85c rfmax-85c
:fmax-0c fmax-0c
:rfmax-0c rfmax-0c}))
My problem is that after a while my program crashes with a jvm heap
error, it crashes on the slurp line (in bold), although that probably
doesn't mean much. The slurp line is the only one that I suspect since it's
the biggest memory consumer in my program.
Now I'm wondering what's going on and why don't the report files get
freed from memory. I'm suspecting this might be caused by lazyness at some
level, but I can't figure out where. Any ideas?
Thanks!
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.comjavascript:
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com javascript:
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+u...@googlegroups.com javascript:.
For more options, visit https://groups.google.com/groups/opt_out.
--
*Alex Nixon*
Software Engineer | SwiftKey
*al...@swiftkey.net javascript:** | http://www.swiftkey.net/*
++
WINNER - MOST INNOVATIVE MOBILE
APPhttp://www.swiftkey.net/swiftkey-wins-most-innovative-app-at-mwc
- GSMA GLOBAL MOBILE AWARDS 2012
Head office: 91-95 Southwark Bridge Road, London, SE1 0AX TouchType is a
limited company registered in England and Wales, number 06671487.
Registered office: 91-95 Southwark Bridge Road, London, SE1 0AX
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: java.lang.OutOfMemoryError when slurping lots of files?
You should show us the calling code, I guess ...
2013/4/5 Adrian Muresan muresan.adrian...@gmail.com:
Hello everyone,
I'm trying to parse a large number of small reports for some data and I'm
doing this by repeatedly calling the following function (with a for) on each
of the files:
(defn get-rep [file]
(let [report (with-open [rdr (io/reader file)](slurp rdr))
reg #;\s+(\d+\.\d+) \s+; (\d+\.\d+) \s+;
match (re-matcher reg report)
found (re-find match)
fmax-85c (second found)
rfmax-85c (nth found 2)
found2 (re-find match)
fmax-0c (second found2)
rfmax-0c (nth found2 2)]
{:fmax-85c fmax-85c
:rfmax-85c rfmax-85c
:fmax-0c fmax-0c
:rfmax-0c rfmax-0c}))
My problem is that after a while my program crashes with a jvm heap error,
it crashes on the slurp line (in bold), although that probably doesn't mean
much. The slurp line is the only one that I suspect since it's the biggest
memory consumer in my program.
Now I'm wondering what's going on and why don't the report files get freed
from memory. I'm suspecting this might be caused by lazyness at some level,
but I can't figure out where. Any ideas?
Thanks!
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: java.lang.OutOfMemoryError when slurping lots of files?
Slurp reads the entire file into memory. Maybe it is a combination of a)
the program taking up more of the heap in other parts as it runs and then
b) a particularly large file?
Is there a reason you can't process the files as a line-seq so you don't
have to load the entire thing into memory all at once?
Also, you could just pass a larger -Xmx value on the command line (or in
the lein java-opts) to make the heap larger.
On Fri, Apr 5, 2013 at 9:57 AM, Adrian Muresan
muresan.adrian...@gmail.comwrote:
Hello everyone,
I'm trying to parse a large number of small reports for some data and I'm
doing this by repeatedly calling the following function (with a for) on
each of the files:
(defn get-rep [file]
(let [report (with-open [rdr (io/reader file)](*slurp rdr*))
reg #;\s+(\d+\.\d+) \s+; (\d+\.\d+) \s+;
match (re-matcher reg report)
found (re-find match)
fmax-85c (second found)
rfmax-85c (nth found 2)
found2 (re-find match)
fmax-0c (second found2)
rfmax-0c (nth found2 2)]
{:fmax-85c fmax-85c
:rfmax-85c rfmax-85c
:fmax-0c fmax-0c
:rfmax-0c rfmax-0c}))
My problem is that after a while my program crashes with a jvm heap error,
it crashes on the slurp line (in bold), although that probably doesn't mean
much. The slurp line is the only one that I suspect since it's the biggest
memory consumer in my program.
Now I'm wondering what's going on and why don't the report files get freed
from memory. I'm suspecting this might be caused by lazyness at some level,
but I can't figure out where. Any ideas?
Thanks!
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: java.lang.OutOfMemoryError when slurping lots of files?
Java substrings prevent the original string from being garbage collected;
perhaps this also happens with regex matches?
You can test the theory by surrounding the values in your map with (String.
) and seeing if the problem goes away.
On 5 April 2013 15:57, Adrian Muresan muresan.adrian...@gmail.com wrote:
Hello everyone,
I'm trying to parse a large number of small reports for some data and I'm
doing this by repeatedly calling the following function (with a for) on
each of the files:
(defn get-rep [file]
(let [report (with-open [rdr (io/reader file)](*slurp rdr*))
reg #;\s+(\d+\.\d+) \s+; (\d+\.\d+) \s+;
match (re-matcher reg report)
found (re-find match)
fmax-85c (second found)
rfmax-85c (nth found 2)
found2 (re-find match)
fmax-0c (second found2)
rfmax-0c (nth found2 2)]
{:fmax-85c fmax-85c
:rfmax-85c rfmax-85c
:fmax-0c fmax-0c
:rfmax-0c rfmax-0c}))
My problem is that after a while my program crashes with a jvm heap error,
it crashes on the slurp line (in bold), although that probably doesn't mean
much. The slurp line is the only one that I suspect since it's the biggest
memory consumer in my program.
Now I'm wondering what's going on and why don't the report files get freed
from memory. I'm suspecting this might be caused by lazyness at some level,
but I can't figure out where. Any ideas?
Thanks!
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
*Alex Nixon*
Software Engineer | SwiftKey
*a...@swiftkey.net** | http://www.swiftkey.net/*
++
WINNER - MOST INNOVATIVE MOBILE
APPhttp://www.swiftkey.net/swiftkey-wins-most-innovative-app-at-mwc
- GSMA GLOBAL MOBILE AWARDS 2012
Head office: 91-95 Southwark Bridge Road, London, SE1 0AX TouchType is a
limited company registered in England and Wales, number 06671487.
Registered office: 91-95 Southwark Bridge Road, London, SE1 0AX
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: java.lang.OutOfMemoryError when slurping lots of files?
Alex Nixon managed to figure it out (further down)
offtopic: I live in the same city as you and I'm also interested in
clojure, email me if you want to have a coffee
Adrian
On Friday, April 5, 2013 5:08:04 PM UTC+2, Laurent PETIT wrote:
You should show us the calling code, I guess ...
2013/4/5 Adrian Muresan muresan@gmail.com javascript::
Hello everyone,
I'm trying to parse a large number of small reports for some data and
I'm
doing this by repeatedly calling the following function (with a for) on
each
of the files:
(defn get-rep [file]
(let [report (with-open [rdr (io/reader file)](slurp rdr))
reg #;\s+(\d+\.\d+) \s+; (\d+\.\d+) \s+;
match (re-matcher reg report)
found (re-find match)
fmax-85c (second found)
rfmax-85c (nth found 2)
found2 (re-find match)
fmax-0c (second found2)
rfmax-0c (nth found2 2)]
{:fmax-85c fmax-85c
:rfmax-85c rfmax-85c
:fmax-0c fmax-0c
:rfmax-0c rfmax-0c}))
My problem is that after a while my program crashes with a jvm heap
error,
it crashes on the slurp line (in bold), although that probably doesn't
mean
much. The slurp line is the only one that I suspect since it's the
biggest
memory consumer in my program.
Now I'm wondering what's going on and why don't the report files get
freed
from memory. I'm suspecting this might be caused by lazyness at some
level,
but I can't figure out where. Any ideas?
Thanks!
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.comjavascript:
Note that posts from new members are moderated - please be patient with
your
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com javascript:
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google
Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to clojure+u...@googlegroups.com javascript:.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: java.lang.OutOfMemoryError when consuming many whole-numbers
When I do that, the REPL starts printing the sequence, filling screens after screens with numbers. By doing that, it realizes the printed part of the sequence, which will eventually lead to an OOM error, since it probably holds on to the reference to the start of the sequence. Doing (set! clojure.core/*print-length* 23) will limit printing to the first 23 items of the sequence. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError when consuming many whole-numbers
Ben berndhe...@gmx.de writes: (defn whole-numbers [] (iterate inc 1)) If I use it like this at the REPL (take (whole-numbers)) I get: Java heap space [Thrown class java.lang.OutOfMemoryError] This unexpectedly is the same result that I expectedly get when binding whole-numbers to a top-level var and use a huge portion of that. Since whole-numbers is lazy I expected it to run for a certain amount of time but never to run out of memory. Has this anything to do with using the function at the REPL or is there another reason for that? Yes, it's because of the REPL. For convenience the REPL holds onto the last 3 values as *1 *2 and *3. Thus there is a reference to the head of the sequence preventing it from being garbage collected. Try (println (take (whole-numbers))) instead. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError when consuming many whole-numbers
On Sat, Jul 30, 2011 at 2:25 PM, Ben berndhe...@gmx.de wrote: Hi, I'm new in the community and new to Clojure and both a really great. A big thanks to Rich Hickey and everyone else involved. I have a question that refers to an example Stuart Halloway used in Programming Clojure. There he defines the whole numbers as fn: (defn whole-numbers [] (iterate inc 1)) If I use it like this at the REPL (take (whole-numbers)) I get: Java heap space [Thrown class java.lang.OutOfMemoryError] This unexpectedly is the same result that I expectedly get when binding whole-numbers to a top-level var and use a huge portion of that. Since whole-numbers is lazy I expected it to run for a certain amount of time but never to run out of memory. Has this anything to do with using the function at the REPL or is there another reason for that? With kind regards Ben Hi, the lazy sequence returned is a caching one... so I guess that the cache blows the heap... Lars -- Mystische Erklärungen: Die mystischen Erklärungen gelten für tief; die Wahrheit ist, dass sie noch nicht einmal oberflächlich sind. -- Friedrich Nietzsche [ Die Fröhliche Wissenschaft Buch 3, 126 ] -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError when consuming many whole-numbers
Limiting *print-length* keeps the OutOfMemoryError away, but I guess it would leave me - when testing more complicated and obscure functions - insecure whether the returned sequence really is a lazy one or will blow up the memory instead. But good to know anyway ... I guess the println function is what I want to use for verifying the laziness of a function. Are there any use cases where it makes sense to bind a lazy sequence to a var and not to a fn, other than knowing the returned sequence will always be small? (Something like: When you want to plug it into this or into that, then you better have it bound to a var ...) Or do I just not even think about it and always bind it to a fn? Ben -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
So in case anyone else stumbles across this topic, I thought I'd share
what little I have learned about the laziness of concat, and by
extension mapcat, as used in this function.
(defn json-seq [dir-name]
(mapcat #(do (print f) (str/split (slurp %) #\nStatusJSONImpl))
(out-files dir-name)))
It seems that because of the particular way concat is written, it
keeps looking ahead by two or three items. However this doesn't
appear to be a necessary aspect of its behaviour. So the following
version of json-seq, incorporating what is essentially a rewritten
concat, doesn't suffer from the same problem.
(defn json-seq [dir-name]
(letfn [(cat [xs fs]
(lazy-seq
(if-let [xs (seq xs)]
(cons (first xs) (cat (rest xs) fs))
(if-let [fs (seq fs)]
(cat (do
(print f)
(str/split (slurp (first fs)) #\nStatusJSONImpl))
(rest fs))]
(cat '() (out-files dir-name
Alistair
On Jul 26, 2:53 pm, atucker agjf.tuc...@googlemail.com wrote:
Hi all! I have been trying to use Clojure on a student project, but
it's becoming a bit of a nightmare. I wonder whether anyone can
help? I'm not studying computer science, and I really need to be
getting on with the work I'm actually supposed to be doing :)
I am trying to work from a lot of Twitter statuses that I saved to
text file. (Unfortunately I failed to escape quotes and such, so the
JSON is not valid. Anyone know a good way of coping with that?)
Here is my function:
(defn json-seq []
(apply concat
(map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl))
out-files)))
Now there are forty files and five thousand statuses per file, which
sounds like a lot, and I don't suppose I can hope to hold them all in
memory at the same time. But I had thought that my function might
produce a lazy sequence that would be more manageable. However I
typically get:
twitter.core (nth (json-seq dir-name) 5)
{createdAt=Fri etc. GOOD
twitter.core (nth (json-seq dir-name) 5000)
Java heap space
[Thrown class java.lang.OutOfMemoryError] BAD
And at this point my REPL is done for. Any further instruction will
result in anotherOutOfMemoryError. (Surely that has to be a bug just
there? Has the garbage collector just given up?)
Anyway I am thinking that the sequence is not behaving as lazily as I
need it to. It's not reading one file at a time, and it's not reading
thirty-two as I might expect from chunks, but something in the
middle. I did try the dechunkifying code from page 339 of Joy of
Clojure, but that doesn't compile at all :(
I do seem to keep running into memory problems with Clojure. I have
2GB RAM and am using Snow Leopard, Aquamacs 2.0, Clojure 1.2.0 beta1
and Leiningen 1.2.0.
Cheers
Alistair
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
the entire sequence being in memory. However, if you retain the head of the sequence elsewhere, you will see the same effect. I don't think my function retains the head? Please correct me if I am wrong. Not that I can see but I don't have the full context. I tried reproducing just now and I was not able to trigger any memory use beyond what I expected. Do you have a self-contained fully working example that you can point to (preferably with your actual data files + code, but otherwise just the code)? -- / Peter Schuller -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
Thanks! This is still driving me mad 'though. On Jul 27, 5:11 pm, Peter Schuller peter.schul...@infidyne.com wrote: The observations that the data structures are non-lazy still apply, even if you could postpone the problem by increasing the heap size. Yes I can see that the sequence returned from str/split is not lazy. Nonetheless it is quite clear that my set-up is quite capable of keeping one, two or three of these fully realised sequences in memory without blowing up. The problem is that it is attempting to hold several. A number larger than one yet smaller than thirty-two. You're quite right, increasing the heap size has merely delayed the inevitable. As for swank, I do believe it's keeping a history of return values from previous commands in a variable whose name I forget. At least that was the case when I was using Common Lisp; I'm not sure if clojure-swank does the same. A common pitfall was exactly this; large structures being retained in this history such that one perceived that memory was apparently leaking. Swank does indeed keep the last three returned values (at least) in memory, under *1 *2 and *3. However I am generally careful not to return large values to the REPL. Quite apart from the memory issues, it takes an age to print them to screen. the entire sequence being in memory. However, if you retain the head of the sequence elsewhere, you will see the same effect. I don't think my function retains the head? Please correct me if I am wrong. A -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
Cheers. But tests suggest to me that (for...) has the same laziness
characteristics --or lack thereof-- as does (map...)
On Jul 26, 6:56 pm, Randy Hudson randy_hud...@mac.com wrote:
You can get a lazy sequence of all the lines in all the files by
something like:
(for [file out-files
line (with-open [r (io/reader file)] (line-seq r))]
line)
If StatusJSONImpl is on a separate line, you can throw in a :when
clause to filter them out:
(for [file out-files
line (with-open [r (io/reader file)] (line-seq r))
:when (not= line StatusJSONImpl)]
line)
If it's a line prefix, you can remove it in the body:
(for [file out-files
line (with-open [r (io/reader file)] (line-seq r))]
(string/replace line StatusJSONImpl ))
This is all assuming io is an alias for clojure.java.io, string for
clojure.string, and that getting your files line by line is useful.
Re OutOfMemoryException: if all the allocated heap memory is really
not freeable, then there's nothing the JVM can do -- it's being asked
to allocate memory for a new object, and there's none available.
On Jul 26, 9:53 am, atucker agjf.tuc...@googlemail.com wrote:
Hi all! I have been trying to use Clojure on a student project, but
it's becoming a bit of a nightmare. I wonder whether anyone can
help? I'm not studying computer science, and I really need to be
getting on with the work I'm actually supposed to be doing :)
I am trying to work from a lot of Twitter statuses that I saved to
text file. (Unfortunately I failed to escape quotes and such, so the
JSON is not valid. Anyone know a good way of coping with that?)
Here is my function:
(defn json-seq []
(apply concat
(map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl))
out-files)))
Now there are forty files and five thousand statuses per file, which
sounds like a lot, and I don't suppose I can hope to hold them all in
memory at the same time. But I had thought that my function might
produce a lazy sequence that would be more manageable. However I
typically get:
twitter.core (nth (json-seq dir-name) 5)
{createdAt=Fri etc. GOOD
twitter.core (nth (json-seq dir-name) 5000)
Java heap space
[Thrown class java.lang.OutOfMemoryError] BAD
And at this point my REPL is done for. Any further instruction will
result in anotherOutOfMemoryError. (Surely that has to be a bug just
there? Has the garbage collector just given up?)
Anyway I am thinking that the sequence is not behaving as lazily as I
need it to. It's not reading one file at a time, and it's not reading
thirty-two as I might expect from chunks, but something in the
middle. I did try the dechunkifying code from page 339 of Joy of
Clojure, but that doesn't compile at all :(
I do seem to keep running into memory problems with Clojure. I have
2GB RAM and am using Snow Leopard, Aquamacs 2.0, Clojure 1.2.0 beta1
and Leiningen 1.2.0.
Cheers
Alistair
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
On Mon, Jul 26, 2010 at 9:53 AM, atucker agjf.tuc...@googlemail.com wrote: Here is my function: (defn json-seq [] (apply concat (map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl)) out-files))) Try removing the apply concat at the front, I'm pretty sure that's making your sequence non-lazy. Mark -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
Here is my function: (defn json-seq [] (apply concat (map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl)) out-files))) Try removing the apply concat at the front, I'm pretty sure that's making your sequence non-lazy. Correct me if I'm wrong but that should only apply to the list of list that concat is supposed to be concatenating; the contents of *those* lists would still be lazy (if the original list was lazy to begin with). -- / Peter Schuller -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
Thanks! but not entirely convinced. At my REPL: user (repeatedly 10 #(do (print f) [(rand-int 10)])) (ff[0] f[8] f[5] f[7] f[1] f[6] f[7] f[3] f[3] [0]) user (take 5 (apply concat (repeatedly 10 #(do (print f) [(rand-int 10)] (7 1 f6 f5 8) Only six fs... so doesn't that mean the (repeatedly... ) is being evaluated lazily? Alistair On Jul 27, 12:43 pm, Mark Nutter manutte...@gmail.com wrote: On Mon, Jul 26, 2010 at 9:53 AM, atucker agjf.tuc...@googlemail.com wrote: Here is my function: (defn json-seq [] (apply concat (map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl)) out-files))) Try removing the apply concat at the front, I'm pretty sure that's making your sequence non-lazy. Mark -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
Thanks Sean, your first suggestion was a very good one :)
Tweaking JVM settings feels like advanced magic, and I am a little
surprised that it is necessary at such an early stage in my Clojure
journey. But googling confirms that the default JVM settings are
miserly to an extreme, and I need at least to insert an :jvm-opts [-
server] in my project.clj. I would suggest to the author of
Leiningen that perhaps this should be made the default?
I am getting a lot further now, but still running into OutOfMemory
errors sometimes. And it is still the case that once I have suffered
an OutOfMemoryError, they keep coming. It does feel as if there must
be some large memory leak in the emacs/lein swank repl. Is this a
recognised issue?
The (print f) is indeed there only for debugging purposes. I don't
think it affects the laziness? And unfortunately I am not quite sure
how to act on your other suggestions regarding processing workflow,
since at the moment this is more of an exploratory project.
I shall read the other suggestions regarding laziness later, and
hopefully get somewhere with those. Thanks all!
Alistair.
On Jul 26, 3:18 pm, Sean Devlin francoisdev...@gmail.com wrote:
My first thought is that you need to tweak your JVM settings. Try
allocation a minimum of 512MB to the total.
My second thought is that you need to use laziness to your advantage.
Remove the print expression from the mapping operation. It's useful
for debugging/prototyping, but shouldn't be in the final version.
Spit the processed json-seq into a file when you're done instead.
This way you can process one input file at a time, and simply append
your results to the output file.
My $.02
Sean
On Jul 26, 9:53 am, atucker agjf.tuc...@googlemail.com wrote:
Hi all! I have been trying to use Clojure on a student project, but
it's becoming a bit of a nightmare. I wonder whether anyone can
help? I'm not studying computer science, and I really need to be
getting on with the work I'm actually supposed to be doing :)
I am trying to work from a lot of Twitter statuses that I saved to
text file. (Unfortunately I failed to escape quotes and such, so the
JSON is not valid. Anyone know a good way of coping with that?)
Here is my function:
(defn json-seq []
(apply concat
(map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl))
out-files)))
Now there are forty files and five thousand statuses per file, which
sounds like a lot, and I don't suppose I can hope to hold them all in
memory at the same time. But I had thought that my function might
produce a lazy sequence that would be more manageable. However I
typically get:
twitter.core (nth (json-seq dir-name) 5)
{createdAt=Fri etc. GOOD
twitter.core (nth (json-seq dir-name) 5000)
Java heap space
[Thrown class java.lang.OutOfMemoryError] BAD
And at this point my REPL is done for. Any further instruction will
result in another OutOfMemoryError. (Surely that has to be a bug just
there? Has the garbage collector just given up?)
Anyway I am thinking that the sequence is not behaving as lazily as I
need it to. It's not reading one file at a time, and it's not reading
thirty-two as I might expect from chunks, but something in the
middle. I did try the dechunkifying code from page 339 of Joy of
Clojure, but that doesn't compile at all :(
I do seem to keep running into memory problems with Clojure. I have
2GB RAM and am using Snow Leopard, Aquamacs 2.0, Clojure 1.2.0 beta1
and Leiningen 1.2.0.
Cheers
Alistair
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
I am getting a lot further now, but still running into OutOfMemory errors sometimes. And it is still the case that once I have suffered an OutOfMemoryError, they keep coming. It does feel as if there must be some large memory leak in the emacs/lein swank repl. Is this a recognised issue? The observations that the data structures are non-lazy still apply, even if you could postpone the problem by increasing the heap size. As for swank, I do believe it's keeping a history of return values from previous commands in a variable whose name I forget. At least that was the case when I was using Common Lisp; I'm not sure if clojure-swank does the same. A common pitfall was exactly this; large structures being retained in this history such that one perceived that memory was apparently leaking. Anyone know if this is true with clojure-swank? The (print f) is indeed there only for debugging purposes. I don't think it affects the laziness? Print does not affect the lazyness of its parameter, but it *will* consume its parameter. Under the assumption that print internally does not retain a reference to its head, this would not directly lead to the entire sequence being in memory. However, if you retain the head of the sequence elsewhere, you will see the same effect. For example: (let [s (for [x [1 2 3 4 5 6]] (do (println (str x being materialized)) x))] (println before first print) (println s) (println before second print) (println s)) This will yield the output: before first print (1 being materialized 2 being materialized 3 being materialized 4 being materialized 5 being materialized 6 being materialized 1 2 3 4 5 6) before second print (1 2 3 4 5 6) So; consuming a lazy data structure while retaining a reference to its head, will result in memory use similar to that which you may have expected if the data structure was not lazy to begin with. -- / Peter Schuller -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
My first thought is that you need to tweak your JVM settings. Try
allocation a minimum of 512MB to the total.
My second thought is that you need to use laziness to your advantage.
Remove the print expression from the mapping operation. It's useful
for debugging/prototyping, but shouldn't be in the final version.
Spit the processed json-seq into a file when you're done instead.
This way you can process one input file at a time, and simply append
your results to the output file.
My $.02
Sean
On Jul 26, 9:53 am, atucker agjf.tuc...@googlemail.com wrote:
Hi all! I have been trying to use Clojure on a student project, but
it's becoming a bit of a nightmare. I wonder whether anyone can
help? I'm not studying computer science, and I really need to be
getting on with the work I'm actually supposed to be doing :)
I am trying to work from a lot of Twitter statuses that I saved to
text file. (Unfortunately I failed to escape quotes and such, so the
JSON is not valid. Anyone know a good way of coping with that?)
Here is my function:
(defn json-seq []
(apply concat
(map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl))
out-files)))
Now there are forty files and five thousand statuses per file, which
sounds like a lot, and I don't suppose I can hope to hold them all in
memory at the same time. But I had thought that my function might
produce a lazy sequence that would be more manageable. However I
typically get:
twitter.core (nth (json-seq dir-name) 5)
{createdAt=Fri etc. GOOD
twitter.core (nth (json-seq dir-name) 5000)
Java heap space
[Thrown class java.lang.OutOfMemoryError] BAD
And at this point my REPL is done for. Any further instruction will
result in another OutOfMemoryError. (Surely that has to be a bug just
there? Has the garbage collector just given up?)
Anyway I am thinking that the sequence is not behaving as lazily as I
need it to. It's not reading one file at a time, and it's not reading
thirty-two as I might expect from chunks, but something in the
middle. I did try the dechunkifying code from page 339 of Joy of
Clojure, but that doesn't compile at all :(
I do seem to keep running into memory problems with Clojure. I have
2GB RAM and am using Snow Leopard, Aquamacs 2.0, Clojure 1.2.0 beta1
and Leiningen 1.2.0.
Cheers
Alistair
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
Here is my function: (defn json-seq [] (apply concat (map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl)) out-files))) Assuming the str namespace is clojure.contrib.string, (str/split ..) won't be lazy. Currently it's implemented as: (defn split Splits string on a regular expression. Optional argument limit is the maximum number of splits. ([#^Pattern re #^String s] (seq (.split re s))) ([#^Pattern re limit #^String s] (seq (.split re s limit So, while the sequence returned by concat would be lazy in and of itself, ultimately there will be the non-lazy return value of Pattern.split() in memory. Also, is it considered idiomatic to rely on lazyness in cases like this, anyone? -- / Peter Schuller -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: java.lang.OutOfMemoryError
You can get a lazy sequence of all the lines in all the files by
something like:
(for [file out-files
line (with-open [r (io/reader file)] (line-seq r))]
line)
If StatusJSONImpl is on a separate line, you can throw in a :when
clause to filter them out:
(for [file out-files
line (with-open [r (io/reader file)] (line-seq r))
:when (not= line StatusJSONImpl)]
line)
If it's a line prefix, you can remove it in the body:
(for [file out-files
line (with-open [r (io/reader file)] (line-seq r))]
(string/replace line StatusJSONImpl ))
This is all assuming io is an alias for clojure.java.io, string for
clojure.string, and that getting your files line by line is useful.
Re OutOfMemoryException: if all the allocated heap memory is really
not freeable, then there's nothing the JVM can do -- it's being asked
to allocate memory for a new object, and there's none available.
On Jul 26, 9:53 am, atucker agjf.tuc...@googlemail.com wrote:
Hi all! I have been trying to use Clojure on a student project, but
it's becoming a bit of a nightmare. I wonder whether anyone can
help? I'm not studying computer science, and I really need to be
getting on with the work I'm actually supposed to be doing :)
I am trying to work from a lot of Twitter statuses that I saved to
text file. (Unfortunately I failed to escape quotes and such, so the
JSON is not valid. Anyone know a good way of coping with that?)
Here is my function:
(defn json-seq []
(apply concat
(map #(do (print f) (str/split (slurp %) #\nStatusJSONImpl))
out-files)))
Now there are forty files and five thousand statuses per file, which
sounds like a lot, and I don't suppose I can hope to hold them all in
memory at the same time. But I had thought that my function might
produce a lazy sequence that would be more manageable. However I
typically get:
twitter.core (nth (json-seq dir-name) 5)
{createdAt=Fri etc. GOOD
twitter.core (nth (json-seq dir-name) 5000)
Java heap space
[Thrown class java.lang.OutOfMemoryError] BAD
And at this point my REPL is done for. Any further instruction will
result in another OutOfMemoryError. (Surely that has to be a bug just
there? Has the garbage collector just given up?)
Anyway I am thinking that the sequence is not behaving as lazily as I
need it to. It's not reading one file at a time, and it's not reading
thirty-two as I might expect from chunks, but something in the
middle. I did try the dechunkifying code from page 339 of Joy of
Clojure, but that doesn't compile at all :(
I do seem to keep running into memory problems with Clojure. I have
2GB RAM and am using Snow Leopard, Aquamacs 2.0, Clojure 1.2.0 beta1
and Leiningen 1.2.0.
Cheers
Alistair
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
