This is excellent news as far as I’m concerned because it shows there’s no specific bug in clojure.java.jdbc that is fundamentally causing the OOM problem you’re seeing!
(that’s not to say there aren’t _other_ bugs in clojure.java.jdbc and the idea of the reducible result set definitely has appeal, which is why there’s a JIRA issue for it, but it doesn’t appear to be necessary in this instance). Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN An Architect's View -- http://corfield.org/ "If you're not annoying somebody, you're not really alive." -- Margaret Atwood On 6/23/17, 2:03 PM, "Luke Burton" <clojure@googlegroups.com on behalf of luke_bur...@me.com> wrote: Well, shoot. I went back and revisited this because it was bugging me … I looked at the code generated with and without usage of ^:once fn* and that led me down the right path. TL;DR – the problem appears to be locals clearing always disabled in Cursive REPL. This is just my current hypothesis based on testing and not 100% certainty. I guess I'll put this in a blog post or something once I validate this hypothesis and get some feedback from Colin (1.6.0-eap1-2017.2 if you're watching, Colin ;) For mailing list posterity, here's the difference, using Lubomir's examples and the decompiler in IntelliJ, between ^:once fn* and regular fn (or fn*) Note the inner-bad function: (defn bad [] (let [x (for [n (range)] (make-array Object 10000)), f (fn* inner-bad [] (nth x 1e6))] (f))) The inner-bad class generated is: public final class core$bad$inner_bad__431 extends AFunction { Object x; public static final Object const__1 = 1000000.0D; public core$bad$inner_bad__431(Object var1) { this.x = var1; } public Object invoke() { return RT.nth(this.x, RT.intCast((Number)const__1)); } } Note the inner-good function: (defn good [] (let [x (for [n (range)] (make-array Object 10000)), f (^:once fn* inner-good [] (nth x 1e6))] (f))) The inner-good class generated is: public final class core$good$inner_good__414 extends AFunction { Object x; public static final Object const__1 = 1000000.0D; public core$good$inner_good__414(Object var1) { this.x = var1; } public Object invoke() { Object var10000 = this.x; this.x = null; return RT.nth(var10000, RT.intCast((Number)const__1)); } } You can see pretty clearly the difference is that each time the function is invoked in the "good" case, the local reference to this.x is cleared, meaning this function really is only good for one invocation. No wonder it's undocumented! => (let [z 1000 f (^:once fn* one-shot [x] (println x)) p (f z)] (p) (p)) 1000 CompilerException java.lang.NullPointerException Anyway, I saw this and jumped to the thought: "locals!! I have fallen victim to uncleared locals!!" So I started some tests, going back to my previous code and trying the JDBC result set that wasn't being lazy before. I put the test1 invocation into -main and tried: · lein run – no OOM. · lein repl, then invoking (-main) – no OOM. · lein repl :headless, then connecting via lein repl :connect "localhost:54321" and invoke (-main) – no OOM. · Cursive nREPL via Leiningen (locals clearing, debug REPL) – OOM!! · Cursive nREPL via Leiningen (no locals clearing, debug REPL) – OOM!! · Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!! · Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!! · lein run, with -Dclojure.compiler.disable-locals-clearing=true – OOM!! I feel slightly better for knowing I can point the finger of blame at disabled locals clearing. It would be great to have this fixed in Cursive, considering that I run quite memory intensive applications and I'm sure locals clearing being permanently disabled has caused me to engineer around some non-existent problems. I have not tested CIDER or any other REPL, YMMV. Luke. On Jun 20, 2017, at 11:26 AM, Lubomir Konstantinov <lubo...@gmail.com> wrote: ^{:once true} should deal with this. Try the following simplified test case with and without it: (defn query [result-fn] (let [x (for [n (range 1e6)] (make-array Object 100000)) f (^:once fn* [rs] (result-fn rs))] (f x))) (defn testq [] (let [myfn (fn [rs] (doseq [r rs] nil))] (query myfn))) On Tuesday, 20 June 2017 03:47:45 UTC+3, James Reeves wrote: This might be a bug in java.jdbc. The code that passes the result set seq to the function is: (^{:once true} fn* [rset] ((^{:once true} fn* [rs] (result-set-fn (if as-arrays? (cons (first rs) (map row-fn (rest rs))) (map row-fn rs)))) (result-set-seq rset opts))) I'm wondering if this function holds onto the head of the seq, since it's bound to "rs". On 20 June 2017 at 00:20, Luke Burton <luke_...@me.com> wrote: Anyone have any insights here? Really the most important thing I'm trying to learn is 2) how to identify when a lazy seq head is being retained, other than waiting for it to become bad enough that your program OOMs. On Jun 16, 2017, at 6:14 PM, Luke Burton <luke_...@me.com> wrote: Riddle me this: https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e Wherein I synthesize a large table in Postgres, then attempt to lazily load the table, discarding each row as I receive it. I tried *many* permutations and experiments, but settled on these two tests to illustrate my point. Which is that I simply can't get it to work with clojure.java.jdbc. test1, according to all my research and reading of the source code involved, should consume the query results lazily. It does not, and I can't for the life of me figure out why. Traffic starts to stream in, and the heap is overwhelmed almost immediately. I've deliberately set the heap to 1 GB. test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99, which is to have ResultSet implement IReduceInit. It consumes a nominal amount of memory. I've verified it's actually doing something by putting counters in, and using YourKit to watch about 20 MB/s of traffic streaming into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage. I used YourKit to track where the memory is being retained for test1. Initially I made the mistake of not setting the fetchSize, so I saw an ArrayList inside the driver holding the reference. The driver documentation confirms that autoCommit must be disabled and the fetchSize set to some non-zero number. After making that change, YourKit confirmed that the GC root holding all the memory was the stack local variable "rs". At least I think it did, as a non-expert in this domain. I tried disassembling the functions using no.disassemble and the IntelliJ decompiler but I'm not really at the point where I understand what to look for. So my questions are: 1) what am I doing wrong with clojure.java.jdbc? Note some things I've already tried: * using row-fn instead of result-set-fn * using prepared statements * explicitly setting auto-commit false on the connection * declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a change in the disassembly when using this) * probably other things I am forgetting 2) in these situations where you suspect that the head of a lazy sequence is being retained, how do you reason about it? I'm kind of lucky this one blew the heap so quickly, who knows how much of my production code might burning memory unnecessarily but not quite as fatally. Do you disassemble the functions and observe some smoking gun? How do you peek under the covers to see where the problem is? Luke. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clo...@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clo...@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- James Reeves booleanknot.com -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
