Anyone have any insights here? Really the most important thing I'm trying to learn is 2) how to identify when a lazy seq head is being retained, other than waiting for it to become bad enough that your program OOMs.
> On Jun 16, 2017, at 6:14 PM, Luke Burton <luke_bur...@me.com> wrote: > > > Riddle me this: > > https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e > <https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e> > > Wherein I synthesize a large table in Postgres, then attempt to lazily load > the table, discarding each row as I receive it. I tried *many* permutations > and experiments, but settled on these two tests to illustrate my point. Which > is that I simply can't get it to work with clojure.java.jdbc. > > test1, according to all my research and reading of the source code involved, > should consume the query results lazily. It does not, and I can't for the > life of me figure out why. Traffic starts to stream in, and the heap is > overwhelmed almost immediately. I've deliberately set the heap to 1 GB. > > test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99 > <https://dev.clojure.org/jira/browse/JDBC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs>, > which is to have ResultSet implement IReduceInit. It consumes a nominal > amount of memory. I've verified it's actually doing something by putting > counters in, and using YourKit to watch about 20 MB/s of traffic streaming > into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage. > > I used YourKit to track where the memory is being retained for test1. > Initially I made the mistake of not setting the fetchSize, so I saw an > ArrayList inside the driver holding the reference. The driver documentation > <https://jdbc.postgresql.org/documentation/head/query.html> confirms that > autoCommit must be disabled and the fetchSize set to some non-zero number. > > After making that change, YourKit confirmed that the GC root holding all the > memory was the stack local variable "rs". At least I think it did, as a > non-expert in this domain. I tried disassembling the functions using > no.disassemble and the IntelliJ decompiler but I'm not really at the point > where I understand what to look for. > > So my questions are: > > 1) what am I doing wrong with clojure.java.jdbc? > > Note some things I've already tried: > > * using row-fn instead of result-set-fn > * using prepared statements > * explicitly setting auto-commit false on the connection > * declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a > change in the disassembly when using this) > * probably other things I am forgetting > > 2) in these situations where you suspect that the head of a lazy sequence is > being retained, how do you reason about it? I'm kind of lucky this one blew > the heap so quickly, who knows how much of my production code might burning > memory unnecessarily but not quite as fatally. Do you disassemble the > functions and observe some smoking gun? How do you peek under the covers to > see where the problem is? > > Luke. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > <http://groups.google.com/group/clojure?hl=en> > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com > <mailto:clojure+unsubscr...@googlegroups.com>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
