Hi Brian, Both plain seqs or reducibles can be consumed without realizing the whole thing in memory, especially important (as you note) when the whole thing is humongous and dwarfs available memory. To do it with a lazy seq, as in clojure.java.jdbc, you have to care of a few things. You must override :result-set-fn from clojure.java.jdbc/query [1] so that you can control realization through doseq or reduce.
Forbidden things: calling `doall` on the seq, or pouring it into a vector, or realizing the whole seq while still hanging on its head, or making a let binding of the resultset that you use more than once, or leak carelessly to some helper. If you have multiple bindings, my guess is that there is some unit of work for every item of a seq realized from a large 'main' query. The same advice would apply here too, you just have to take care on many more queries to allow the garbage collector to do its job. If you have a lot of "layers" perhaps you could decompose that into some simpler computations, or use a work queue mechanism or core.async channel. Do you have a concrete example of the multiple bindings scenario? Make sure that the underlying JDBC driver is not working against you by buffering the whole thing! MySQL's JDBC driver, for example, requires a few magical settings [2] to prevent it from buffering internally [1] http://clojure.github.io/java.jdbc/#clojure.java.jdbc/query [2] From https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html: stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY, java.sql.ResultSet.CONCUR_READ_ONLY); stmt.setFetchSize(Integer.MIN_VALUE); The combination of a forward-only, read-only result set, with a fetch size of Integer.MIN_VALUE serves as a signal to the driver to stream result sets row-by-row. After this, any result sets created with the statement will be retrieved row-by-row. On Thursday, May 4, 2017 at 1:35:48 PM UTC-4, Brian Craft wrote: > > The with-open style is used a lot in the jdbc lib, and elsewhere. It's > pretty simple when data is very small, as you can just evaluate the entire > result with doall, etc. > > How do you deal with larger data, where you need to evaluate iteratively? > If there's only one with-open it can be reasonably simple to pass the > consumer into that context (though from ring it's pretty convoluted, due to > needing to pass control to ring first). But if there are multiple with-open > you have to nest them, with a continuation passing style, or middleware > pattern, or something, which quickly becomes onerous as it affects all the > code surrounding the with-open. > > Is there some simpler pattern? > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
