Thank Ken, your suggestion solved my problem with the OOM exception.
I tried your suggestion to run it in parallel but I didn't see much
difference. Instead I called future on the let call and that helped
the performance.
On Dec 17, 2:55 pm, Ken Wesson kwess...@gmail.com wrote:
On Fri, Dec 17,
(defn persist-rows
[headers rows id]
(let [mrows (transform-rows rows id)]
(with-db *db* (try
(apply insert-into-table
:my-table
[:col1 :col2 :col3]
mrows)))
nil ))
(defn filter-data
[rows item-id header id]
(persist-rows
On Fri, Dec 17, 2010 at 5:39 PM, clj123 ariela2...@gmail.com wrote:
(defn persist-rows
[headers rows id]
(let [mrows (transform-rows rows id)]
(with-db *db* (try
(apply insert-into-table
:my-table
[:col1 :col2 :col3]
mrows)))
Hello,
I'm trying to insert in a database large number of records, however
it's not scaling correctly. For 100 records it takes 10 seconds, for
100 records it takes 2 min to save. But for 250 records it
throws Java Heap out of memory exception.
I've tried separting the records processing
You might be coming to near OOM with using in-memory processing but
don't know it, and the batched (lazy) version is probably holding onto
data creating the mem leak. Would you be able to post the relevant
source?
--
You received this message because you are subscribed to the Google
Groups
On Thu, Dec 16, 2010 at 09:19, clj123 ariela2...@gmail.com wrote:
Hello,
I'm trying to insert in a database large number of records, however
it's not scaling correctly. For 100 records it takes 10 seconds, for
100 records it takes 2 min to save. But for 250 records it
throws Java
