Post the arguments you used to start the jvm. All you probably need to do is increase the Heap Space. If you are using the default jvm setting the default heap space is really small. To increase the space use the -Xmx(Size) flag.
Its using StringBuilder which is the best way to create large strings. Its running out of memory when the StringBuilder tries to allocate more space to make the character array bigger. It does this by allocating a bigger array and copying the old array contents into the new array. Here is information on how StringBuilder works: http://docs.oracle.com/javase/1.7.0/docs/api/java/lang/StringBuilder.html . On Fri, Jan 4, 2013 at 5:06 PM, larry google groups < lawrencecloj...@gmail.com> wrote: > > I am still somewhat new to Clojure and the JVM. I am querying a database > and trying to output some XML. For the XML, I am using this library: > > https://github.com/clojure/data.xml > > Apparently the database queries worked, and I was able to get the data > into an XML structure, but at some point the data gets lost and nothing is > being output. I decided, for the sake of debugging, I would add in some > println statements and output it to the terminal. > > I was storing the xml in an atom called recent-activity. I attempted to > store this as clojure.data.xml elements in a vector. I got no output so I > decided t switch to emit-str and store it as a string. Then suddenly I > got an OutOfMemory error. I found this very surprising. Are strings that > expensive on memory? > > The code looks like this. The first function does the query against the > database: > > (defn update-recent-discourse-from-this-site [db k] > (jdbc/with-connection db > (jdbc/with-query-results database-results > [(str > " SELECT > d.id, d.description, d.created_at, > p.id as profile_id, p.first_name, p.last_name, > u.username " > " FROM discourse as d, sf_guard_user as u, sf_guard_user_profile > as p " > " WHERE d.user_id=p.user_id " > " AND d.user_id=u.id " > " AND p.user_id=u.id " > " AND d.question_id = 0 " > " AND d.answer_id = 0 " > " AND d.discourse_id = 0 " > " AND d.created_at > ? " > " ORDER BY d.created_at DESC " > " LIMIT 100 ") > (das/one-month-ago-as-a-string-for-the-database)] > (let [feed (transform-posts-into-a-feed database-results k > "discourse")] > (swap! recent-activity concat feed @recent-activity))))) > > and this function was suppose to put the content into an atom. At first I > did not use emit-str, but I added that on my last attempt to figure out > what is going on. > > I was using "conj", but then switched to "concat" when I switched to using > emit-str. > > This is the function that formed the XML: > > (defn transform-posts-into-a-feed [database-results k > what-type-of-item-is-this] > (let [site-url (make-site-url k) > map-of-xml-elements (reduce > (fn [feed db-row] > (conj feed > (xml/emit-str > (xml/element :item {} > (xml/element > :what-type-of-item-is-this {} (str what-type-of-item-is-this)) > (xml/element :username > {} (str (make-user-nice-name db-row))) > (xml/element > :user-profile-url {} (str (make-profile-url site-url (:profile_id > db-row)))) > (xml/element > :in-response-to-url {} (str (make-in-response-to-url site-url > what-type-of-item-is-this (:in_response_to_id db-row)))) > (xml/element :site-url > {} (str site-url)) > (xml/element :title {} > (str (:title db-row))) > (xml/element :item-url > {} (str (make-item-url site-url what-type-of-item-is-this (:id db-row)))) > (xml/element > :description {} (str (:description db-row))) > (xml/element :date {} > (str (:created_at db-row))) > )))) > [] database-results)] > map-of-xml-elements)) > > For awhile (before I used emit-str), at the end of this function, I had > this: > > (println (apply str map-of-xml-elements)) > > and I could see that the data in map-of-xml-elements was what I expected. > And yet the atom "recent-activity" seemed to remain empty, which I found > very confusing. > > The function that basically drives this app (called from main) is the one > that throws an error: > > (defn iterate-through-sites-and-output-files [] > "2012-11-10 - The TMA server might have 10 or 20 or more websites, each > with their own database config. We need to update every site. > fg/database-connections holds a map where the key is the name of the site > and the value is another map that has all of the info needed to connect to > that sites database." > (println "Now we will iterate over the sites again. The time is: " > (das/current-time-as-string)) > (doseq [[k db] @fg/database-connections] > (println (apply str " We will now update " (str k))) > (ura/update-recent-activity-from-this-site db k) > (println (mem/show-stats-regarding-resources-used-by-this-app)) > (println (apply str (debug/thread-top))) > (println " We are processing " (str k)) > (println "This is what we currently have stored up as recent activity: > ") > (println "We will now wait 1 hour, then iterate over all of the sites > again. The time is: " (das/current-time-as-string)) > (. java.lang.Thread sleep 3600000) > (iterate-through-sites-and-output-files)) > > The error I get: > > Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Unknown Source) > at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source) > at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown > Source) > at java.lang.AbstractStringBuilder.append(Unknown Source) > at java.lang.StringBuilder.append(Unknown Source) > at clojure.core$str$fn__3501.invoke(core.clj:500) > at clojure.core$str.doInvoke(core.clj:502) > at clojure.lang.RestFn.applyTo(RestFn.java:139) > at clojure.core$apply.invoke(core.clj:600) > at > recent_activity.core$iterate_through_sites_and_output_files.invoke(core.clj:33) > at clojure.lang.AFn.run(AFn.java:24) > at java.lang.Thread.run(Unknown Source) > > line 33 is: > > (println "We will now wait 1 hour, then iterate over all of the sites > again. The time is: " (das/current-time-as-string)) > > That line looks innocent and it did not cause a problem before. It just > stared causing a problem when I starting using emit-str to store the XML as > a string in the atom recent-activity. > > That leads to 2 questions: > > 1.) are strings expensive on memory? > > 2.) what are the simplest profiling tools I can use to compare the memory > use of emit-str versus what I was doing previously? > > I am giving up on the use of emit-str and I'm going to try a different > approach. But I would be grateful for any insights about why I might have > gotten the OutOfMemory error. > > > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- Matt Hoyt -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
