Paul, I can understand the concerns about memory footprint if you work in a restricted environment (lack of physical memory or other system resources) or if your memory load is very high.
However, these things are less common these days with the kind of hardware we can buy compared to what we could get for the same amount 8 years ago. Assembly languages or C appear much more economical than Java ... but nobody would suggest using these as generic day to day tools. It's more practical and efficient to use Java as an alternative to these things (even if sometimes I miss Cobol and Fortran :))). I am not sure I would bother now with the "real cost" of a PersistentList versus an ArrayList. If you look at the JVM and Java, Java 1.2 is a lot different internally than Java 5 or 6. Performance wise it's night and day. In the near future, Java, the JVM, the hardware, can improve, ... so there's still some performance gains to expect in the bottom layers that will eventually benefit Clojure. The same way we find out that keeping references on objects is bad in Java, we will learn the things we should avoid when coding in Clojure (and maybe write a cookbook about these things eventually :)))). In the short term that should be enough to avoid most problems. We are now facing some visible hardware limits. We need to change the way we design applications to adapt to the twists the hardware industry is using to pack more power in the same space. Clojure brings into play concepts that may be our salvation but we need to rethink the way we create software. There's much work to do in the design area... I think we can postponed the data structure footprint issue until we assert that the trade off (memory versus programing power) of these representations is paying off or not. What do you think ? Luc On Mon, 2009-02-02 at 13:48 -0800, Keith Bennett wrote: > Paul - > > Clojure definitely has its benefits, but in terms of memory footprint, > Java appears to be *much* more economical, unless elements can be > discarded shortly after use as Christian describes, in which case it's > much *less* economical. > > In a Java ArrayList, only a single ArrayList object is used to store > all n elements added. The objects are stored in an internal Object > []. In the test, the 100,000 elements resulted in an increase of > about 1,000,000 bytes for the object array, a cost of 10 bytes each. > (Actually, probably less than 10, since ArrayList.ensureCapacity() > increases the Object [] by 50% each time it needs increasing, so there > is probably a significant amount of unused capacity in that 1,000,000 > bytes.) > > In contrast, Clojure creates a new holder object for each item added, > at a cost of 48 bytes per item for the PersistentList objects, and 52 > bytes per item for the lazy cons. > > So the overhead of Clojure appears to be 5 times the overhead of Java, > for lists that need to stay around in memory for more than a single > operation. > > I don't mean to imply that Java is better than Clojure; in the vast > majority of cases this extra memory consumption will not be a problem, > and I am really enjoying working with Clojure, not only for the way > functional programming is stretching my mind, but also for the > productivity improvements I anticipate getting from adding this tool > to my toolkit. However, I want to be objective and scientific about > the relative costs of the different approaches so that decisions I > make are made in an informed and objective way. > > The Java code I used to test this is at http://www.pastie.org/377716. > > - Keith > > > On Feb 2, 1:41 pm, Paul Barry <pauljbar...@gmail.com> wrote: > > Ketih, > > > > I think what you have done, at least at the JVM level, is create 100,000 > > lists, with basically each list containing an element and a pointer to the > > next element. Because one list points to the next, none of them can be > > garbage collected. It seems to me that this would be roughly equivalent to > > one list with 100,000 elements, in terms of memory usage. In your map > > example, the memory usage is 52 * 100,000 bytes, so that's about 5MB. How > > much memory is used by the equivalent Java code? > > > > List l = new ArrayList(); > > for(int i = 0; i < 100000; i++) { > > l.add("x"); > > > > } > > > > On Mon, Feb 2, 2009 at 1:06 PM, Keith Bennett > > <keithrbenn...@gmail.com>wrote: > > > > > > > > > All - > > > > > I'm curious to know how to minimize memory consumption in Clojure for > > > large lists. I did the following test in repl: > > > > > (def d ()) > > > (dotimes [_ 100000] (def d (cons "x" d))) > > > > > Then, I used VisualVM, an awesome free profiling tool (https:// > > > visualvm.dev.java.net/) to examine the results. It indicated that the > > > number of clojure.lang.PersistentList instances increased by 100,000. > > > Each instance appeared to consume 48 bytes (not including the actual > > > value, but only its reference, I presume). I don't think any were > > > eligible for garbage collection, because I initiated gc several times, > > > and they were not removed. (I know that gc() is not deterministic, > > > but am pretty sure that they would have been removed. Feel free to > > > correct me.) > > > > > Thinking that maybe the special functions like map did some magic to > > > save memory, I tried this: > > > > > (def a (map #(* 2 %) (range 100000))) > > > > > The result was 100,000 clojure.lang.LazyCons objects, each of which > > > consuming 52 bytes. > > > > > Are there alternate strategies for building a large sequence that > > > would consume less memory per element? > > > > > Thanks, > > > Keith > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
