Hello everyone! I would like to reassure you that all is right in the world. After a large number of tests I finally tracked the problem down to an entry in a Hashtbl not being deleted. It was a one line fix!
One question does remain though: In my tests I would do some work that would cause ~1GB of RAM to be under control of the Gc. Then I would do something that, at the time I didn't understand, would case the Gc to compact all of its memory and go back down to less than 1 meg, but the RES value in top would only drop to about 400 megs. Is this expected behavior? I know the malloc implementation might hold on to some data for itself but 400x the amount of memory the Ocaml RTS actually needs seems a bit excessive. I know there is a bug report floating around from Martin about large Arrays not being properly freed, in this case my issue was with a Hashtbl. I do not know if Hashtbl is implemented with an Array underneath, but could that be the cause of my overhead if so? Thank you On Jan 1, 2012, at 7:44 AM, Richard W.M. Jones wrote: > On Sat, Dec 31, 2011 at 10:33:19AM -0500, [email protected] wrote: >> Being on the C side is not even something I had considered. In this >> case, I think the only piece of code not part of the Ocaml RTS that is >> talking to C is Lwt. It is possible that there is a memory leak in >> there somewhere. The upside, though, is there seems to be some >> residue of it in the Ocaml side. My heap numbers given earlier are >> ~65megs which is significantly larger than it should be, so I might be >> able to track it down from the Ocaml side. > > A couple of other ideas: > > Is compaction disabled? lablgtk disables it unconditionally by > setting the global Gc max_overhead (see also the Gc documentation): > > src/gtkMain.ml: > let () = Gc.set {(Gc.get()) with Gc.max_overhead = 1000000} > > If something in your program or Lwt does the same, you may get > fragmentation of the C malloc heap or perhaps the OCaml heap. I've > experienced fragmentation in very long-running C programs and it's > insidious because it's very hard to understand what's really going on, > and impossible IME to remedy it. > > Second suggestion is to look at /proc/<pid>/maps and/or smaps. > That'll tell you without doubt where the 2GB of memory is being used. > Most likely in the heap from the way you describe it, but it is worth > checking that top isn't reporting something innocuous such as a big > file-backed mmap in one of your C libraries. > > Attached is a script that you can adapt to help you interpret > /proc/<pid>/maps. > > Rich. > > -- > Richard Jones > Red Hat > <maps.pl> -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
