David, > ...the part that is being worked on in memory is, according to my > estimates a maximum of 300MiB. The rest, some 1800MiB (max) should be > pushed into swap or onto the disk for retrieval in a few seconds, (the > processor should be done with the other section by then). The method > of access is guaranteed as sequential in relation to the sections to > process. A section is not bigger then an array of size 2000*2000*24. > Though only about 2000*2000*4 or 5 is expected to be non-NULL at any > one given time.
OK. Well you might be out of luck two ways: - libJudy has no internal or inherent capability to swap out part of an array to disk. All you get is whatever your underlying OS provides for virtual memory. And of course any time you have a disk cache miss and have to go out to disk, you are looking at huge (orders of magnitude) delays compared to the performance built into libJudy when the data is in memory. - libJudy has no special treatment of its memory to group together related index extents (internal tree nodes). It gets memory from plain old malloc() in small chunks, and leaves further optimization to the malloc() library of your choice. A strength of libJudy is that adjacent populated subexpanses are nearby within each tree node, but sibling and child nodes don't necessarily have any good (or bad) relationship within the heap. So even sequential insertion followed by retrieval MIGHT not result in efficient swap patterns, depending on your malloc()'s behavior. We did think about both of these concerns, but had limited time and resources to do more than try to perfect the in-memory library before HP canceled the project and product in 2002 and we open-sourced it. Sorry... Alan Silverstein ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Judy-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/judy-devel
