Re: [OpenDX2-users] dx_memory
Hi, Matyas - I'm afraid when we first developed DX in the late '80s to early '90s we didn't anticipate the move to 64 bit architectures and built a 2G limit into DX, which you are now hitting. A few years ago I built a version which uses 64 bit addresses, so arenas can grow past that limit, but since DX does a lot of address arithmetic in 32 bit integers, no single object (that is, Array) can be larger than 2G. While it would be possible to find and fix all the places that happens, it'd be very painful and error-prone. If you'd like to try the version I 'fixed', its on SourceForge - google SourceForge OpenDX. It might work for you; I'd suggest you partition the data immediately after loading it. It also uses threading, rather than multiple processes, to take advantage of all your processor cores. Alternatively, there are some pretty good alternatives; personally, I use Paraview, but VisIt (google LLNL visit) has believers as well. Both will take advantage of distributed systems, spreading the computation across multiple nodes using MPI. Good luck! Greg From: opendx2-users-boun...@lists.berlios.de [mailto:opendx2-users-boun...@lists.berlios.de] On Behalf Of herein mátyás Sent: Monday, March 03, 2014 9:05 AM To: opendx2-users@lists.berlios.de Subject: [OpenDX2-users] dx_memory Dear dx Users, I have very large dx files (~800 MB; output from a mantle convection code) and I tried to use opendx to visualize the 3d temeprature field. But I could not since there is a memory error. The communication window says: out of memory reached limit in large arena. I have 32 GB of rams in my workstation and I also ran the dx with the command flag -memory 32000 but the large item in the arena never exceeds the 2 GB (1920 MB - exactly). I think there is a memory bug. Is there any other options to tell dx to use more memory than 2 GB ? Any suggestions are very welcome. This project is a part of my Phd. thanks; Matyas. -- Herein Matyas ELTE TTK Geofizikai es Urtudomanyi Tanszek Eotvos University Department of Geophysics and Space Sciences 1117 Budapest, Pázmány Péter sétány 1/C http://geophysics.elte.hu/ ___ OpenDX2-users mailing list OpenDX2-users@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opendx2-users
Re: [OpenDX2-users] OpenDX development
Hi, Marco - I'll be distributing the code I've been working on - I just want to do a bit more testing. I think I'll be using a SourceForge repository. The changes I've been making affect the standard shared-memory model of OpenDX. The work I did a long time ago on an MPI-based distributed-memory version was built on top of the shared-memory model; whether that is multi-process or multi-thread should be immaterial. However, I'm not sure I'm going to try to resurrect that stuff; it was only a prototype and never really became real. I don't believe the system has been flushing the cache at all, in the past. Thats why it would run until it ran out of memory, take a few half-hearted efforts at scavenging, and then stop. At which point you could explicitly flush the cache, and it'd proceed along just fine - until it ran out of memory again. I have a test progeam that steps up and back through 1000 isovalues; the old code would not run very long before stopping with an out-of-memory error. With the fix in, even in very constrained memory it keeps running a *long* time. Greg Gregory D. Abram Texas Advanced Computing Center JJ Pickle Research Campus - ROC 1.101 Bldg 196 10100 Burnet Road Austin, Texas 78758-4497 g...@tacc.utexas.edu (512) 471-8196 From: Marco Morandini [morand...@aero.polimi.it] Sent: Monday, October 26, 2009 5:49 AM To: Greg Abram Cc: opendx2-users@lists.berlios.de Subject: Re: [OpenDX2-users] OpenDX development Greg Abram wrote: Hi - Anyone out there? I'm wondering if anyone is interested in some work I've been doing on OpenDX - I've replaced the multi-process model with a multi-threaded one, enabling a much (much!) simplified memory manager. This enables me to remove the limit on memory size (though I bet you'd better not allocate any individual object larger than 2GB). It'll just consume virtual-memory page-space until it hits an arbitrary limit (suggestions?) when it will begin to flush the cache. Fragmentation is also much less of a problem. It would be great if you could post the patches, so that they remain archived somewhere, waiting for someone to re-vitalize the project. Do the multi-thread model rules out the possibility to use mpi? I'm also very interested in the cache bug you mentioned in the forum. Thanks, Marco ___ OpenDX2-users mailing list OpenDX2-users@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opendx2-users
[OpenDX2-users] OpenDX development
Hi - Anyone out there? I'm wondering if anyone is interested in some work I've been doing on OpenDX - I've replaced the multi-process model with a multi-threaded one, enabling a much (much!) simplified memory manager. This enables me to remove the limit on memory size (though I bet you'd better not allocate any individual object larger than 2GB). It'll just consume virtual-memory page-space until it hits an arbitrary limit (suggestions?) when it will begin to flush the cache. Fragmentation is also much less of a problem. Gregory D. Abram Texas Advanced Computing Center JJ Pickle Research Campus - ROC 1.101 Bldg 196 10100 Burnet Road Austin, Texas 78758-4497 g...@tacc.utexas.edu (512) 471-8196 ___ OpenDX2-users mailing list OpenDX2-users@lists.berlios.de https://lists.berlios.de/mailman/listinfo/opendx2-users
