Dear Kai and all, On Mon, 26 Oct 2009, Liu Kai wrote:
> I guess the reason may be from the pointer length difference between > 32 bit and 64 bit. Many pointers are stored in Elem data structure. So > this amount will double after switching 32 bit to 64 bit. Let me try to find out what this means in detail. For each Elem, we have: member of Elem 32bit 64bit ---------------------------------------------------------------- DofObject* DofObject::old_dof_object; 4 8 unsigned int DofObject::_id 4 4 unsigned short int DofObject::_processor_id 2 2 unsigned char DofObject::_n_systems 1 1 (padding) 1 1 unsigned char** DofObject::_n_v_comp 4 8 unsigned int** DofObject::_dof_ids 4 8 Node** Elem::_nodes 4 8 Elem** Elem::_neighbors 4 8 Elem* Elem::_parent 4 8 Elem** Elem::_children 4 8 unsigned char Elem::_rflag 1 1 unsigned char Elem::_pflag 1 1 unsigned char Elem::_p_level 1 1 subdomain_id_type Elem::_sbd_id 1 1 (padding) 0 4 ---------------------------------------------------------------- total 40 72 For each Node, we have: member of Node 32bit 64bit ---------------------------------------------------------------- DofObject* DofObject::old_dof_object; 4 8 unsigned int DofObject::_id 4 4 unsigned short int DofObject::_processor_id 2 2 unsigned char DofObject::_n_systems 1 1 (padding) 1 1 unsigned char** DofObject::_n_v_comp 4 8 unsigned int** DofObject::_dof_ids 4 8 T Node::_coords[LIBMESH_DIM] 24 24 ---------------------------------------------------------------- total 44 56 This looks like quite an amount of extra memory for 64-bit. Also, the _n_v_comp and _dof_ids members of both Elem and Node are arrays of pointers and hence store more pointers. _n_v_comp contains as many pointers as there are systems in the application (which I think is 8 in my case, i.e. additional 32 or 64 bytes of memory for 32 or 64bit mode, respectively). The same holds for _dof_ids. If we look only at the difference between 32bit and 64bit mode, we finally get 160 extra bytes of memory for each Elem and 140 extra bytes for each Node. In the first time step, I have 268152 Elem instances and 316744 Node instances. This results in a total surplus of 80913600 bytes (roughly 80 MB) in 64bit mode. Each cluster node has a total memory of 8 GB available, hence the difference between 32bit and 64bit should be within 1% of the total memory per thread. Hmm, this does not explain the behaviour I experienced. In my application, each thread now requires more than 30% of the available node memory (i.e. I cannot use more than 3 threads per node), where it was less than 16% when I ran in 32bit mode before my vacations (6 threads per node). (The total number of threads are unchanged.) Several thoughs come to my mind: 1. Did I miss something essential in my above calculation? 2. I now understand completely why you guys are working on the ParallelGrid so heavily. (-: 3. _n_v_comp should have identical entries for each DofObject in the mesh; why is it stored for each DofObject individually? (Not doing that would save nearly 40 MB per thread for my case.) 4. When I configured libMesh for 64bit mode, I did *not* do "--enable-ghosted". As far as I remember, this is the default now, or should I be a fool and remember this wrong? Anyway, libmesh_config.h contains "#define LIBMESH_ENABLE_GHOSTED 1", so this should be okay, shouldn't it? Anyway, I'm a little bit confused. Any suggestions are welcome. Best Regards, Tim -- Dr. Tim Kroeger [email protected] Phone +49-421-218-7710 [email protected] Fax +49-421-218-4236 Fraunhofer MEVIS, Institute for Medical Image Computing Universitaetsallee 29, 28359 Bremen, Germany ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Libmesh-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-users
