On 08/17/2012 01:51 PM, Kirk, Benjamin (JSC-EG311) wrote: >>> The data is split between many RB basis functions (about 1GB of data >>> split between ca 60000 files). So I was thinking the cause of the very >>> slow reading was overhead induced by the large number of individual >>> calls to read_serialized_data, and perhaps related memory management? >>> >>> Any ideas on this would be very appreciated! >> A lot of filesystem types do behave badly when you put too many >> entries in the same directory. How long does copying take if your >> "1GB of data" is in those 60000 separate files? That is what takes only a few seconds. >> On the other hand, we may have some performance bug. Can you stick >> logging into the read_serialized_data internals and figure out where >> the holdup is? > Or maybe if you could create a simple test case I can help profile it here > too. > > What is the average vector size? > > Note that the overhead of the system header and all the information in XDR > restart files will at some point become overwhelming. This usage case is > different than what was in mind for restart files - I at least always > thought of restarting an entire simulation, so there might be a lot of data > contained in a handful of vectors. There may be a better implementation for > this case... I'm doing some more detailed profiling now.
It's doubles stored in binary format, so the average vector/basis function size would be about 2000. Thanks for your help! Jens > > -Ben > > ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Libmesh-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-users
