if the file is stored in the PETSc format, you can use PETSc to pull the file in using MPIIO, which should be (hopefully) faster.
http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Viewer/PetscViewerBinarySetMPIIO.html Feel free to pop these questions to petsc-users (cc'd), there are more of them and they usually know more than me :) -Aron On Sat, Jan 8, 2011 at 2:21 PM, Zuhair Khayyat <zuhair.khayyat at kaust.edu.sa>wrote: > Dear Aron, > > Currently I am using PETSc for my research to implement a graph mining > algorithm on a large cluster, and I would like ask you some questions due to > your experience with this tool. > > Have you ever tried to optimize allocating a very large matrix from input > file in parallel? I have a very large file (around 10 GB) and it takes too > long to allocate the matrix through the main node. I am planning to > implement a distributed parallel file reader that split the original file > and make each node reads it separately into a common matrix. > > Is there and other tools that are comparable to PETSc in which they have > parallel file reader? > > Thank you for your help > > Regards, > Zuhair Khayyat > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110108/fa0c2ef2/attachment.htm>
