On Jan 9, 2011, at 1:45 AM, Zuhair Khayyat wrote:
> Dear Aron,
>
> Actually the file is an output of another application, and is stored in a
> simple text file. I have read about MPIIO, however couldn't figure out if it
> is a parallel file reader or not.
>
> Have you ever tried or worked on a parallel file reader to a common global
> matrix in PETSc, where each processor has part of the file? Thank you
If the file is ASCII there is no way to load it in efficiently if it is
large. Say more than a couple of megabytes. If it is ASCII and small just read
it in on process 0.
If it is large you will have to change the other application to save in a
binary format instead of ASCII.
Barry
>
> Regards,
> Zuhair Khayyat
>
> On Sat, Jan 8, 2011 at 8:42 PM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>
> wrote:
> if the file is stored in the PETSc format, you can use PETSc to pull the file
> in using MPIIO, which should be (hopefully) faster.
>
> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Viewer/PetscViewerBinarySetMPIIO.html
>
> Feel free to pop these questions to petsc-users (cc'd), there are more of
> them and they usually know more than me :)
>
> -Aron
>
>
> On Sat, Jan 8, 2011 at 2:21 PM, Zuhair Khayyat <zuhair.khayyat at
> kaust.edu.sa> wrote:
> Dear Aron,
>
> Currently I am using PETSc for my research to implement a graph mining
> algorithm on a large cluster, and I would like ask you some questions due to
> your experience with this tool.
>
> Have you ever tried to optimize allocating a very large matrix from input
> file in parallel? I have a very large file (around 10 GB) and it takes too
> long to allocate the matrix through the main node. I am planning to implement
> a distributed parallel file reader that split the original file and make each
> node reads it separately into a common matrix.
>
> Is there and other tools that are comparable to PETSc in which they have
> parallel file reader?
>
> Thank you for your help
>
> Regards,
> Zuhair Khayyat
>
>
