Hi Marty Barnaby wrote: > To return to this discussion, in recent testing, I have found that > writing to a Lustre FS via a higher level library, like PNetCDF, fails > because the default for value for romio_ds_write is not disable. This > is set in the mpich code in the file /src/mpi/romio/adio/common/ad_hints.c You can use MPI_Info_set to disable romio_ds_write. What is the fail? flock? since data-sieving need flock. > > I believe it has something to do with locking issues. I'm not sure how > best to handle this, I'd prefer the data sieving default be disable, > though I don't know all the implications there. I agree data sieving should be disable. And also it check the contiguous buftype or filetype only by fileview, which is not enough sometimes, and trigger unnecessary read-modify-write even for contiguous write(especially for those higher level library, if you choose collective write). Since lustre has client cache and also the overhead of flock and read-modify-write, so I doubt the performance improvements we could get from data-sieving on lustre, although I do not have performance data to prove that. > Maybe an ad_lustre_open should be a place where the _ds_ hints are > set to disable. Yes, we should disable this for stride write in lustre. ad_lustre_open seems a right place to do this.
Thanks WangDi > > Marty Barnaby > > > Weikuan Yu wrote: >> Andreas Dilger wrote: >> >>> On Mar 11, 2008 16:10 -0600, Marty Barnaby wrote: >>> >>>> I'm not actually sure what ROMIO abstract device the multiple CFS >>>> deployments I utilize were defined with. Probably just UFS, or maybe NFS. >>>> Did you have a recommended option yourself. >>>> >>> The UFS driver is the one used for Lustre if no other one exists. >>> >>> >>>> Besides the fact that most of the adio that were created over the years are >>>> completely obsolete and could be cleaned from ROMIO, what will the new one >>>> for Lustre offer? Particularly with respect to controls via the lfs utility >>>> that I can already get? >>>> >>> There is improved collective IO that aligns the IO on Lustre stripe >>> boundaries. Also the hints given to the MPIIO layer (before open, >>> not after) result in lustre picking a better stripe count/size. >>> >>> >> >> In addition, the one integrated into MPICH2-1.0.7 contains direct I/O >> support. Lockless I/O support was purged out due into my lack of >> confidence in low-level file system support. But it can be revived when >> possible. >> >> -- >> Weikuan Yu <+> 1-865-574-7990 >> http://ft.ornl.gov/~wyu/ >> >> > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss > -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
