Kor, > I work on a modelling framework which is used by domain experts like > hydrologists and soil scientists to build forward simulation models. HPX > is used in the implementation to distribute computations over CPU cores > and compute nodes. > > Typically, large amounts of data are being read and written, not only at > the start and end of a simulation, but also while iterating through time. > We mostly work with gridded data (rasters). > > I/O has become a performance and scalability bottleneck and I am looking > at ways to solve this. Together with a parallel I/O expert I have started > working on combining AMT / HPX with parallel I/O. Technically this works > now, using HDF5 as a data format, and on a GPFS file system. > > Our current approach is quite simple. Instead of performing computations > on partitions of data in HPX threads, we perform (independent) I/O. We > plan on performing performance and scalability experiments in the near > future. > > To prevent that we miss something important I would like to ask if you > have information that is related specifically to *combining AMT / HPX with > parallel I/O*, for example:
The usual caveats apply whenever you run additional threads (possibly unknown) to HPX. I'd try dedicating specific cores to those threads while keeping HPX threads off of those cores. > - HPX facilities that are useful in this context HPX thread affinities, HPX executors. > - Lessons learned doing something similar > - Pointers to code that does something similar Alireza (cc'ed) is working on parallel I/O using HPX so he might have additional pointers. Thanks! Regards Hartmut --------------- https://hpx.stellar-group.org https://github.com/STEllAR-GROUP/hpx _______________________________________________ hpx-users mailing list [email protected] https://mail.cct.lsu.edu/mailman/listinfo/hpx-users
