Hi Rob and Brad, Sorry for the late (and perhaps misplaced) answer, but as I have been some work on Chapel IO I would like to comment on it...
On the easier to use side I have constructed on the overloading properties of Chapel to overload the = sign to do IO from a high level. In this way you can create a "file" object and assign an array to it, so it is written to the file easily, having a very clean, high level, source code. On the performance side, things are not so clean if you want to go to the limit. I have used gasnet directly to provide a very fast mechanism to transfer data between nodes, accessing the data right from the memory and not through the normal access to them. This implies diving in the Domain data structures to know how data is partitioned, and where it resides in memory, at the byte level. Also distributed file systems have been taken into account, writting data using aggregators and other algorithms to maximize bandwidth. I hope all this will be published soon, we are polishing the paper. If someone needs some clue (or code) about how all this can be done, you only need to ask me. Greets, Rafael > Hi Rob -- > > > What, if any, mechanisms exist in Chapel for structured I/O ? > Assuming I'm interpreting your question correctly, my overarching thought > on this question is that I believe Chapel has a number of the key > mechanisms that one would want/need to do structured I/O, but that we > haven't yet done a good job of abstracting them into a productive > interface for the benefit of both end- and expert-users. > > In particular, we have: > > * domain maps, which support the mapping of index sets or arrays to > units of architectural locality (typically compute nodes/memories, > but this could be extended to other resources as well, if desired, > like disks); > > * support for distinct concepts for files vs. channels such that a > single logical file can support multiple channels for parallel > I/O to distinct parts of the file; > > * initial support for distributed file systems like HDFS and Lustre, > and the runtime architecture to support plugging in other similar > technologies (these are "initial" in the sense that they haven't had > much use yet). > > So my hypothesis is that a sufficiently motivated user could use these > features to perform structured I/O in a reasonable way, but that it would > be lower-level than a typical end-user might want -- if you'll forgive the > Argonne-oriented analogy, it'd be more like doing MPI-level programming in > Chapel than the global-view multiresolution programming that Chapel > aspires to. > > Instead, I'd really like to have a story in which one could trivially > express things like "blast this distributed array out to the file system > in a way that makes best use of the resources and parallelism available > without burdening me with the deatils." And then to permit expert users > to plug in their own underlying policies for expressing that scheduling, > locality, and partitioning in a way that requires the end user to change > little-to-no code in logical places (like the declarations of the files, > channels, or arrays). This would continue the theme that Chapel has > > Part of the reason we haven't pursued this is a lack of deep experience in > this area within our immediate team. I visited and spoke with Rob Ross > many years ago about constructing a better story for structured I/O and > distributed file systems in Chapel, but that never led anywhere due to a > lack of time and funding to pursue it (on both sides). If you think > things have changed enough that there might be a collaboration to be > pursued here, I think that it continues to be of great interest to us. > > Best wishes, > -Brad > > > ---------------------------------------------------------------------------- > -- Comprehensive Server Monitoring with Site24x7. > Monitor 10 servers for $9/Month. > Get alerted through email, SMS, voice calls or mobile push notifications. > Take corrective actions from your mobile device. > http://p.sf.net/sfu/Zoho > _______________________________________________ > Chapel-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/chapel-users -- Rafael Larrosa Jiménez Centro de Supercomputación y Bioinformática - http://www.scbi.uma.es Universidad de Málaga EMAIL: [email protected] Edificio de Bioinnovación TELEF: + 34951952788 C/ Severo Ochoa 34 FAX : +34951952792 Parque Tecnológico de Andalucía 29590 Málaga (SPAIN) ------------------------------------------------------------------------------ _______________________________________________ Chapel-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-users
