Hi Rob -- > What, if any, mechanisms exist in Chapel for structured I/O ?
Assuming I'm interpreting your question correctly, my overarching thought on this question is that I believe Chapel has a number of the key mechanisms that one would want/need to do structured I/O, but that we haven't yet done a good job of abstracting them into a productive interface for the benefit of both end- and expert-users. In particular, we have: * domain maps, which support the mapping of index sets or arrays to units of architectural locality (typically compute nodes/memories, but this could be extended to other resources as well, if desired, like disks); * support for distinct concepts for files vs. channels such that a single logical file can support multiple channels for parallel I/O to distinct parts of the file; * initial support for distributed file systems like HDFS and Lustre, and the runtime architecture to support plugging in other similar technologies (these are "initial" in the sense that they haven't had much use yet). So my hypothesis is that a sufficiently motivated user could use these features to perform structured I/O in a reasonable way, but that it would be lower-level than a typical end-user might want -- if you'll forgive the Argonne-oriented analogy, it'd be more like doing MPI-level programming in Chapel than the global-view multiresolution programming that Chapel aspires to. Instead, I'd really like to have a story in which one could trivially express things like "blast this distributed array out to the file system in a way that makes best use of the resources and parallelism available without burdening me with the deatils." And then to permit expert users to plug in their own underlying policies for expressing that scheduling, locality, and partitioning in a way that requires the end user to change little-to-no code in logical places (like the declarations of the files, channels, or arrays). This would continue the theme that Chapel has Part of the reason we haven't pursued this is a lack of deep experience in this area within our immediate team. I visited and spoke with Rob Ross many years ago about constructing a better story for structured I/O and distributed file systems in Chapel, but that never led anywhere due to a lack of time and funding to pursue it (on both sides). If you think things have changed enough that there might be a collaboration to be pursued here, I think that it continues to be of great interest to us. Best wishes, -Brad ------------------------------------------------------------------------------ Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho _______________________________________________ Chapel-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-users
