Hi Mohamad,
thanks again for your help! Once the program failed, the structure was
corrupted and h5dump was unable to read the file. Any other file which
was also opened at the time but not by the crashing node survived the
crash intact. I have rewritten the dumping part and moved from multiple
classes writing to the same file to having one class that stores all
variables and dumps on demand which appears to work better (tested with
192 nodes). In the preceding approach, I opened and closed the file a
lot of times to add just one variable or an array of variables, may be
this was the problem? Although I still find it weird that it worked
flawlessly on my personal computer ...
Peter
On 11/20/2012 03:29 PM, Mohamad Chaarawi wrote:
> Hi Peter,
>
> Yes nothing seems unusual for me in the case that each process
> accesses its own file.
> Since you mentioned that this works on your local filesystem, did you
> try and check the structure of the files and make sure that they are
> correct their (using h5dump or other tools)?
> Otherwise I'm not sure what could be wrong. I'm not familiar with C++
> either, so someone else could also have other comments.
>
> Mohamad
>
> On 11/19/2012 11:11 AM, Peter Boertz wrote:
>> Hi Mohamad,
>>
>> thanks for your reply. The reason I suspected Lustre of being the
>> culprit is simply that the error does not appear on my personal
>> computer. I thought that maybe the files are written/opened too fast or
>> too many at the same time for the synchronization process of Lustre to
>> handle.
>>
>> I am inserting various pieces of code that show how I am calling the
>> HDF5 library. Any comment on proper ways of doing so is much
>> appreciated!
>>
>> To open the file, I use the following code:
>>
>>
>> ==================================================================
>>
>> int H5Interface::OpenFile (std::string filename, int flag) {
>>
>> bool tried_once = false;
>>
>> struct timespec timesp;
>> timesp.tv_sec = 0;
>> timesp.tv_nsec = 200000000;
>>
>> for (int tries = 0; tries < 300; tries++) {
>> try {
>> H5::Exception::dontPrint();
>> if(flag == 0) {
>> file = H5::H5File (filename, H5F_ACC_TRUNC);
>> } else if (flag == 1) {
>> file.openFile(filename, H5F_ACC_RDONLY);
>> } else if (flag == 2) {
>> file.openFile(filename, H5F_ACC_RDWR);
>> }
>>
>> if (tried_once) {
>> std::cout << "Opening " << filename << " succeded
>> after "
>> << tries << " several tries" << std::endl;
>> }
>> return 0;
>>
>> } catch( FileIException error ) {
>> tried_once = true;
>> }
>>
>> catch( DataSetIException error ) {
>> tried_once = true;
>> }
>>
>> catch( DataSpaceIException error ) {
>> tried_once = true;
>> }
>> nanosleep(×p, NULL);
>> }
>> std::cerr << "H5Interface:\tOpening " << filename << " failed";
>> return -1;
>> }
>>
>> It often happens that opening a file succeeds only after 1 or 2 tries.
>>
>> I write and read strings like this:
>>
>>
>> ==================================================================
>>
>> int H5Interface::WriteString(std::string path, std::string value) {
>> try {
>> H5::Exception::dontPrint();
>> H5::StrType str_t(H5::PredType::C_S1, H5T_VARIABLE);
>> H5std_string str (value);
>> hsize_t dims[1] = { 1 };
>> H5::DataSpace str_space(uint(1), dims, NULL);
>> H5::DataSet str_set;
>> if (H5Lexists(file.getId(), path.c_str(), H5P_DEFAULT)) {
>> str_set = file.openDataSet(path);
>> } else {
>> str_set = file.createDataSet(path, str_t, str_space);
>> }
>> str_set.write (str, str_t);
>> str_set.close();
>> }
>> catch( FileIException error ) {
>> // error.printError();
>> return -1;
>> }
>>
>> catch( DataSetIException error ) {
>> // error.printError();
>> return -1;
>> }
>>
>> catch( DataSpaceIException error ) {
>> // error.printError();
>> return -1;
>> }
>> return 0;
>> }
>>
>>
>> ==================================================================
>>
>>
>> int H5Interface::ReadString(std::string path, std::string * data) {
>> try {
>> H5::Exception::dontPrint();
>> if (H5Lexists(file.getId(), path.c_str(), H5P_DEFAULT)) {
>> H5::StrType str_t(H5::PredType::C_S1, H5T_VARIABLE);
>> H5std_string str;
>> H5::DataSet str_set = file.openDataSet(path);
>> str_set.read (str, str_t);
>> str_set.close();
>> *data = std::string(str);
>> }
>> }
>> catch( FileIException error ) {
>> // error.printError();
>> return -1;
>> }
>>
>> catch( DataSetIException error ) {
>> // error.printError();
>> return -1;
>> }
>>
>> catch( DataSpaceIException error ) {
>> // error.printError();
>> return -1;
>> }
>> return 0;
>> }
>>
>>
>>
>> And finally for writing and reading boost::multi_arrays, for example:
>>
>>
>>
>> ==================================================================
>>
>>
>> int H5Interface::Read2IntMultiArray(std::string path,
>> boost::multi_array<int,2>& data) {
>> try {
>> H5::DataSet v_set = file.openDataSet(path);
>> H5::DataSpace space = v_set.getSpace();
>> hsize_t dims[2];
>>
>> int rank = space.getSimpleExtentDims( dims );
>>
>> DataSpace mspace(rank, dims);
>> int data_out[dims[0]][dims[1]];
>> data.resize(boost::extents[dims[0]][dims[1]]);
>> v_set.read( data_out, PredType::NATIVE_INT, mspace, space );
>> for (int i = 0; i < int(dims[0]); i++) {
>> for (int j = 0; j < int(dims[1]); j++) {
>> data[i][j] = data_out[i][j];
>> }
>> }
>> v_set.close();
>> }
>> [...]
>>
>>
>> ==================================================================
>>
>>
>> int H5Interface::WriteIntMatrix(std::string path, uint rows,
>> uint cols, int * data) {
>> try {
>> H5::Exception::dontPrint();
>> hsize_t dims_m[2] = { rows, cols };
>> H5::DataSpace v_space (2, dims_m);
>> H5::DataSet v_set;
>> if (H5Lexists(file.getId(), path.c_str(), H5P_DEFAULT)) {
>> v_set = file.openDataSet(path);
>> } else {
>> v_set = file.createDataSet(path, H5::PredType::NATIVE_INT,
>> v_space);
>> }
>> v_set.write(data, H5::PredType::NATIVE_INT);
>> v_set.close();
>> }
>> [...]
>>
>>
>>
>> As far as the workflow goes, a scheduler provides the basic h5 file with
>> all the parameters and tells the workers to load this file and then put
>> their measurements in. So they are enlarging the file as time goes by.
>>
>> Have a nice day, Peter
>>
>>
>>
>> On 11/19/2012 03:36 PM, Mohamad Chaarawi wrote:
>>> Hi Peter,
>>>
>>> The problem does sound strange.
>>> I do not understand why file locking helped reduce errors. I though
>>> you said each process writes to its own file anyway, so locking the
>>> file or having one process manage the reads/writes should not matter
>>> anyway.
>>>
>>> Is it possible you could send me a piece of code from your simulation
>>> that is performing I/O, that I can look at and diagnose further?
>>> A program that I can run and replicates the problem (on Lustre) would
>>> be great. If that is not possible, then please just describe or
>>> copy-paste how you are calling into the HDF5 library for your I/O.
>>>
>>> Thanks,
>>> Mohamad
>>>
>>> On 11/18/2012 10:24 AM, Peter Boertz wrote:
>>>> Hello everyone,
>>>>
>>>> I run simulations on a cluster (using OpenMPI) with a Lustre
>>>> filesystem
>>>> and I use HDF5 1.8.9 for data output. Each process has its own
>>>> file, so
>>>> I believe there is no need for the parallel HDF5 version, is this
>>>> correct?
>>>>
>>>> When a larger number (> 4) processes want to dump their data at the
>>>> same
>>>> time, I get various errors of paths and objects not found or any other
>>>> operation failing. I can't really make out the reason for it, as the
>>>> code works fine on my personal workstation and runs for days with
>>>> writes
>>>> / reads every 5 minutes without failing.
>>>>
>>>> What I have tried so far is having one process manage all the
>>>> read/write
>>>> operations so that all other processes have to check whether anyone
>>>> else
>>>> is already dumping their data. I also implemented
>>>> boost::interprocess:file_lock to prevent writing in the same file,
>>>> which
>>>> is however excluded by the queuing system anyway, so this was more
>>>> of a
>>>> paranoid move to be absolutely sure. All that helped reducing the
>>>> number
>>>> fatal errors significantly, but did not completely get rid of them.
>>>> The
>>>> biggest problem is, that some of the files get corrupted when the
>>>> program crashes which is especially inconvenient.
>>>>
>>>> My question is, if there is any obvious mistake I am making and how I
>>>> would go about solving this issue. My initial guess is that the Lustre
>>>> filesystem plays some role in this, since it is the only difference to
>>>> my personal computer where everything runs smoothly. As I said,
>>>> neither
>>>> the errors messages nor the traceback show any consistency.
>>>>
>>>> bye, Peter
>>>>
>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> [email protected]
>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
