Donald,
We have our own C# wrapper for hdf. We saw the same error you site,
and we resolved it, maybe the cause is the same for you. We use the
.net code on Ubuntu, Windows, and Mac. This error arose only on
Windows and Mac and could be reliably replicated by doing reads from
multiple threads. Multi-threaded reading on Ubuntu worked without
error. Our fix was to use locks. So, we have a class:
private class ThreadLock
{
private ReaderWriterLockSlim threadLock = new
ReaderWriterLockSlim();
public void EnterReadLock() {threadLock.EnterWriteLock();}
public void ExitReadLock() {threadLock.ExitWriteLock();}
public void EnterWriteLock() {threadLock.EnterWriteLock();}
public void ExitWriteLock() {threadLock.ExitWriteLock();}
}
Notice, that both read and write locking invokes the write lock, which
for ReaderWriterLockSlim is exclusive. On Ubuntu, the read lock code
looks like:
public void EnterReadLock() {threadLock.EnterReadLock();}
public void ExitReadLock() {threadLock.ExitReadLock();}
which permits multi-threaded reading (read locks are not exclusive).
Then, hdf operations are wrapped as:
public bool StoreSequence(string path, double [] data)
{
bool ans;
threadLock.EnterWriteLock();
try {
ans = hdfFile.StoreVector(path, data);
hdfFile.Flush();
if (logger != null) {logger.DebugFormat("{0} =
StoreSequence({1}, double[{2}])", ans, path, data.Length);}
} finally {
threadLock.ExitWriteLock();
}
return ans;
}
Here hdfFile is an instance of our in-house hdf wrapper. The
StoreVector call makes all the hdf library calls one would expect and
because of the locks, they function correctly.
If you want any clarification, please feel free to ask. If this is not
helpful in your situation, I'd like to hear, as even though it fixed
our problems, it may not have been addressing the root cause (and we'd
like to get multithreaded reads back on Windows and Mac).
Matt
On Mon, Sep 19, 2011 at 5:22 PM, Donald Brandon <[email protected]> wrote:
> Yea, adding the appropriate close methods did not make a difference. Here
> is the loop I am running:
>
> foreach(String curGroup3Name in curGroup3Names)
> {
> H5FileId hdfFile = H5F.open(pathToHDF, H5F.OpenMode.ACC_RDONLY);
> H5DataSetId curDataset = H5D.open(hdfFile, curGroup1Name + "/" +
> curGroup2Name + "/" + curGroup3Name + "/dataset");
>
> float[] curDatasetArray = GetDatasetArray(curDataset);
>
> H5D.close(curDataset);
> H5F.close(hdfFile);
>
> MessageBox.Show("Finished dataset for tile " + curTileName + "...");
> }
>
> Just as in my other code, I get to about the seventh iteration and i hangs
> on the call to:
>
> H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));
>
> ...that happens inside the method getDatasetArray(curDataset). That code is
> listed below...
>
> Any ideas anyone?
>
> Thanks again,
> DB
>
> On 9/19/2011 3:31 PM, Donald Brandon wrote:
>
> Hey, Scott. Yea, I caught that after I sent the e-mail. I am writing a
> test program now. I will let you know if that solved it.
>
> Thanks,
> DB
>
> On 9/19/2011 3:27 PM, Mitchell, Scott - IS wrote:
>
> Hey Donald,
>
>
>
> You need to be close()ing your H5T type objects. I’m not sure if that’s it,
> but….
>
>
>
>
>
> Scott
>
>
>
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Donald Brandon
> Sent: Monday, September 19, 2011 3:22 PM
> To: HDF Users Discussion List
> Subject: [Hdf-forum] memory issues when reading multiple datasets
>
>
>
> Hello all:
>
> I have another problem that I could use some help with. In a nutshell, I
> have multiple datasets that I am trying to read arrays from and then work
> with those arrays one by one. The structure of the .h5 file kinda look like
> this:
>
> Group<root>
> Group<Group1>
> Group<GroupA>
> Group<Group_a>
> Dataset<1>
> Group<group_b>
> Dataset<1>
> Group<GroupB>
> Group<Group_a>
> Dataset<1>
> Group<group_b>
> Dataset<1>
> Group<Group2>
> Group<Group_a>
> Dataset<1>
> Group<group_b>
> Dataset<1>
> Group<GroupB>
> Group<Group_a>
> Dataset<1>
> Group<group_b>
> Dataset<1>
>
> I am recursively going through each group and then executing code that looks
> something like this:
>
> public float[] GetDepthArray(H5DataSetId dataset)
> {
> H5DataTypeId datasetType = H5D.getType(dataset);
> H5T.H5TClass datasetClass = H5T.getClass(datasetType);
> H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType,
> H5T.Direction.ASCEND);
>
> long storageSZ = H5D.getStorageSize(dataset);
>
> float[] dArray = new float[storageSZ];
>
> H5D.read(dataset, datasetNativeType, new
> H5Array<float>(dArray));
>
> return dArray;
> }
>
> After about the seventh iteration, the code will fail with an
> AccessViolationException - "Attempted to read or write protected memory.
> This is often an indication that other memory is corrupt."
>
> Anyone have advice as to whether or not I should be cleaning something up
> after each iteration that I am not seeing? My system RAM seems to be
> un-phased after each iteration so at least it doesn't look like I am losing
> memory anywhere...
>
> Thanks for the help!
> DB
>
>
>
> ________________________________
> This e-mail and any files transmitted with it may be proprietary and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this e-mail in error please notify the
> sender.
> Please note that any views or opinions presented in this e-mail are solely
> those of the author and do not necessarily represent those of ITT
> Corporation. The recipient should check this e-mail and any attachments for
> the presence of viruses. ITT accepts no liability for any damage caused by
> any virus transmitted by this e-mail.
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
