2016-09-22 21:27 GMT+02:00 Werner Benger <wer...@cct.lsu.edu>:
> On 22.09.2016 20:34, Elvis Stansvik wrote:
>
>> 2016-09-22 20:25 GMT+02:00 Werner Benger <wer...@cct.lsu.edu>:
>>>
>>> There was some recent discussion that only the C API of HDF5 is
>>> threadsafe,
>>> but not the C++ layer on top of it. To be safe you should probably keep
>>> at
>>> the C API.
>>
>> Aha. Thanks for the heads up Werner.
>>
>> The FAQ has this to say:
>>
>>      "By default, you cannot build either Parallel HDF5 with C++ or
>> Parallel HDF5 with the thread-safe feature. You will receive a
>> configure error if you try either of these combinations."
>>
>> But it does not say that C++ with the thread-safe is unsupported (when
>> using serial HDF5). Maybe the FAQ should be updated?
>>
>> This is quite unfortunate since our app is C++ and the C++ API is so
>> much more convenient, but I guess I'll port our code to the C API
>> (it's not that much code).
>
> What you probably could do as well is to have your own lock with one global
> mutex around each call to the C++ API. HDF5 itself allows only one running
> thread internally as well and just puts such a lock inside the API, so it
> would come to the same.

This is what I ended up doing, so thanks a lot for the suggestion.

It was much easier than I thought because, since I was using Qt
anyway, I could just create a global QMutex, and then create a
QMutexLocker (RAII-style mutex locker) on the stack in each function
(just two of them) where I use HDF5.

This sort of coarse locking is OK for me since overwhelming majority
of time is spent in a HDF5 H5::DataSpace::read(..) call (I'm reading
big compressed data). The other HDF5 calls are very quick by
comparison.

Elvis

>
>              Werner
>
>
>
>> So far we haven't had any problems under Ubuntu (where the package is
>> built with --enable-threadsafe --enable-unsupported), but I guess
>> we've just been lucky.
>>
>> Elvis
>>
>>> Cheers,
>>>
>>>             Werner
>>>
>>>
>>>
>>> On 22.09.2016 20:15, Elvis Stansvik wrote:
>>>>
>>>> 2016-09-22 19:58 GMT+02:00 Scot Breitenfeld <brtn...@hdfgroup.org>:
>>>>>
>>>>> Yes it is still the case that you cannot enable C++ or Fortran (or the
>>>>> High Level APIs) when threadsafe is enabled. —enable-unsupported can
>>>>> override this behavior.
>>>>
>>>> Aha, so that's why I had to enable that option :)
>>>>
>>>> I see now that this is what the Ubuntu package does. I'll ask the Arch
>>>> package maintainer if he's willing to do the same.
>>>>
>>>> I've confirmed now that I don't have any problems with my program if I
>>>> re-built the Arch package with --enable-threadsafe
>>>> --enable-unsupported.
>>>>
>>>> Thanks for the info!
>>>>
>>>> Elvis
>>>>
>>>>> Scot
>>>>>
>>>>>
>>>>>> On Sep 22, 2016, at 12:36 PM, Elvis Stansvik
>>>>>> <elvis.stans...@orexplore.com> wrote:
>>>>>>
>>>>>> 2016-09-22 19:23 GMT+02:00 Elvis Stansvik
>>>>>> <elvis.stans...@orexplore.com>:
>>>>>>>
>>>>>>> 2016-09-22 19:17 GMT+02:00 Dana Robinson <derob...@hdfgroup.org>:
>>>>>>>>
>>>>>>>> Hi Elvis,
>>>>>>>>
>>>>>>>> Did you build your HDF5 library with thread-safety enabled
>>>>>>>> (--enable-threadsafe w/ configure)?
>>>>>>>
>>>>>>> Hi Dana, and thanks for the quick reply. I think we just e-mailed
>>>>>>> past
>>>>>>> each other (see my previous mail).
>>>>>>>
>>>>>>> I wrongly called it --thread-safe in that mail, but it was
>>>>>>> --enable-threadsafe I was referring to. But yes, I'm pretty sure this
>>>>>>> is the problem.
>>>>>>>
>>>>>>> I'm rebuilding the Arch package now with --enable-threadsafe.
>>>>>>
>>>>>> I spoke a little too soon. I now found this bug filed against the Arch
>>>>>> package:
>>>>>>
>>>>>>      https://bugs.archlinux.org/task/33805
>>>>>>
>>>>>> The reporter asked the package maintainer to add --enable-threadsafe,
>>>>>> but the package maintainer closed the bug saying that
>>>>>> --enable-threadsafe is not compatible with the Fortran build (in Arch,
>>>>>> the C++ and Fortran APIs are bundled into one package
>>>>>> hdf5-cpp-fortran).
>>>>>>
>>>>>> Anyone know if that is still the case? If so I can't open a bug
>>>>>> against the package again asking for --enable-threadsafe to be added.
>>>>>> But I could open a bug asking the package to be split I guess.
>>>>>>
>>>>>> Elvis
>>>>>>
>>>>>>> Elvis
>>>>>>>
>>>>>>>> Dana Robinson
>>>>>>>> Software Engineer
>>>>>>>> The HDF Group
>>>>>>>>
>>>>>>>> Get Outlook for Android
>>>>>>>>
>>>>>>>> From: Elvis Stansvik
>>>>>>>> Sent: Thursday, September 22, 12:43
>>>>>>>> Subject: [Hdf-forum] Simply using the library from separate threads
>>>>>>>> (C++
>>>>>>>> API)
>>>>>>>> To: HDF Users Discussion List
>>>>>>>>
>>>>>>>> Hi all, I'm using the C++ API to read HDF5 files from separate
>>>>>>>> threads
>>>>>>>> (no
>>>>>>>> writing). None of my threads read the same file, but they do execute
>>>>>>>> simultaneously. The reason I'm using threading is not to speed
>>>>>>>> things
>>>>>>>> up or
>>>>>>>> get better throughput, but simply to not block the UI (it's Qt
>>>>>>>> application)
>>>>>>>> while reading. So this is not about "Parallel HDF5" or anything,
>>>>>>>> just
>>>>>>>> simply
>>>>>>>> using the serial library "from scratch" from multiple threads. This
>>>>>>>> has been
>>>>>>>> working fine when testing on Ubuntu 16.04 (our target OS), which has
>>>>>>>> HDF5
>>>>>>>> 1.8.16. I recently tested on my personal Arch Linux machine though,
>>>>>>>> which
>>>>>>>> has HDF5 1.10.0, and got this segmentation fault: (gdb) bt #0
>>>>>>>> 0x00007ffff67c57d9 in H5SL_search () from /usr/lib/libhdf5.so.100 #1
>>>>>>>> 0x00007ffff678dd19 in H5P_copy_plist () from /usr/lib/libhdf5.so.100
>>>>>>>> #2
>>>>>>>> 0x00007ffff66a7fc0 in H5F_new () from /usr/lib/libhdf5.so.100 #3
>>>>>>>> 0x00007ffff66a8f55 in H5F_open () from /usr/lib/libhdf5.so.100 #4
>>>>>>>> 0x00007ffff66a155d in H5Fopen () from /usr/lib/libhdf5.so.100 #5
>>>>>>>> 0x00007ffff6b79546 in H5::H5File::p_get_file(char const*, unsigned
>>>>>>>> int,
>>>>>>>> H5::FileCreatPropList const&, H5::FileAccPropList const&) () from
>>>>>>>> /usr/lib/libhdf5_cpp.so.100 #6 0x00007ffff6b79750 in
>>>>>>>> H5::H5File::H5File(char
>>>>>>>> const*, unsigned int, H5::FileCreatPropList const&,
>>>>>>>> H5::FileAccPropList
>>>>>>>> const&) () from /usr/lib/libhdf5_cpp.so.100 #7 0x000000000041f00e in
>>>>>>>> HDF5ImageReader::RequestInformation (this=0x7fffbc002de0,
>>>>>>>> request=0x7fffbc010da0, inputVector=0x0,
>>>>>>>> outputVector=0x7fffbc0039d0)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> /home/estan/Projekt/orexplore/dev/src/insight/src/model/HDF5ImageReader.cpp:91
>>>>>>>> #8 0x00007fffee8200d0 in
>>>>>>>> vtkExecutive::CallAlgorithm(vtkInformation*,
>>>>>>>> int,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #9 0x00007fffee837fa9 in
>>>>>>>>
>>>>>>>> vtkStreamingDemandDrivenPipeline::ExecuteInformation(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #10 0x00007fffee81ce05 in
>>>>>>>> vtkDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #11 0x00007fffee835c55 in
>>>>>>>> vtkStreamingDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #12 0x00007fffee816e1a in
>>>>>>>> vtkCompositeDataPipeline::ForwardUpstream(vtkInformation*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #13 0x00007fffee81ccb5 in
>>>>>>>> vtkDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #14 0x00007fffee835c55 in
>>>>>>>> vtkStreamingDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #15 0x00007fffee816e1a in
>>>>>>>> vtkCompositeDataPipeline::ForwardUpstream(vtkInformation*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #16 0x00007fffee81ccb5 in
>>>>>>>> vtkDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #17 0x00007fffee835c55 in
>>>>>>>> vtkStreamingDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #18 0x00007fffee816e1a in
>>>>>>>> vtkCompositeDataPipeline::ForwardUpstream(vtkInformation*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #19 0x00007fffee81ccb5 in
>>>>>>>> vtkDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #20 0x00007fffee835c55 in
>>>>>>>> vtkStreamingDemandDrivenPipeline::ProcessRequest(vtkInformation*,
>>>>>>>> vtkInformationVector**, vtkInformationVector*) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #21 0x00007fffee836482 in
>>>>>>>> vtkStreamingDemandDrivenPipeline::Update(int) () from
>>>>>>>> /usr/lib/libvtkCommonExecutionModel.so.1 #22 0x00007ffff1289a76 in
>>>>>>>> vtkAbstractVolumeMapper::GetBounds() () from
>>>>>>>> /usr/lib/libvtkRenderingCore.so.1 #23 0x00007ffff13459f9 in
>>>>>>>> vtkVolume::GetBounds() () from /usr/lib/libvtkRenderingCore.so.1 #24
>>>>>>>> 0x000000000043f235 in createVolume (image=..., from=0,
>>>>>>>> to=2.7803999378532183, opacityFunction=..., colorFunction=...) at
>>>>>>>>
>>>>>>>>
>>>>>>>> /home/estan/Projekt/orexplore/dev/src/insight/src/view/Pipeline.cpp:123
>>>>>>>>  #25
>>>>>>>> 0x00000000004295c4 in CreateVolume::operator() (this=0x829248,
>>>>>>>> image=...) at
>>>>>>>> /home/estan/Projekt/orexplore/dev/src/insight/src/view/Pipeline.h:45
>>>>>>>> #26
>>>>>>>> 0x000000000042bc7a in
>>>>>>>> QtConcurrent::MappedEachKernel::const_iterator,
>>>>>>>> CreateVolume>::runIteration (this=0x829210, it=...,
>>>>>>>> result=0x7fffbc002da8)
>>>>>>>> at /usr/include/qt/QtConcurrent/qtconcurrentmapkernel.h:176 #27
>>>>>>>> 0x000000000042bd5d in
>>>>>>>> QtConcurrent::MappedEachKernel::const_iterator,
>>>>>>>> CreateVolume>::runIterations (this=0x829210,
>>>>>>>> sequenceBeginIterator=...,
>>>>>>>> begin=1, end=2, results=0x7fffbc002da8) at
>>>>>>>> /usr/include/qt/QtConcurrent/qtconcurrentmapkernel.h:186 #28
>>>>>>>> 0x000000000042c4e1 in QtConcurrent::IterateKernel::const_iterator,
>>>>>>>> vtkSmartPointer >::forThreadFunction (this=0x829210) at
>>>>>>>> /usr/include/qt/QtConcurrent/qtconcurrentiteratekernel.h:256 #29
>>>>>>>> 0x000000000042bedc in QtConcurrent::IterateKernel::const_iterator,
>>>>>>>> vtkSmartPointer >::threadFunction (this=0x829210) at
>>>>>>>> /usr/include/qt/QtConcurrent/qtconcurrentiteratekernel.h:218 #30
>>>>>>>> 0x00007ffff7bd5cfd in QtConcurrent::ThreadEngineBase::run() () from
>>>>>>>> /usr/lib/libQt5Concurrent.so.5 #31 0x00007ffff489a01f in ?? () from
>>>>>>>> /usr/lib/libQt5Core.so.5 #32 0x00007ffff489dd78 in ?? () from
>>>>>>>> /usr/lib/libQt5Core.so.5 #33 0x00007fffeb3f5454 in start_thread ()
>>>>>>>> from
>>>>>>>> /usr/lib/libpthread.so.0 #34 0x00007fffec5f07df in clone () from
>>>>>>>> /usr/lib/libc.so.6 (gdb) Before I start digging into what is
>>>>>>>> happening
>>>>>>>> here
>>>>>>>> I'd just like to ask: Do I have to do something special when using
>>>>>>>> the
>>>>>>>> HDF5
>>>>>>>> library from two different threads? I'm not reading the same files
>>>>>>>> or
>>>>>>>> anything, it's simply two completely separate usages of the library
>>>>>>>> in
>>>>>>>> threads that run in parallel. Does the library have any global
>>>>>>>> structures or
>>>>>>>> something that must be initialized before spawning any threads that
>>>>>>>> use it?
>>>>>>>> The reason I'm a little worried is that perhaps I've just been lucky
>>>>>>>> when
>>>>>>>> running under Ubuntu / HDF5 1.8.16. My usage in each thread
>>>>>>>> basically
>>>>>>>> looks
>>>>>>>> like: 1) Create a H5::H5File 2) Open a dataset using
>>>>>>>> file.openDataset
>>>>>>>> 3) Get
>>>>>>>> the dataspace for the dataset and select a hyperslab 4) Create a
>>>>>>>> memory
>>>>>>>> dataspace 5) Perform a single read(..) operation from the dataset
>>>>>>>> dataspace
>>>>>>>> to the memory dataspace And it's always different files that the
>>>>>>>> threads
>>>>>>>> work with. Is there some step 0 I'm missing? Thanks in advance for
>>>>>>>> any
>>>>>>>> advice. Elvis _______________________________________________
>>>>>>>> Hdf-forum is
>>>>>>>> for HDF software users discussion. Hdf-forum@lists.hdfgroup.org
>>>>>>>>
>>>>>>>>
>>>>>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>>>> Twitter: https://twitter.com/hdf5
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>>>> Hdf-forum@lists.hdfgroup.org
>>>>>>>>
>>>>>>>>
>>>>>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>>>> Twitter: https://twitter.com/hdf5
>>>>>>
>>>>>> _______________________________________________
>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>> Hdf-forum@lists.hdfgroup.org
>>>>>>
>>>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>> Twitter: https://twitter.com/hdf5
>>>>>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> Hdf-forum@lists.hdfgroup.org
>>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>> Twitter: https://twitter.com/hdf5
>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> Hdf-forum@lists.hdfgroup.org
>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>> Twitter: https://twitter.com/hdf5
>>>
>>>
>>> --
>>>
>>> ___________________________________________________________________________
>>> Dr. Werner Benger                Visualization Research
>>> Center for Computation & Technology at Louisiana State University
>>> (CCT/LSU)
>>> 2019  Digital Media Center, Baton Rouge, Louisiana 70803
>>> Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362
>>>
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum@lists.hdfgroup.org
>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>> Twitter: https://twitter.com/hdf5
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@lists.hdfgroup.org
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>
>
> --
> ___________________________________________________________________________
> Dr. Werner Benger                Visualization Research
> Center for Computation & Technology at Louisiana State University (CCT/LSU)
> 2019  Digital Media Center, Baton Rouge, Louisiana 70803
> Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Reply via email to