Hi,
Today I spent some more time on this issue, and I recompiled everything
(the Blosc library and its plugin) against HDF5 1.10.1 (the binaries from
hdfgroup.org) in 64 bit mode. Again, I can read Blosc-compressed HDF5
files with h5ls and h5dump (so I am pretty sure that the plugin is
correctly compiled), but h5repack is still failing:
"""
C:\tmp
λ h5repack -v -f UD=32001,8,0,0,0,0,0,9,1,2 carray1.h5 carray1-repack.h5
Objects to modify layout are...
Objects to apply filter are...
User Defined 32001
Making file <carray1-repack.h5>...
-----------------------------------------
Type Filter (Compression) Name
-----------------------------------------
group /
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
#000:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Pocpl.c line
1037 in H5Pget_filter_by_id2(): can't
find object for ID
major: Object atom
minor: Unable to find atom information (already closed?)
#001:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Pint.c line
3815 in H5P_object_verify(): property
list is not a member of the class
major: Property lists
minor: Unable to register new atom
#002:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Pint.c line
3765 in H5P_isa_class(): not a propert
y list
major: Invalid arguments to routine
minor: Inappropriate type
warning: could not create dataset </carray>. Applying original settings
dset /carray
<warning: could not apply the filter to /carray>
"""
At this point, I don't know what to do. I am using VS2017 Community
Edition; could it be possible that I need VS2015 for compiling the plugin?
Also, I have tried to use MingW64 to compile the plugin, but cmake has
difficulties in finding the HDF5 installation:
"""
faltet@FRANCESCALT476E MINGW64 /c/Users/faltet/hdf5-blosc/build
$ HDF5_ROOT="/c/Program Files/HDF_Group/HDF5/1.10.1" cmake ..
System is unknown to cmake, create:
Platform/MINGW64_NT-10.0 to use this system, please send your config file
to cma [email protected] so it can be added to cmake
Your CMakeCache.txt file was copied to CopyOfCMakeCache.txt. Please send
that fi le to [email protected].
BLOSC_PREFIX='/c/Users/faltet/hdf5-blosc/build/blosc'
BLOSC_INSTALL_DIR='/c/Users/faltet/hdf5-blosc/build/blosc'
BLOSC_CMAKE_ARGS='-DCMAKE_INSTALL_PREFIX=/c/Users/faltet/hdf5-blosc/build/blosc'
GIT_EXECUTABLE='/usr/bin/git.exe'
CMake Error at
/usr/share/cmake-3.6.2/Modules/FindPackageHandleStandardArgs.cmak
e:148 (message):
Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_HL_LIBRARIES) (found
version "1.10.1")
Call Stack (most recent call first):
/usr/share/cmake-3.6.2/Modules/FindPackageHandleStandardArgs.cmake:388
(_FPHSA _FAILURE_MESSAGE)
/usr/share/cmake-3.6.2/Modules/FindHDF5.cmake:820
(find_package_handle_standar d_args)
CMakeLists.txt:48 (find_package)
-- Configuring incomplete, errors occurred!
See also "/c/Users/faltet/hdf5-blosc/build/CMakeFiles/CMakeOutput.log".
"""
Thanks!
2017-06-28 22:25 GMT+02:00 Werner Benger <[email protected]>:
> Francesc, I am getting this message, which is correct since I don't have
> it compiled with zstd.
>
> h5repack -f UD=32001,8,0,0,0,0,0,9,1,5 myRGBA.f5 myRGBA-out.f5
> HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 10592:
> #000: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5D.c line
> 426 in H5Dclose(): can't decrement count on dataset ID
> major: Dataset
> minor: Unable to decrement reference count
> #001: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5I.c line
> 1549 in H5I_dec_app_ref_always_close(): can't decrement ID ref count
> major: Object atom
> minor: Unable to decrement reference count
> #002: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5I.c line
> 1491 in H5I_dec_app_ref(): can't decrement ID ref count
> major: Object atom
> minor: Unable to decrement reference count
> #003: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c
> line 1544 in H5D_close(): unable to destroy chunk cache
> major: Dataset
> minor: Unable to release object
> #004: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c
> line 5164 in H5D__chunk_dest(): unable to flush one or more raw data chunks
> major: Low-level I/O
> minor: Unable to flush data from cache
> #005: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c
> line 2641 in H5D__chunk_cache_evict(): cannot flush indexed storage buffer
> major: Low-level I/O
> minor: Write failed
> #006: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c
> line 2528 in H5D__chunk_flush_entry(): output pipeline failed
> major: Data filters
> minor: Filter operation failed
> #007: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Z.c line
> 1412 in H5Z_pipeline(): filter returned failure
> major: Data filters
> minor: Write failed
> #008: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/h5zblosc/blosc_filter.c
> line 190 in blosc_filter(): this Blosc library does not have support for the
> 'zstd' compressor, but only for: blosclz,lz4,lz4hc,zlib
> major: Data filters
> minor: Callback failed
> #009: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c
> line 1507 in H5D_close(): unable to flush cached dataset info
> major: Dataset
> minor: Write failed
> #010: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c
> line 2438 in H5D__flush_real(): unable to flush raw data
> major: Dataset
> minor: Unable to flush data from cache
> #011: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c
> line 2112 in H5D__chunk_flush(): unable to flush one or more raw data chunks
> major: Dataset
> minor: Unable to flush data from cache
> #012: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c
> line 2528 in H5D__chunk_flush_entry(): output pipeline failed
> major: Data filters
> minor: Filter operation failed
> #013: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Z.c line
> 1412 in H5Z_pipeline(): filter returned failure
> major: Data filters
> minor: Write failed
> #014: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/h5zblosc/blosc_filter.c
> line 190 in blosc_filter(): this Blosc library does not have support for the
> 'zstd' compressor, but only for: blosclz,lz4,lz4hc,zlib
> major: Data filters
> minor: Callback failed
> h5repack error: <myRGBA.f5>: Could not copy data to: myRGBA-out.f5
>
> But when using LZ4HC as filter it works fine:
>
> Wrk $ h5repack -f UD=32001,8,0,0,0,0,0,9,1,2 myRGBA.f5 myRGBA-out.f5
>
> Wrk $ ll
>
> -rw-r--r-- 1 Wrk None 12M Jan 25 16:43 myRGBA.f5
>
> -rw-r--r-- 1 Wrk None 5.0M Jun 28 22:18 myRGBA-out.f5
>
> I'm using 1.8.17, evidently.
>
> Btw, the restriction with the lib-prefix has the advantage that the HDF5
> plugin loader does not look at DLL's such as "blosc.dll", it would not even
> try to load it as a plugin. Just something better than "lib" would probably
> be preferable, and independent of platform and compiler. I guess in your
> version it even tries to load the blosc.dll itself as a plugin - which
> could lead to issues if such a plugin would contain some startup code like
> global c++ constructors or code in DllMain which does "something". That's
> unlikely to be related to your problem though, I don't see from the output
> what the problem could be, it works for me.
>
> Werner
>
>
> On 28.06.2017 19:01, Francesc Alted wrote:
>
> Werner, I think the lib- prefix does not apply to my case because I am
> using VS2015 for compiling the libraries. But agreed that HDF5 taking
> different conventions depending on whether a MSVC or GCC-based is a bit
> confusing.
>
> BTW, although the h5dump can read blosc files with the plugin, ptrepack
> cannot create them on Windows:
>
> C:\Users\faltet>h5repack -f UD=32001,8,0,0,0,0,0,9,1,5 carray1-blosc.h5
> carray1-repack.h5
> HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0:
> #000:
> C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pocpl.c
> line 1100 in H5Pget_filter_by_id2(): can't find object for ID
> major: Object atom
> minor: Unable to find atom information (already closed?)
> #001:
> C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
> line 3379 in H5P_object_verify(): property list is not a member of the class
> major: Property lists
> minor: Unable to register new atom
> #002:
> C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
> line 3329 in H5P_isa_class(): not a property list
> major: Invalid arguments to routine
> minor: Inappropriate type
> HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0:
> #000:
> C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pocpl.c
> line 1100 in H5Pget_filter_by_id2(): can't find object for ID
> major: Object atom
> minor: Unable to find atom information (already closed?)
> #001:
> C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
> line 3379 in H5P_object_verify(): property list is not a member of the class
> major: Property lists
> minor: Unable to register new atom
> #002:
> C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
> line 3329 in H5P_isa_class(): not a property list
> major: Invalid arguments to routine
> minor: Inappropriate type
> h5repack error: <carray1-blosc.h5>: Could not copy data to:
> carray1-repack.h5
>
> I have tested with both 1.10.1 and 1.8.19 (above).
>
> whereas it works perfectly well on a Mac box:
>
> Francescs-MacBook-Pro:PyTables faltet$ h5repack -f
> UD=32001,8,0,0,0,0,0,9,1,5 carray1-blosc.h5 carray1-repack.h5
> Francescs-MacBook-Pro:PyTables faltet$ h5ls -v carray1-repack.h5
> Opened "carray1-repack.h5" with sec2 driver.
> carray Dataset {200/218, 300/300}
> Location: 1:800
> Links: 1
> Chunks: {218, 300} 65400 bytes
> Storage: 60000 logical bytes, 58 allocated bytes, 103448.28%
> utilization
> Filter-0: blosc-32001 {2, 2, 1, 65400, 9, 1, 5, 5}
> Type: native unsigned char
>
> Any hints?
>
> 2017-06-28 15:15 GMT+02:00 Werner Benger <[email protected]>:
>
>> Hi Francesc,
>>
>> it took me a while to figure out this detail, too, I had the blosc.dll
>> in the PATH at first, and it seems to work under some circumstances but not
>> others... not entirely clear to me when it needs to be in the PATH or in
>> the calling's DLL's directory outside of the PATH.
>>
>> There's some code in H5PL.c that checks for a lib prefix:
>>
>> #ifndef __CYGWIN__
>> if(!HDstrncmp(dp->d_name, "lib", (size_t)3) &&
>> (HDstrstr(dp->d_name, ".so") || HDstrstr(dp->d_name,
>> ".dylib"))) {
>> #else
>> ...
>>
>> #endif
>>
>> So I guess in your case that prefix check did not apply, whereas for me
>> using gcc under windows it did. Thus it seems best for compatibility to
>> keep the plugins named with lib- starting under Windows, once compiled that
>> C code should not matter whether it was produced by GCC or MSC. Otherwise
>> yes, the name does not matter indeed.
>>
>> Werner
>>
>> On 28.06.2017 15:00, Francesc Alted wrote:
>>
>> Hi Werner,
>>
>> Right, what I was missing was basically moving the blosc.dll library into
>> the HDF5 plugin directory. I thought that putting blosc.dll in a directory
>> in the %PATH% was going to be enough, but apparently this is not the case
>> (I suppose this is a Windows trickery).
>>
>> For what is worth, the name of the DLL plugin is not that important as I
>> have tried with libHDF5blosc.dll, libH5Zblosc.dll, H5Zblosc.dll and even
>> blosc_plugin.dll and all of them work. Apparently the HDF5 library tries
>> all the DLLs in the plugin directory and uses the one that registers the
>> necessary filter ID; pretty smart.
>>
>> Thanks so much!
>>
>> 2017-06-28 14:31 GMT+02:00 Werner Benger <[email protected]>:
>>
>>> Just to add:
>>>
>>> under windows, the plugins need to start with lib as prefix. It's
>>> technically not required, but the HDF5 plugin loader expects that. So the
>>> liblz4.dll in my plugins dir is the LZ4 plugin,the libHDF5blosc.dll the
>>> blosc-filter; the blosc.dll is used by the blosc filter and needs to be in
>>> the same directory if compiled as dynamic library as well.
>>>
>>> Werner
>>>
>>> On 28.06.2017 14:24, Werner Benger wrote:
>>>
>>> Hi,
>>>
>>> I am using HDF5 with blosc dynamic plugin loading regularly under
>>> Windows, and this are the files as needed in the plugin directory that
>>> HDF5 looks for:
>>>
>>> HDF5Plugins $ ll
>>> total 1.2M
>>> -rwxr-xr-x 1 Wrk None 382K Jun 26 23:30 blosc.dll*
>>> -rwxr-xr-x 1 Wrk None 330K Jun 26 23:30 libHDF5blosc.dll*
>>> -rwxr-xr-x 1 Wrk None 410K Jun 26 23:30 liblz4.dll*
>>>
>>> The blosc.dll is the blosc library, libHDF5blosc.dll is the HDF5 plugin
>>> using that library.
>>>
>>> I'm using Mingw64, which is gcc 6.3 under MSYS2 .
>>>
>>> Werner
>>>
>>> On 28.06.2017 13:44, Francesc Alted wrote:
>>>
>>> Hi,
>>>
>>> I have been trying to give support to HDF5 on Windows for dynamically
>>> loading the Blosc plugin (https://github.com/Blosc/hdf5-blosc), but no
>>> success so far.
>>>
>>> I basically have compiled the plugin with:
>>>
>>> > cl /I"C:\Program Files (x86)\blosc\include"
>>> /I"C:\HDF_Group\HDF5\1.8.12\include" libH5Zblosc.c blosc_filter.c
>>> /OUT:libH5Zblosc.dll /LD /link /LIBPATH:"C:\Program Files (x86)\blosc\lib"
>>> /LIBPATH:"C:\HDF_Group\HDF5\1.8.12\lib" blosc.lib hdf5.lib
>>>
>>> and then copied the libH5Zblosc.dll to
>>> "%ALLUSERSPROFILE%/hdf5/lib/plugin",
>>> but when I try to use a h5dump for getting the data of an HDF5 with some
>>> datasets compressed with Blosc, I am getting:
>>>
>>> >h5dump \tmp\carray1-blosc.h5
>>> HDF5 "\tmp\carray1-blosc.h5" {
>>> GROUP "/" {
>>> DATASET "carray" {
>>> DATATYPE H5T_STD_U8LE
>>> DATASPACE SIMPLE { ( 200, 300 ) / ( 218, 300 ) }
>>> DATA {h5dump error: unable to print data
>>>
>>> }
>>> }
>>> }
>>> }
>>>
>>> When asking for more errors, I am getting a more informative message:
>>>
>>> >h5dump --enable-error-stack \tmp\carray1-blosc.h5
>>> HDF5 "\tmp\carray1-blosc.h5" {
>>> GROUP "/" {
>>> DATASET "carray" {
>>> DATATYPE H5T_STD_U8LE
>>> DATASPACE SIMPLE { ( 200, 300 ) / ( 218, 300 ) }
>>> DATA {HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
>>> #000: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dio.c
>>> line 171 in H5Dread(): can't read data
>>> major: Dataset
>>> minor: Read failed
>>> #001: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dio.c
>>> line 544 in H5D__read(): can't read data
>>> major: Dataset
>>> minor: Read failed
>>> #002:
>>> C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dchunk.c
>>> line 2050 in H5D__chunk_read(): unable to read raw data chunk
>>> major: Low-level I/O
>>> minor: Read failed
>>> #003:
>>> C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dchunk.c
>>> line 3405 in H5D__chunk_lock(): data pipeline read failed
>>> major: Data filters
>>> minor: Filter operation failed
>>> #004: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Z.c
>>> line 1349 in H5Z_pipeline(): required filter 'blosc' is not registered
>>> major: Data filters
>>> minor: Read failed
>>> #005: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5PL.c
>>> line 389 in H5PL_load(): search in paths failed
>>> major: Plugin for dynamically loaded library
>>> minor: Can't get value
>>> #006: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5PL.c
>>> line 812 in H5PL__find(): can't open directory
>>> major: Plugin for dynamically loaded library
>>> minor: Can't open directory or file
>>> h5dump error: unable to print data
>>>
>>> }
>>> }
>>> }
>>> }
>>> H5tools-DIAG: Error detected in HDF5:tools (1.10.1) thread 0:
>>> #000:
>>> C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\tools\lib\h5tools_dump.c
>>> line 1639 in h5tools_dump_simple_dset(): H5Dread failed
>>> major: Failure in tools library
>>> minor: error in function
>>>
>>> I suppose the meaningful part is here:
>>>
>>> 10vs14\build\hdfsrc\src\H5PL.c line 812 in H5PL__find(): can't open
>>> directory
>>>
>>> b
>>> ut I have no clues on 1) what directory the dynamic plugin machinery in
>>> HDF5 is looking at and 2) which name the DLL should have (as per manual, I
>>> am using libH5Zblosc.dll, but I have tried with H5Zblosc.dll and blosc.dll
>>> with no success).
>>>
>>> I have also tried to use the HDF5_PLUGIN_PATH environment variable, so
>>> as to direct the plugin machinery in HDF5 to look into the specific
>>> %ALLUSERSPROFILE%/hdf5/lib/plugin
>>> place,
>>> but no luck.
>>>
>>> I tried with HDF5 1.8.12 and 1.10.1, with same results. Finally, if
>>> it is of any help, I am using Windows 10 64 bit.
>>>
>>> As a suggestion, it may help to debug situations like this if the HDF5
>>> backtrace of the plugin machinery would provide info about 1) which
>>> directory it is looking at for finding the HDF5 plugin and 2) the name of
>>> the DLL that it is trying to load.
>>>
>>> Thanks in advance,
>>>
>>> --
>>> Francesc Alted
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users
>>> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>> Twitter: https://twitter.com/hdf5
>>>
>>>
>>> --
>>> ___________________________________________________________________________
>>> Dr. Werner Benger Visualization Research
>>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803
>>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.:
>>> +1 225 578-5362 <%28225%29%20578-5362>
>>>
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users
>>> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>> Twitter: https://twitter.com/hdf5
>>>
>>>
>>> --
>>> ___________________________________________________________________________
>>> Dr. Werner Benger Visualization Research
>>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803
>>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.:
>>> +1 225 578-5362 <%28225%29%20578-5362>
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>> Twitter: https://twitter.com/hdf5
>>>
>>
>>
>>
>> --
>> Francesc Alted
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users
>> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>>
>>
>> --
>> ___________________________________________________________________________
>> Dr. Werner Benger Visualization Research
>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803
>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1
>> 225 578-5362 <%28225%29%20578-5362>
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>>
>
>
>
> --
> Francesc Alted
>
>
> _______________________________________________
> Hdf-forum is for HDF software users
> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
>
> --
> ___________________________________________________________________________
> Dr. Werner Benger Visualization Research
> Center for Computation & Technology at Louisiana State University (CCT/LSU)
> 2019 Digital Media Center, Baton Rouge, Louisiana 70803
> Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225
> 578-5362 <(225)%20578-5362>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
--
Francesc Alted
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
