Hi, Today I spent some more time on this issue, and I recompiled everything (the Blosc library and its plugin) against HDF5 1.10.1 (the binaries from hdfgroup.org) in 64 bit mode. Again, I can read Blosc-compressed HDF5 files with h5ls and h5dump (so I am pretty sure that the plugin is correctly compiled), but h5repack is still failing:
""" C:\tmp λ h5repack -v -f UD=32001,8,0,0,0,0,0,9,1,2 carray1.h5 carray1-repack.h5 Objects to modify layout are... Objects to apply filter are... User Defined 32001 Making file <carray1-repack.h5>... ----------------------------------------- Type Filter (Compression) Name ----------------------------------------- group / HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0: #000: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Pocpl.c line 1037 in H5Pget_filter_by_id2(): can't find object for ID major: Object atom minor: Unable to find atom information (already closed?) #001: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Pint.c line 3815 in H5P_object_verify(): property list is not a member of the class major: Property lists minor: Unable to register new atom #002: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Pint.c line 3765 in H5P_isa_class(): not a propert y list major: Invalid arguments to routine minor: Inappropriate type warning: could not create dataset </carray>. Applying original settings dset /carray <warning: could not apply the filter to /carray> """ At this point, I don't know what to do. I am using VS2017 Community Edition; could it be possible that I need VS2015 for compiling the plugin? Also, I have tried to use MingW64 to compile the plugin, but cmake has difficulties in finding the HDF5 installation: """ faltet@FRANCESCALT476E MINGW64 /c/Users/faltet/hdf5-blosc/build $ HDF5_ROOT="/c/Program Files/HDF_Group/HDF5/1.10.1" cmake .. System is unknown to cmake, create: Platform/MINGW64_NT-10.0 to use this system, please send your config file to cma k...@www.cmake.org so it can be added to cmake Your CMakeCache.txt file was copied to CopyOfCMakeCache.txt. Please send that fi le to cm...@www.cmake.org. BLOSC_PREFIX='/c/Users/faltet/hdf5-blosc/build/blosc' BLOSC_INSTALL_DIR='/c/Users/faltet/hdf5-blosc/build/blosc' BLOSC_CMAKE_ARGS='-DCMAKE_INSTALL_PREFIX=/c/Users/faltet/hdf5-blosc/build/blosc' GIT_EXECUTABLE='/usr/bin/git.exe' CMake Error at /usr/share/cmake-3.6.2/Modules/FindPackageHandleStandardArgs.cmak e:148 (message): Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_HL_LIBRARIES) (found version "1.10.1") Call Stack (most recent call first): /usr/share/cmake-3.6.2/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA _FAILURE_MESSAGE) /usr/share/cmake-3.6.2/Modules/FindHDF5.cmake:820 (find_package_handle_standar d_args) CMakeLists.txt:48 (find_package) -- Configuring incomplete, errors occurred! See also "/c/Users/faltet/hdf5-blosc/build/CMakeFiles/CMakeOutput.log". """ Thanks! 2017-06-28 22:25 GMT+02:00 Werner Benger <wer...@cct.lsu.edu>: > Francesc, I am getting this message, which is correct since I don't have > it compiled with zstd. > > h5repack -f UD=32001,8,0,0,0,0,0,9,1,5 myRGBA.f5 myRGBA-out.f5 > HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 10592: > #000: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5D.c line > 426 in H5Dclose(): can't decrement count on dataset ID > major: Dataset > minor: Unable to decrement reference count > #001: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5I.c line > 1549 in H5I_dec_app_ref_always_close(): can't decrement ID ref count > major: Object atom > minor: Unable to decrement reference count > #002: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5I.c line > 1491 in H5I_dec_app_ref(): can't decrement ID ref count > major: Object atom > minor: Unable to decrement reference count > #003: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c > line 1544 in H5D_close(): unable to destroy chunk cache > major: Dataset > minor: Unable to release object > #004: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c > line 5164 in H5D__chunk_dest(): unable to flush one or more raw data chunks > major: Low-level I/O > minor: Unable to flush data from cache > #005: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c > line 2641 in H5D__chunk_cache_evict(): cannot flush indexed storage buffer > major: Low-level I/O > minor: Write failed > #006: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c > line 2528 in H5D__chunk_flush_entry(): output pipeline failed > major: Data filters > minor: Filter operation failed > #007: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Z.c line > 1412 in H5Z_pipeline(): filter returned failure > major: Data filters > minor: Write failed > #008: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/h5zblosc/blosc_filter.c > line 190 in blosc_filter(): this Blosc library does not have support for the > 'zstd' compressor, but only for: blosclz,lz4,lz4hc,zlib > major: Data filters > minor: Callback failed > #009: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c > line 1507 in H5D_close(): unable to flush cached dataset info > major: Dataset > minor: Write failed > #010: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c > line 2438 in H5D__flush_real(): unable to flush raw data > major: Dataset > minor: Unable to flush data from cache > #011: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c > line 2112 in H5D__chunk_flush(): unable to flush one or more raw data chunks > major: Dataset > minor: Unable to flush data from cache > #012: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c > line 2528 in H5D__chunk_flush_entry(): output pipeline failed > major: Data filters > minor: Filter operation failed > #013: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Z.c line > 1412 in H5Z_pipeline(): filter returned failure > major: Data filters > minor: Write failed > #014: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/h5zblosc/blosc_filter.c > line 190 in blosc_filter(): this Blosc library does not have support for the > 'zstd' compressor, but only for: blosclz,lz4,lz4hc,zlib > major: Data filters > minor: Callback failed > h5repack error: <myRGBA.f5>: Could not copy data to: myRGBA-out.f5 > > But when using LZ4HC as filter it works fine: > > Wrk $ h5repack -f UD=32001,8,0,0,0,0,0,9,1,2 myRGBA.f5 myRGBA-out.f5 > > Wrk $ ll > > -rw-r--r-- 1 Wrk None 12M Jan 25 16:43 myRGBA.f5 > > -rw-r--r-- 1 Wrk None 5.0M Jun 28 22:18 myRGBA-out.f5 > > I'm using 1.8.17, evidently. > > Btw, the restriction with the lib-prefix has the advantage that the HDF5 > plugin loader does not look at DLL's such as "blosc.dll", it would not even > try to load it as a plugin. Just something better than "lib" would probably > be preferable, and independent of platform and compiler. I guess in your > version it even tries to load the blosc.dll itself as a plugin - which > could lead to issues if such a plugin would contain some startup code like > global c++ constructors or code in DllMain which does "something". That's > unlikely to be related to your problem though, I don't see from the output > what the problem could be, it works for me. > > Werner > > > On 28.06.2017 19:01, Francesc Alted wrote: > > Werner, I think the lib- prefix does not apply to my case because I am > using VS2015 for compiling the libraries. But agreed that HDF5 taking > different conventions depending on whether a MSVC or GCC-based is a bit > confusing. > > BTW, although the h5dump can read blosc files with the plugin, ptrepack > cannot create them on Windows: > > C:\Users\faltet>h5repack -f UD=32001,8,0,0,0,0,0,9,1,5 carray1-blosc.h5 > carray1-repack.h5 > HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0: > #000: > C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pocpl.c > line 1100 in H5Pget_filter_by_id2(): can't find object for ID > major: Object atom > minor: Unable to find atom information (already closed?) > #001: > C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c > line 3379 in H5P_object_verify(): property list is not a member of the class > major: Property lists > minor: Unable to register new atom > #002: > C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c > line 3329 in H5P_isa_class(): not a property list > major: Invalid arguments to routine > minor: Inappropriate type > HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0: > #000: > C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pocpl.c > line 1100 in H5Pget_filter_by_id2(): can't find object for ID > major: Object atom > minor: Unable to find atom information (already closed?) > #001: > C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c > line 3379 in H5P_object_verify(): property list is not a member of the class > major: Property lists > minor: Unable to register new atom > #002: > C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c > line 3329 in H5P_isa_class(): not a property list > major: Invalid arguments to routine > minor: Inappropriate type > h5repack error: <carray1-blosc.h5>: Could not copy data to: > carray1-repack.h5 > > I have tested with both 1.10.1 and 1.8.19 (above). > > whereas it works perfectly well on a Mac box: > > Francescs-MacBook-Pro:PyTables faltet$ h5repack -f > UD=32001,8,0,0,0,0,0,9,1,5 carray1-blosc.h5 carray1-repack.h5 > Francescs-MacBook-Pro:PyTables faltet$ h5ls -v carray1-repack.h5 > Opened "carray1-repack.h5" with sec2 driver. > carray Dataset {200/218, 300/300} > Location: 1:800 > Links: 1 > Chunks: {218, 300} 65400 bytes > Storage: 60000 logical bytes, 58 allocated bytes, 103448.28% > utilization > Filter-0: blosc-32001 {2, 2, 1, 65400, 9, 1, 5, 5} > Type: native unsigned char > > Any hints? > > 2017-06-28 15:15 GMT+02:00 Werner Benger <wer...@cct.lsu.edu>: > >> Hi Francesc, >> >> it took me a while to figure out this detail, too, I had the blosc.dll >> in the PATH at first, and it seems to work under some circumstances but not >> others... not entirely clear to me when it needs to be in the PATH or in >> the calling's DLL's directory outside of the PATH. >> >> There's some code in H5PL.c that checks for a lib prefix: >> >> #ifndef __CYGWIN__ >> if(!HDstrncmp(dp->d_name, "lib", (size_t)3) && >> (HDstrstr(dp->d_name, ".so") || HDstrstr(dp->d_name, >> ".dylib"))) { >> #else >> ... >> >> #endif >> >> So I guess in your case that prefix check did not apply, whereas for me >> using gcc under windows it did. Thus it seems best for compatibility to >> keep the plugins named with lib- starting under Windows, once compiled that >> C code should not matter whether it was produced by GCC or MSC. Otherwise >> yes, the name does not matter indeed. >> >> Werner >> >> On 28.06.2017 15:00, Francesc Alted wrote: >> >> Hi Werner, >> >> Right, what I was missing was basically moving the blosc.dll library into >> the HDF5 plugin directory. I thought that putting blosc.dll in a directory >> in the %PATH% was going to be enough, but apparently this is not the case >> (I suppose this is a Windows trickery). >> >> For what is worth, the name of the DLL plugin is not that important as I >> have tried with libHDF5blosc.dll, libH5Zblosc.dll, H5Zblosc.dll and even >> blosc_plugin.dll and all of them work. Apparently the HDF5 library tries >> all the DLLs in the plugin directory and uses the one that registers the >> necessary filter ID; pretty smart. >> >> Thanks so much! >> >> 2017-06-28 14:31 GMT+02:00 Werner Benger <wer...@cct.lsu.edu>: >> >>> Just to add: >>> >>> under windows, the plugins need to start with lib as prefix. It's >>> technically not required, but the HDF5 plugin loader expects that. So the >>> liblz4.dll in my plugins dir is the LZ4 plugin,the libHDF5blosc.dll the >>> blosc-filter; the blosc.dll is used by the blosc filter and needs to be in >>> the same directory if compiled as dynamic library as well. >>> >>> Werner >>> >>> On 28.06.2017 14:24, Werner Benger wrote: >>> >>> Hi, >>> >>> I am using HDF5 with blosc dynamic plugin loading regularly under >>> Windows, and this are the files as needed in the plugin directory that >>> HDF5 looks for: >>> >>> HDF5Plugins $ ll >>> total 1.2M >>> -rwxr-xr-x 1 Wrk None 382K Jun 26 23:30 blosc.dll* >>> -rwxr-xr-x 1 Wrk None 330K Jun 26 23:30 libHDF5blosc.dll* >>> -rwxr-xr-x 1 Wrk None 410K Jun 26 23:30 liblz4.dll* >>> >>> The blosc.dll is the blosc library, libHDF5blosc.dll is the HDF5 plugin >>> using that library. >>> >>> I'm using Mingw64, which is gcc 6.3 under MSYS2 . >>> >>> Werner >>> >>> On 28.06.2017 13:44, Francesc Alted wrote: >>> >>> Hi, >>> >>> I have been trying to give support to HDF5 on Windows for dynamically >>> loading the Blosc plugin (https://github.com/Blosc/hdf5-blosc), but no >>> success so far. >>> >>> I basically have compiled the plugin with: >>> >>> > cl /I"C:\Program Files (x86)\blosc\include" >>> /I"C:\HDF_Group\HDF5\1.8.12\include" libH5Zblosc.c blosc_filter.c >>> /OUT:libH5Zblosc.dll /LD /link /LIBPATH:"C:\Program Files (x86)\blosc\lib" >>> /LIBPATH:"C:\HDF_Group\HDF5\1.8.12\lib" blosc.lib hdf5.lib >>> >>> and then copied the libH5Zblosc.dll to >>> "%ALLUSERSPROFILE%/hdf5/lib/plugin", >>> but when I try to use a h5dump for getting the data of an HDF5 with some >>> datasets compressed with Blosc, I am getting: >>> >>> >h5dump \tmp\carray1-blosc.h5 >>> HDF5 "\tmp\carray1-blosc.h5" { >>> GROUP "/" { >>> DATASET "carray" { >>> DATATYPE H5T_STD_U8LE >>> DATASPACE SIMPLE { ( 200, 300 ) / ( 218, 300 ) } >>> DATA {h5dump error: unable to print data >>> >>> } >>> } >>> } >>> } >>> >>> When asking for more errors, I am getting a more informative message: >>> >>> >h5dump --enable-error-stack \tmp\carray1-blosc.h5 >>> HDF5 "\tmp\carray1-blosc.h5" { >>> GROUP "/" { >>> DATASET "carray" { >>> DATATYPE H5T_STD_U8LE >>> DATASPACE SIMPLE { ( 200, 300 ) / ( 218, 300 ) } >>> DATA {HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0: >>> #000: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dio.c >>> line 171 in H5Dread(): can't read data >>> major: Dataset >>> minor: Read failed >>> #001: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dio.c >>> line 544 in H5D__read(): can't read data >>> major: Dataset >>> minor: Read failed >>> #002: >>> C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dchunk.c >>> line 2050 in H5D__chunk_read(): unable to read raw data chunk >>> major: Low-level I/O >>> minor: Read failed >>> #003: >>> C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dchunk.c >>> line 3405 in H5D__chunk_lock(): data pipeline read failed >>> major: Data filters >>> minor: Filter operation failed >>> #004: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Z.c >>> line 1349 in H5Z_pipeline(): required filter 'blosc' is not registered >>> major: Data filters >>> minor: Read failed >>> #005: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5PL.c >>> line 389 in H5PL_load(): search in paths failed >>> major: Plugin for dynamically loaded library >>> minor: Can't get value >>> #006: C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5PL.c >>> line 812 in H5PL__find(): can't open directory >>> major: Plugin for dynamically loaded library >>> minor: Can't open directory or file >>> h5dump error: unable to print data >>> >>> } >>> } >>> } >>> } >>> H5tools-DIAG: Error detected in HDF5:tools (1.10.1) thread 0: >>> #000: >>> C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\tools\lib\h5tools_dump.c >>> line 1639 in h5tools_dump_simple_dset(): H5Dread failed >>> major: Failure in tools library >>> minor: error in function >>> >>> I suppose the meaningful part is here: >>> >>> 10vs14\build\hdfsrc\src\H5PL.c line 812 in H5PL__find(): can't open >>> directory >>> >>> b >>> ut I have no clues on 1) what directory the dynamic plugin machinery in >>> HDF5 is looking at and 2) which name the DLL should have (as per manual, I >>> am using libH5Zblosc.dll, but I have tried with H5Zblosc.dll and blosc.dll >>> with no success). >>> >>> I have also tried to use the HDF5_PLUGIN_PATH environment variable, so >>> as to direct the plugin machinery in HDF5 to look into the specific >>> %ALLUSERSPROFILE%/hdf5/lib/plugin >>> place, >>> but no luck. >>> >>> I tried with HDF5 1.8.12 and 1.10.1, with same results. Finally, if >>> it is of any help, I am using Windows 10 64 bit. >>> >>> As a suggestion, it may help to debug situations like this if the HDF5 >>> backtrace of the plugin machinery would provide info about 1) which >>> directory it is looking at for finding the HDF5 plugin and 2) the name of >>> the DLL that it is trying to load. >>> >>> Thanks in advance, >>> >>> -- >>> Francesc Alted >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users >>> discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >>> >>> -- >>> ___________________________________________________________________________ >>> Dr. Werner Benger Visualization Research >>> Center for Computation & Technology at Louisiana State University (CCT/LSU) >>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803 >>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: >>> +1 225 578-5362 <%28225%29%20578-5362> >>> >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users >>> discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >>> >>> -- >>> ___________________________________________________________________________ >>> Dr. Werner Benger Visualization Research >>> Center for Computation & Technology at Louisiana State University (CCT/LSU) >>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803 >>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: >>> +1 225 578-5362 <%28225%29%20578-5362> >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> Hdf-forum@lists.hdfgroup.org >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >> >> >> >> -- >> Francesc Alted >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users >> discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> >> >> -- >> ___________________________________________________________________________ >> Dr. Werner Benger Visualization Research >> Center for Computation & Technology at Louisiana State University (CCT/LSU) >> 2019 Digital Media Center, Baton Rouge, Louisiana 70803 >> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 >> 225 578-5362 <%28225%29%20578-5362> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> Hdf-forum@lists.hdfgroup.org >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> > > > > -- > Francesc Alted > > > _______________________________________________ > Hdf-forum is for HDF software users > discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > > > -- > ___________________________________________________________________________ > Dr. Werner Benger Visualization Research > Center for Computation & Technology at Louisiana State University (CCT/LSU) > 2019 Digital Media Center, Baton Rouge, Louisiana 70803 > Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 > 578-5362 <(225)%20578-5362> > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@lists.hdfgroup.org > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > -- Francesc Alted
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5