Francesc, I am getting this message, which is correct since I don't have
it compiled with zstd.
h5repack -f UD=32001,8,0,0,0,0,0,9,1,5 myRGBA.f5 myRGBA-out.f5
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 10592: #000:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5D.c line 426
in H5Dclose(): can't decrement count on dataset ID major: Dataset
minor: Unable to decrement reference count #001:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5I.c line 1549
in H5I_dec_app_ref_always_close(): can't decrement ID ref count
major: Object atom minor: Unable to decrement reference count
#002: C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5I.c
line 1491 in H5I_dec_app_ref(): can't decrement ID ref count major:
Object atom minor: Unable to decrement reference count #003:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c line
1544 in H5D_close(): unable to destroy chunk cache major: Dataset
minor: Unable to release object #004:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c line
5164 in H5D__chunk_dest(): unable to flush one or more raw data chunks
major: Low-level I/O minor: Unable to flush data from cache
#005:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c line
2641 in H5D__chunk_cache_evict(): cannot flush indexed storage buffer
major: Low-level I/O minor: Write failed #006:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c line
2528 in H5D__chunk_flush_entry(): output pipeline failed major: Data
filters minor: Filter operation failed #007:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Z.c line 1412
in H5Z_pipeline(): filter returned failure major: Data filters
minor: Write failed #008:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/h5zblosc/blosc_filter.c
line 190 in blosc_filter(): this Blosc library does not have support for
the 'zstd' compressor, but only for: blosclz,lz4,lz4hc,zlib major:
Data filters minor: Callback failed #009:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c line
1507 in H5D_close(): unable to flush cached dataset info major:
Dataset minor: Write failed #010:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dint.c line
2438 in H5D__flush_real(): unable to flush raw data major: Dataset
minor: Unable to flush data from cache #011:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c line
2112 in H5D__chunk_flush(): unable to flush one or more raw data chunks
major: Dataset minor: Unable to flush data from cache #012:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Dchunk.c line
2528 in H5D__chunk_flush_entry(): output pipeline failed major: Data
filters minor: Filter operation failed #013:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/hdf5-1.8.17/H5Z.c line 1412
in H5Z_pipeline(): filter returned failure major: Data filters
minor: Write failed #014:
C:/msys64/home/Wrk/vish/fish/fiber/FiberHDF5/h5zblosc/blosc_filter.c
line 190 in blosc_filter(): this Blosc library does not have support for
the 'zstd' compressor, but only for: blosclz,lz4,lz4hc,zlib major:
Data filters minor: Callback failed h5repack error: <myRGBA.f5>:
Could not copy data to: myRGBA-out.f5
But when using LZ4HC as filter it works fine:
Wrk $ h5repack -f UD=32001,8,0,0,0,0,0,9,1,2 myRGBA.f5 myRGBA-out.f5
Wrk $ ll
-rw-r--r-- 1 Wrk None 12M Jan 25 16:43 myRGBA.f5
-rw-r--r-- 1 Wrk None 5.0M Jun 28 22:18 myRGBA-out.f5
I'm using 1.8.17, evidently.
Btw, the restriction with the lib-prefix has the advantage that the HDF5
plugin loader does not look at DLL's such as "blosc.dll", it would not
even try to load it as a plugin. Just something better than "lib" would
probably be preferable, and independent of platform and compiler. I
guess in your version it even tries to load the blosc.dll itself as a
plugin - which could lead to issues if such a plugin would contain some
startup code like global c++ constructors or code in DllMain which does
"something". That's unlikely to be related to your problem though, I
don't see from the output what the problem could be, it works for me.
Werner
On 28.06.2017 19:01, Francesc Alted wrote:
Werner, I think the lib- prefix does not apply to my case because I am
using VS2015 for compiling the libraries. But agreed that HDF5 taking
different conventions depending on whether a MSVC or GCC-based is a
bit confusing.
BTW, although the h5dump can read blosc files with the plugin,
ptrepack cannot create them on Windows:
C:\Users\faltet>h5repack -f UD=32001,8,0,0,0,0,0,9,1,5
carray1-blosc.h5 carray1-repack.h5
HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0:
#000:
C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pocpl.c
line 1100 in H5Pget_filter_by_id2(): can't find object for ID
major: Object atom
minor: Unable to find atom information (already closed?)
#001:
C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
line 3379 in H5P_object_verify(): property list is not a member of the
class
major: Property lists
minor: Unable to register new atom
#002:
C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
line 3329 in H5P_isa_class(): not a property list
major: Invalid arguments to routine
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0:
#000:
C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pocpl.c
line 1100 in H5Pget_filter_by_id2(): can't find object for ID
major: Object atom
minor: Unable to find atom information (already closed?)
#001:
C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
line 3379 in H5P_object_verify(): property list is not a member of the
class
major: Property lists
minor: Unable to register new atom
#002:
C:\Users\buildbot\Buildbot\hdf518-StdRelease-code-vs14\build\hdfsrc\src\H5Pint.c
line 3329 in H5P_isa_class(): not a property list
major: Invalid arguments to routine
minor: Inappropriate type
h5repack error: <carray1-blosc.h5>: Could not copy data to:
carray1-repack.h5
I have tested with both 1.10.1 and 1.8.19 (above).
whereas it works perfectly well on a Mac box:
Francescs-MacBook-Pro:PyTables faltet$ h5repack -f
UD=32001,8,0,0,0,0,0,9,1,5 carray1-blosc.h5 carray1-repack.h5
Francescs-MacBook-Pro:PyTables faltet$ h5ls -v carray1-repack.h5
Opened "carray1-repack.h5" with sec2 driver.
carray Dataset {200/218, 300/300}
Location: 1:800
Links: 1
Chunks: {218, 300} 65400 bytes
Storage: 60000 logical bytes, 58 allocated bytes, 103448.28% utilization
Filter-0: blosc-32001 {2, 2, 1, 65400, 9, 1, 5, 5}
Type: native unsigned char
Any hints?
2017-06-28 15:15 GMT+02:00 Werner Benger <wer...@cct.lsu.edu
<mailto:wer...@cct.lsu.edu>>:
Hi Francesc,
it took me a while to figure out this detail, too, I had the
blosc.dll in the PATH at first, and it seems to work under some
circumstances but not others... not entirely clear to me when it
needs to be in the PATH or in the calling's DLL's directory
outside of the PATH.
There's some code in H5PL.c that checks for a lib prefix:
#ifndef __CYGWIN__
if(!HDstrncmp(dp->d_name, "lib", (size_t)3) &&
(HDstrstr(dp->d_name, ".so") ||
HDstrstr(dp->d_name, ".dylib"))) {
#else
...
#endif
So I guess in your case that prefix check did not apply, whereas
for me using gcc under windows it did. Thus it seems best for
compatibility to keep the plugins named with lib- starting under
Windows, once compiled that C code should not matter whether it
was produced by GCC or MSC. Otherwise yes, the name does not
matter indeed.
Werner
On 28.06.2017 15:00, Francesc Alted wrote:
Hi Werner,
Right, what I was missing was basically moving the blosc.dll
library into the HDF5 plugin directory. I thought that putting
blosc.dll in a directory in the %PATH% was going to be enough,
but apparently this is not the case (I suppose this is a Windows
trickery).
For what is worth, the name of the DLL plugin is not that
important as I have tried with libHDF5blosc.dll, libH5Zblosc.dll,
H5Zblosc.dll and even blosc_plugin.dll and all of them work.
Apparently the HDF5 library tries all the DLLs in the plugin
directory and uses the one that registers the necessary filter
ID; pretty smart.
Thanks so much!
2017-06-28 14:31 GMT+02:00 Werner Benger <wer...@cct.lsu.edu
<mailto:wer...@cct.lsu.edu>>:
Just to add:
under windows, the plugins need to start with lib as prefix.
It's technically not required, but the HDF5 plugin loader
expects that. So the liblz4.dll in my plugins dir is the LZ4
plugin,the libHDF5blosc.dll the blosc-filter; the blosc.dll
is used by the blosc filter and needs to be in the same
directory if compiled as dynamic library as well.
Werner
On 28.06.2017 14:24, Werner Benger wrote:
Hi,
I am using HDF5 with blosc dynamic plugin loading regularly
under Windows, and this are the files as needed in the
plugin directory that HDF5 looks for:
HDF5Plugins $ ll
total 1.2M
-rwxr-xr-x 1 Wrk None 382K Jun 26 23:30 blosc.dll*
-rwxr-xr-x 1 Wrk None 330K Jun 26 23:30 libHDF5blosc.dll*
-rwxr-xr-x 1 Wrk None 410K Jun 26 23:30 liblz4.dll*
The blosc.dll is the blosc library, libHDF5blosc.dll is the
HDF5 plugin using that library.
I'm using Mingw64, which is gcc 6.3 under MSYS2 .
Werner
On 28.06.2017 13:44, Francesc Alted wrote:
Hi,
I have been trying to give support to HDF5 on Windows for
dynamically loading the Blosc plugin
(https://github.com/Blosc/hdf5-blosc
<https://github.com/Blosc/hdf5-blosc>), but no success so far.
I basically have compiled the plugin with:
> cl /I"C:\Program Files (x86)\blosc\include"
/I"C:\HDF_Group\HDF5\1.8.12\include" libH5Zblosc.c
blosc_filter.c /OUT:libH5Zblosc.dll /LD /link
/LIBPATH:"C:\Program Files (x86)\blosc\lib"
/LIBPATH:"C:\HDF_Group\HDF5\1.8.12\lib" blosc.lib hdf5.lib
and then copied the libH5Zblosc.dll
to "%ALLUSERSPROFILE%/hdf5/lib/plugin", but when I try to
use a h5dump for getting the data of an HDF5 with some
datasets compressed with Blosc, I am getting:
>h5dump \tmp\carray1-blosc.h5
HDF5 "\tmp\carray1-blosc.h5" {
GROUP "/" {
DATASET "carray" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 200, 300 ) / ( 218, 300 ) }
DATA {h5dump error: unable to print data
}
}
}
}
When asking for more errors, I am getting a more
informative message:
>h5dump --enable-error-stack \tmp\carray1-blosc.h5
HDF5 "\tmp\carray1-blosc.h5" {
GROUP "/" {
DATASET "carray" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 200, 300 ) / ( 218, 300 ) }
DATA {HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
#000:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dio.c
line 171 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dio.c
line 544 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#002:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dchunk.c
line 2050 in H5D__chunk_read(): unable to read raw data chunk
major: Low-level I/O
minor: Read failed
#003:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Dchunk.c
line 3405 in H5D__chunk_lock(): data pipeline read failed
major: Data filters
minor: Filter operation failed
#004:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5Z.c
line 1349 in H5Z_pipeline(): required filter 'blosc' is not
registered
major: Data filters
minor: Read failed
#005:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5PL.c
line 389 in H5PL_load(): search in paths failed
major: Plugin for dynamically loaded library
minor: Can't get value
#006:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\src\H5PL.c
line 812 in H5PL__find(): can't open directory
major: Plugin for dynamically loaded library
minor: Can't open directory or file
h5dump error: unable to print data
}
}
}
}
H5tools-DIAG: Error detected in HDF5:tools (1.10.1) thread 0:
#000:
C:\autotest\hdf5110-StdRelease-code-10vs14\build\hdfsrc\tools\lib\h5tools_dump.c
line 1639 in h5tools_dump_simple_dset(): H5Dread failed
major: Failure in tools library
minor: error in function
I suppose the meaningful part is here:
10vs14\build\hdfsrc\src\H5PL.c line 812 in H5PL__find():
can't open directory
b
ut I have no clues on 1) what directory the dynamic plugin
machinery in HDF5 is looking at and 2) which name the DLL
should have (as per manual, I am using libH5Zblosc.dll, but
I have tried with H5Zblosc.dll and blosc.dll with no success).
I have also tried to use the HDF5_PLUGIN_PATH environment
variable, so as to direct the plugin machinery in HDF5 to
look into the specific
%ALLUSERSPROFILE%/hdf5/lib/plugin
place,
but no luck.
I tried with HDF5 1.8.12 and 1.10.1, with same results.
Finally, if it is of any help, I am using Windows 10 64 bit.
As a suggestion, it may help to debug situations like this
if the HDF5 backtrace of the plugin machinery would provide
info about 1) which directory it is looking at for finding
the HDF5 plugin and 2) the name of the DLL that it is
trying to load.
Thanks in advance,
--
Francesc Alted
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
--
Francesc Alted
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
--
Francesc Alted
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
