It's exciting to be able to show the collective filtered IO feature as
part of a full software stack. Thank you for your hard work on this,
and please let me know what more I can do to help keep it on glide
path for release.
On Thu, Nov 9, 2017 at 3:22 PM, Jordan Henderson
<jhender...@hdfgroup.org> wrote:
> Thanks! I'll discuss this with others and see what the best way to proceed
> forward from this is. I think this has been a very productive discussion and
> very useful feedback.
> ________________________________
> From: Michael K. Edwards <m.k.edwa...@gmail.com>
> Sent: Thursday, November 9, 2017 5:01:33 PM
> To: Jordan Henderson
> Cc: HDF Users Discussion List
> Subject: Re: [Hdf-forum] Collective IO and filters
>
> And here's the change to H5Z-blosc (still using the private H5MM APIs):
>
> diff --git a/src/blosc_filter.c b/src/blosc_filter.c
> index bfd8c3e..9bc1a42 100644
> --- a/src/blosc_filter.c
> +++ b/src/blosc_filter.c
> @@ -16,6 +16,7 @@
> #include <string.h>
> #include <errno.h>
> #include "hdf5.h"
> +#include "H5MMprivate.h"
> #include "blosc_filter.h"
>
> #if defined(__GNUC__)
> @@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
> /* We're compressing */
> if (!(flags & H5Z_FLAG_REVERSE)) {
>
> - /* Allocate an output buffer exactly as long as the input data; if
> - the result is larger, we simply return 0. The filter is flagged
> - as optional, so HDF5 marks the chunk as uncompressed and
> - proceeds.
> + /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes
> + larger than the input data, to accommodate the BLOSC header.
> + If compression with the requested parameters causes the data itself
> + to grow (thereby causing the compressed data, with header, to exceed
> + the output buffer size), fall back to memcpy mode (clevel=0).
> */
>
> - outbuf_size = (*buf_size);
> + outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;
>
> #ifdef BLOSC_DEBUG
> fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",
> nbytes, outbuf_size);
> #endif
>
> - outbuf = malloc(outbuf_size);
> + outbuf = H5MM_malloc(outbuf_size);
>
> if (outbuf == NULL) {
> PUSH_ERR("blosc_filter", H5E_CALLBACK,
> @@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>
> blosc_set_compressor(compname);
> status = blosc_compress(clevel, doshuffle, typesize, nbytes,
> - *buf, outbuf, nbytes);
> + *buf, outbuf, outbuf_size);
> + if (status < 0) {
> + status = blosc_compress(0, doshuffle, typesize, nbytes,
> + *buf, outbuf, outbuf_size);
> + }
> if (status < 0) {
> PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");
> goto failed;
> @@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
> /* declare dummy variables */
> size_t cbytes, blocksize;
>
> - free(outbuf);
> + H5MM_xfree(outbuf);
>
> /* Extract the exact outbuf_size from the buffer header.
> *
> @@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
> fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",
> nbytes, outbuf_size);
> #endif
>
> - outbuf = malloc(outbuf_size);
> + if (outbuf_size == 0) {
> + H5MM_xfree(*buf);
> + *buf = NULL;
> + *buf_size = outbuf_size;
> + return 0; /* Size of compressed/decompressed data */
> + }
> +
> + outbuf = H5MM_malloc(outbuf_size);
>
> if (outbuf == NULL) {
> PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate
> decompression buffer");
> @@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
> } /* compressing vs decompressing */
>
> if (status != 0) {
> - free(*buf);
> + H5MM_xfree(*buf);
> *buf = outbuf;
> *buf_size = outbuf_size;
> return status; /* Size of compressed/decompressed data */
> }
>
> failed:
> - free(outbuf);
> + H5MM_xfree(outbuf);
> return 0;
>
> } /* End filter function */
>
> On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson
> <jhender...@hdfgroup.org> wrote:
>> As the filtered collective path simply calls through the filter pipeline
>> by
>> way of the H5Z_pipeline() function, it would seem that either the filter
>> pipeline itself is not handling this case correctly, or this is somewhat
>> unexpected behavior for the pipeline to deal with.
>>
>>
>> Either way, I think a pull request/diff file would be very useful for
>> going
>> over this. If you're able to generate a diff between what you have now and
>> the current develop branch/H5Z-blosc code and put it here that would be
>> useful. I don't think that there should be too much in the way of
>> logistics
>> for getting this code in, we just want to make sure that we approach the
>> solution in the right way without breaking something else.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
