Hi Quincey,
thank you for your answer. I tried not setting a fill value, but I think the
dataset is not valid than. I could not figure out how to identify not valid
chunks. Also HDFView was not able to read those files. Therefore I suppose it
is not the common way to use chunked datasets. Isn't it?
Jan
On 09.03.2010, at 18:20, Quincey Koziol wrote:
> Hi Jan,
>
> On Mar 9, 2010, at 5:22 AM, Jan Linxweiler wrote:
>
>> It seems like simply enabling compression does not change anything. The file
>> sizes for sparse and dense matrices still have the same size.
>>
>> Can anyone give me a hint on how to work this out?
>
> Hmm, I would think that you are correct in your expectations. Can you
> try without setting the fill value and see what happens?
>
> Quincey
>
>> On 09.03.2010, at 12:01, Jan Linxweiler wrote:
>>
>>> Hallo all,
>>>
>>> I have a question concerning the allocated file space of chunked datasets.
>>> I use a chunked dataset with a fill value and incremental allocation time
>>> as followed:
>>>
>>> hsize_tchunk_dims[3]={10,10,10};
>>> const int rank = 3;
>>>
>>> H5::DSetCreatPropList cparms;
>>> cparms.setChunk( rank, chunk_dims );
>>>
>>> /* Set fill value for the dataset. */
>>> double fill_val = -999.999;
>>> cparms.setFillValue( datatype, &fill_val );
>>>
>>> /* Set allocation time. */
>>> cparms.setAllocTime(H5D_ALLOC_TIME_INCR);
>>>
>>> /*
>>> * create dataspace with min/max dimensions.
>>> */
>>> hsize_t min_dims[] = {10000,1000,1000};
>>>
>>> hsize_t max_dims[] = {
>>> H5S_UNLIMITED,
>>> H5S_UNLIMITED,
>>> H5S_UNLIMITED
>>> };
>>>
>>> H5::DataSpace dataspace( rank, min_dims, max_dims );
>>>
>>> ....
>>>
>>> As I understand, memory is only allocated for chunks where data is actually
>>> written to. In other words, no data is allocated for chunks that contain
>>> only fill values. My question is, is this also true for the file space on
>>> the disk? My observance is, that memory for the whole dataset (also "empty"
>>> chunks) is allocated on the disk. I compared sparse matrices with full
>>> matrices and the allocated memory is nearly identical. Is there a way to
>>> reduce the size of sparse matrices on the disc? I am thinking of using
>>> compression. Is this a common procedure to achive this, or do you recommend
>>> something different?
>>>
>>> Thank you in advance,
>>>
>>> Jannis
>>>
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
