Hi Jan,
On Mar 9, 2010, at 5:22 AM, Jan Linxweiler wrote:
> It seems like simply enabling compression does not change anything. The file
> sizes for sparse and dense matrices still have the same size.
>
> Can anyone give me a hint on how to work this out?
Hmm, I would think that you are correct in your expectations. Can you
try without setting the fill value and see what happens?
Quincey
> On 09.03.2010, at 12:01, Jan Linxweiler wrote:
>
>> Hallo all,
>>
>> I have a question concerning the allocated file space of chunked datasets. I
>> use a chunked dataset with a fill value and incremental allocation time as
>> followed:
>>
>> hsize_tchunk_dims[3]={10,10,10};
>> const int rank = 3;
>>
>> H5::DSetCreatPropList cparms;
>> cparms.setChunk( rank, chunk_dims );
>>
>> /* Set fill value for the dataset. */
>> double fill_val = -999.999;
>> cparms.setFillValue( datatype, &fill_val );
>>
>> /* Set allocation time. */
>> cparms.setAllocTime(H5D_ALLOC_TIME_INCR);
>>
>> /*
>> * create dataspace with min/max dimensions.
>> */
>> hsize_t min_dims[] = {10000,1000,1000};
>>
>> hsize_t max_dims[] = {
>> H5S_UNLIMITED,
>> H5S_UNLIMITED,
>> H5S_UNLIMITED
>> };
>>
>> H5::DataSpace dataspace( rank, min_dims, max_dims );
>>
>> ....
>>
>> As I understand, memory is only allocated for chunks where data is actually
>> written to. In other words, no data is allocated for chunks that contain
>> only fill values. My question is, is this also true for the file space on
>> the disk? My observance is, that memory for the whole dataset (also "empty"
>> chunks) is allocated on the disk. I compared sparse matrices with full
>> matrices and the allocated memory is nearly identical. Is there a way to
>> reduce the size of sparse matrices on the disc? I am thinking of using
>> compression. Is this a common procedure to achive this, or do you recommend
>> something different?
>>
>> Thank you in advance,
>>
>> Jannis
>>
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
