Hi Quincey,
I tried again.. I did not set the the fill value but set the allocation time to
be incremental. In this case the fill value equals 0... The file size is the
same...
Jan
On 09.03.2010, at 18:45, Quincey Koziol wrote:
> Hi Jan,
>
> On Mar 9, 2010, at 11:42 AM, Jan Linxweiler wrote:
>
>> Hi Quincey,
>>
>> thank you for your answer. I tried not setting a fill value, but I think the
>> dataset is not valid than. I could not figure out how to identify not valid
>> chunks. Also HDFView was not able to read those files. Therefore I suppose
>> it is not the common way to use chunked datasets. Isn't it?
>
> When you didn't set the fill value, was the file smaller (because the
> chunks weren't allocated)? If that's true, then I think this is a bug in the
> HDF5 library and we should fix it so that it doesn't allocate chunks (when
> the allocation time is increment or late) when a fill value is defined.
>
> Quincey
>
>> Jan
>>
>> On 09.03.2010, at 18:20, Quincey Koziol wrote:
>>
>>> Hi Jan,
>>>
>>> On Mar 9, 2010, at 5:22 AM, Jan Linxweiler wrote:
>>>
>>>> It seems like simply enabling compression does not change anything. The
>>>> file sizes for sparse and dense matrices still have the same size.
>>>>
>>>> Can anyone give me a hint on how to work this out?
>>>
>>> Hmm, I would think that you are correct in your expectations. Can you
>>> try without setting the fill value and see what happens?
>>>
>>> Quincey
>>>
>>>> On 09.03.2010, at 12:01, Jan Linxweiler wrote:
>>>>
>>>>> Hallo all,
>>>>>
>>>>> I have a question concerning the allocated file space of chunked
>>>>> datasets. I use a chunked dataset with a fill value and incremental
>>>>> allocation time as followed:
>>>>>
>>>>> hsize_tchunk_dims[3]={10,10,10};
>>>>> const int rank = 3;
>>>>>
>>>>> H5::DSetCreatPropList cparms;
>>>>> cparms.setChunk( rank, chunk_dims );
>>>>>
>>>>> /* Set fill value for the dataset. */
>>>>> double fill_val = -999.999;
>>>>> cparms.setFillValue( datatype, &fill_val );
>>>>>
>>>>> /* Set allocation time. */
>>>>> cparms.setAllocTime(H5D_ALLOC_TIME_INCR);
>>>>>
>>>>> /*
>>>>> * create dataspace with min/max dimensions.
>>>>> */
>>>>> hsize_t min_dims[] = {10000,1000,1000};
>>>>>
>>>>> hsize_t max_dims[] = {
>>>>> H5S_UNLIMITED,
>>>>> H5S_UNLIMITED,
>>>>> H5S_UNLIMITED
>>>>> };
>>>>>
>>>>> H5::DataSpace dataspace( rank, min_dims, max_dims );
>>>>>
>>>>> ....
>>>>>
>>>>> As I understand, memory is only allocated for chunks where data is
>>>>> actually written to. In other words, no data is allocated for chunks that
>>>>> contain only fill values. My question is, is this also true for the file
>>>>> space on the disk? My observance is, that memory for the whole dataset
>>>>> (also "empty" chunks) is allocated on the disk. I compared sparse
>>>>> matrices with full matrices and the allocated memory is nearly identical.
>>>>> Is there a way to reduce the size of sparse matrices on the disc? I am
>>>>> thinking of using compression. Is this a common procedure to achive this,
>>>>> or do you recommend something different?
>>>>>
>>>>> Thank you in advance,
>>>>>
>>>>> Jannis
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> [email protected]
>>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> [email protected]
>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
