The documentation at http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html#Scaleoffset states:
"The relative error for each value should be no more than 5* (10^(D-scaling factor +1))." Based on the description of the D-scaling algorithm, it seems the above statement is incorrect (scale factor of 2 produces relative error of 5000?). If I have an array of floats "a", the round trip produces an error of: a-(round(a-min(a))*10^scalefactor)/10^scalefactor+min(a)) Testing yields an *absolute* error (not relative error) of 5*10^-(scalefactor+1) (note the minus sign). Could someone please comment? This question is related to a contribution to h5py, we just want to get the documentation right before releasing it (see https://github.com/h5py/h5py/pull/255). Thank you, Darren _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
