FYI I asked this question on stackoverflow.
http://stackoverflow.com/questions/18668871/accessing-long-double-bit-representation
On 7 September 2013 10:04, Andrea Bedini <andrea.bed...@gmail.com> wrote:
> Hi Ray,
>
> yes, this is compatible with what I observed in my tests. Honestly I am
> not sure the problem has a simple solution at all. If the standard doesn't
> guarantee us the pad bit have a consistent value, there's no way we can
> expect this to work in a portable way: at some point in the future a
> compiler will be smart enough to revert all the obstacles we throw at its
> way.
>
> If I may, I would suggest we replace the entire mechanism with something
> simpler. There are not that many floating point formats floating around
> (pardon the pun), even considering the many architectures. Can't we just
> hardcode them? The configuration system has surely enough information to
> determine the native floating point format without bit fiddling.
>
> What are the supported architectures?
>
> Andrea
>
> PS: it seems the disappearing memset problem can be solved by asking gcc
> to not replace memset with its builtin version, i.e. passing the option
> -fno-builtin-memset, but this of course won't work with other compilers.
>
>
> On 7 September 2013 08:24, Raymond Lu <songy...@hdfgroup.org> wrote:
>
>> Andrea,
>>
>> My coworker Neil helped me in this afternoon to find out that when GCC
>> 4.8 compiler assigns constant values to variables (value1 and value2) like
>> this,
>>
>> for(i = 0, value1 = 0.0, value2 = 1.0; i < (int)sizeof(long double);
>> i++) {
>> value3 = value1;
>> value1 += value2;
>> value2 /= 256.0;
>> :
>> :
>>
>> it introduces some garbage to the two padding bytes of value1 and value2.
>> Then the garbage confuses our algorithm, especially the value of
>> "last_mbyte" gets wrong. To fix it in a simple way, use an intermediate
>> variable like this:
>>
>> long double tmp_value, divisor;
>>
>> tmp_value = 0.0;
>> value1 = tmp_value;
>> tmp_value = 1.0;
>> value2 = tmp_value;
>> tmp_value = 256.0;
>> divisor = tmp_value;
>>
>> for(i = 0; i < (int)sizeof(long double); i++) {
>> value3 = value1;
>> value1 += value2;
>> value2 /= divisor;
>> :
>> :
>>
>> How do you think about it?
>>
>> Ray
>>
>> On Sep 6, 2013, at 1:49 AM, Andrea Bedini wrote:
>>
>> Hi Rey,
>>
>> thanks for that, it really helped. I checked thoroughly and the memset of
>> the temporary variables disappears randomly.
>> It doesn't depends only on optimization though, on my machine putting a
>> printf("%Lf\n", value2); just before the loop changes the result.
>> I'm not sure who gets the blame here, poking into the padding bits of a
>> long double might just be unspecified or undefined behaviour.
>>
>> Andrea
>>
>>
>> On 6 September 2013 06:40, Raymond Lu <songy...@hdfgroup.org> wrote:
>>
>>> I isolated part of the DETECT_F into a C program as attached (detect.c).
>>> It only contains the algorithm for detecting the byte order of long
>>> double. When I compile it with gcc -g, -O0, or no flag, it reports
>>> little-endian. When I compile it with -O1, -O2, or -O3, it reports VAX
>>> order. I don't know where goes wrong yet. But I suspect GCC's
>>> optimization has bugs. Maybe you can help me.
>>>
>>> I haven't tried the algorithms for other parts in DETECT_F yet. The
>>> alignment problem you talked about is one of the other algorithms.
>>>
>>> Ray
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sep 4, 2013, at 8:40 PM, Andrea Bedini wrote:
>>>
>>> Thanks George.
>>>
>>> For anyone interested in debugging this problem, debian has an extensive
>>> collection of build logs over many architectures
>>> https://buildd.debian.org/status/logs.php?pkg=hdf5 (going back 12
>>> years!)
>>>
>>> As far as I know, the corruption is limited to the H5T_NATIVE_LDOUBLE
>>> type. You can check your particular build with the following test
>>>
>>> #include <hdf5.h>
>>> int main() {
>>> return !(H5Tget_order(H5T_NATIVE_LDOUBLE) == H5Tget_order(
>>> H5T_NATIVE_DOUBLE));
>>> }
>>>
>>> It exits with code 1 if the long double has different byte ordering than
>>> double (which is technically possible, but highly suspicious).
>>>
>>> Otherwise the patch I sent earlier in this thread seems to do the trick,
>>> although what exactly is going wrong is still beyond my understanding.
>>>
>>> Third option: you can define an equivalent of H5T_NATIVE_LDOUBLE
>>> yourself. The following creates a data type representing a long double as
>>> implemented by gcc on x86 architectures (see
>>> http://en.wikipedia.org/wiki/Long_double#Implementations for details)
>>>
>>> hid_t ldouble_datatype = H5Tcopy(H5T_NATIVE_DOUBLE);
>>> H5Tset_size(ldouble_datatype, sizeof(long double));
>>> H5Tset_precision(ldouble_datatype, 80);
>>> H5Tset_fields (ldouble_datatype, 79, 64, 15, 0, 64);
>>> H5Tset_pad(ldouble_datatype, H5T_PAD_ZERO, H5T_PAD_ZERO);
>>> H5Tset_inpad(ldouble_datatype, H5T_PAD_ZERO);
>>> H5Tset_ebias(ldouble_datatype, 16383);
>>> H5Tset_norm(ldouble_datatype, H5T_NORM_NONE);
>>>
>>> Best wishes,
>>> Andrea
>>>
>>> On 4 September 2013 22:58, George N. White III <gnw...@gmail.com> wrote:
>>>
>>>> Another historical reference to the obscurity of this code is: <
>>>> https://bugs.gentoo.org/show_bug.cgi?id=118777>.
>>>>
>>>> I've been building HDF5 libraries for use with NASA SeaDAS, and
>>>> recently have started using HDF5 with R and GDAL. The SeaDAS builds are
>>>> static, and I don't find the "unable to calculate alignment for long
>>>> double" message in my SeaDAS build logs on linux and OS X. For R and GDAL,
>>>> however, I need dynamic libraries and those build logs do have the "unable
>>>> to calculate alignment for long double" message on both linux and OS X.
>>>>
>>>>
>>>> On Tue, Sep 3, 2013 at 9:50 PM, Andrea Bedini
>>>> <andrea.bed...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I found something else (I know, I should stop :)). I am not entirely
>>>>> sure but it seems that when H5detect fails it writes "unable to calculate
>>>>> alignment for long double" on stderr so this message should be observable
>>>>> on build logs (although buried by other warnings). The packages on debian
>>>>> sid and testing for both i386 and x86-64 seem to be affected:
>>>>>
>>>>>
>>>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=amd64&ver=1.8.11-3%2Bb1&stamp=1377024563
>>>>>
>>>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=i386&ver=1.8.11-3%2Bb1&stamp=1377025110
>>>>>
>>>>> But here's the exciting part: look what I found
>>>>>
>>>>>
>>>>> http://www.unidata.ucar.edu/mailing_lists/archives/gembud/2010/msg00052.html
>>>>>
>>>>> It's a build log from 2010 for HDF5 v1.6.5 and gcc-4.4.3 that says "unable
>>>>> to calculate alignment for long double".
>>>>>
>>>>> If my understanding is correct, nor 1.8.11 or gcc 4.8.0 would be the
>>>>> problem and it would be that piece of code just doesn't work properly.
>>>>>
>>>>> Best wishes,
>>>>> Andrea
>>>>>
>>>>>
>>>>>
>>>>> On 4 September 2013 08:00, Andrea Bedini <andrea.bed...@gmail.com>wrote:
>>>>>
>>>>>> Hi Ray,
>>>>>>
>>>>>> thanks for giving it a look. Antonio made me notice that something
>>>>>> else might be at work since the macro DETECT_F already zeroes the
>>>>>> structure
>>>>>> right before anything else:
>>>>>>
>>>>>> memset(&INFO, 0, sizeof(INFO)); #L299
>>>>>>
>>>>>> so I don't understand how the perm fields need to be zeroed again
>>>>>> around line #L308. This still considering the "Byte Order" loop as a
>>>>>> black
>>>>>> box.
>>>>>>
>>>>>> As a side question: isn't there a more portable way of doing this? I
>>>>>> am pretty sure H5detect.c might invoke a bunch of undefined behaviours
>>>>>> given the amount of warning the compiler generates and of bit trickery.
>>>>>>
>>>>>> Best wishes,
>>>>>> Andrea
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4 September 2013 05:43, Raymond Lu <songy...@hdfgroup.org> wrote:
>>>>>>
>>>>>>> Andrea,
>>>>>>>
>>>>>>> We've verified that your solution is correct. We're putting your
>>>>>>> fix into the library. Thanks for helping us.
>>>>>>>
>>>>>>> Ray
>>>>>>>
>>>>>>> On Sep 3, 2013, at 3:32 AM, Andrea Bedini wrote:
>>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> I think I have found the problem. The issue is in H5detect.c.
>>>>>>> Macros DETECT_F and DETECT_I do not initialize properly the perm field
>>>>>>> in
>>>>>>> the detected_t struct. As a result the routine fix_order is passed some
>>>>>>> uninitialized memory which makes it fail. I have a small patch against
>>>>>>> H5detect.c which fixes the problem by simply initializing the perm field
>>>>>>> with zeros. Valgrind's tool memcheck would have exposed the problem.
>>>>>>>
>>>>>>> Best wishes,
>>>>>>> Andrea
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 3 September 2013 15:30, Andrea Bedini <andrea.bed...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am experiencing the following issue with hdf5 and gcc 4.8.0
>>>>>>>>
>>>>>>>> Consider this very simple test
>>>>>>>>
>>>>>>>> #include <hdf5.h>
>>>>>>>>
>>>>>>>> int main() {
>>>>>>>> switch (H5Tget_order(H5T_NATIVE_LDOUBLE)) {
>>>>>>>> case H5T_ORDER_LE:
>>>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE\n");
>>>>>>>> break;
>>>>>>>> case H5T_ORDER_BE:
>>>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_BE\n");
>>>>>>>> break;
>>>>>>>> case H5T_ORDER_VAX:
>>>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX\n");
>>>>>>>> break;
>>>>>>>> case H5T_ORDER_MIXED:
>>>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_MIXED\n");
>>>>>>>> break;
>>>>>>>> case H5T_ORDER_NONE:
>>>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_NONE\n");
>>>>>>>> break;
>>>>>>>> default:
>>>>>>>> printf("here are dragons\n");
>>>>>>>> }
>>>>>>>> return 0;
>>>>>>>> }
>>>>>>>>
>>>>>>>> on the same x86_64 GNU/Linux machine I get
>>>>>>>>
>>>>>>>> $ hdf5-1.8.11-gcc-4.7.0/my_test # compiled with gcc 4.7.0
>>>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE
>>>>>>>>
>>>>>>>> $ hdf5-1.8.11-gcc-4.8.0/my_test # compiled with gcc 4.8.0
>>>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX
>>>>>>>>
>>>>>>>> So H5T_NATIVE_LDOUBLE is mis-detected. I tried to dig deeper and
>>>>>>>> basically the fault must be in src/H5detect.c which is used to
>>>>>>>> generate the
>>>>>>>> definitions in src/H5Tinit.c
>>>>>>>> I could not figure out what H5detect.c does wrong (it is not very
>>>>>>>> readable, given its extensive use of macros) but the compiler does
>>>>>>>> emit a
>>>>>>>> lot of warnings (see https://gist.github.com/andreabedini/6419975).
>>>>>>>>
>>>>>>>> I think this must be related to the failure of dt_arith long double
>>>>>>>> test observed recently.
>>>>>>>>
>>>>>>>> Any suggestion on how to fix this ?
>>>>>>>>
>>>>>>>> Best wishes,
>>>>>>>> Andrea
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Andrea Bedini <andrea.bed...@gmail.com>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Andrea Bedini <andrea.bed...@gmail.com>
>>>>>>> <hdf5_uninitialized.patch>
>>>>>>> _______________________________________________
>>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>>> Hdf-forum@lists.hdfgroup.org
>>>>>>>
>>>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>>> Hdf-forum@lists.hdfgroup.org
>>>>>>>
>>>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andrea Bedini <andrea.bed...@gmail.com>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andrea Bedini <andrea.bed...@gmail.com>
>>>>>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> Hdf-forum@lists.hdfgroup.org
>>>>>
>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> George N. White III <aa...@chebucto.ns.ca>
>>>> Head of St. Margarets Bay, Nova Scotia
>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> Hdf-forum@lists.hdfgroup.org
>>>>
>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Andrea Bedini <andrea.bed...@gmail.com>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum@lists.hdfgroup.org
>>>
>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum@lists.hdfgroup.org
>>>
>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>
>>>
>>
>>
>> --
>> Andrea Bedini <andrea.bed...@gmail.com>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@lists.hdfgroup.org
>>
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@lists.hdfgroup.org
>>
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>
>
> --
> Andrea Bedini <andrea.bed...@gmail.com>
>
--
Andrea Bedini <andrea.bed...@gmail.com>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
