It's also said here, at the end of file : https://www.cs.auckland.ac.nz/~patrice/210LN/DR4.pdf
"add 1 to the left, with the binary point" 0.10000..... ________________________________ From: CCP4 bulletin board <[email protected]> on behalf of Zhijie Li <[email protected]> Sent: Tuesday, November 13, 2018 7:43 PM To: [email protected] Subject: Re: [ccp4bb] VERY old mtz file.. Hi all, I think I know why it is a division of 4 instead of 2 is involved in conversion from VAX to IEEE now. Short answer: a 2 is in the exponent bits (bias of 128 instead of 127, visible), another 2 is hidden in the scientific notation. I found this explanation+example on VAX F-float: http://courseweb.stthomas.edu/tpsturm/private/notes/qm300/FLOATPT.html So for IEEE754 float32, if we want to represent a same 12.75 (1100.11) in the above example, we would first conceptually write it in scientific notation as 1.10011 x 1000 in binary. Then the mantissa part is the part after the dot filled with zero to 23 bits: '10011000000000000000000', the exponent part is 3+127=130 (dec)=10000010(bin). Then the binary IEEE754 float32 number is 0[10000010][10011000000000000000000]. (You can check it here: https://www.h-schmidt.net/FloatConverter/IEEE754.html) Now compare this with the VAX 12.75 in the linked example, we can find that besides the bias becoming 128, the conceptual binary scientific notation is actually 0.110011 x 10000, instead of 1.10011 x 1000. So the exponent needed is 4 instead of 3. Then the exponent bits are 4+128=132=10000100 and the VAX float32 becomes 0[10000100][10011000000000000000000] ---if we write in a IEEE-style order. Note that the mantissa appears to be same as the ieee mantissa, and the exponent to be applied is 132-128=4. If this number is interpreted as IEEE754, then it will be 1.10011 x 2exp(132-127)=1.10011 x 100000, four times of what it should be. So, for normalised values, rearranging the VAX F-float bytes, reading as IEEE, then dividing by 4 gives the correct value. (The C[0]-1 treatment in the ccp4 lib is neat.) In this link describing VAX floats, it is unfortunate that it only states that the bias for F-float is 128, but not that the mantissa starts from 0.01 instead of 0.1. Therefore the confusion. https://nssdc.gsfc.nasa.gov/nssdc/formats/VAXFloatingPoint.htm Thanks to all responded! Zhijie ________________________________ From: Ian Tickle <[email protected]> Sent: Tuesday, November 13, 2018 4:54 PM To: Zhijie Li Cc: [email protected] Subject: Re: [ccp4bb] VERY old mtz file.. Hi Zhijie It's definitely a factor 4. The code is in subroutine QTIEEE in the Fortran source I mentioned previously at this line: See line: A(I)=((A(I)+SIGN(2,A(I)))/4.AND..NOT.MNAN).OR.MDN2 If you prefer it in C code it's in function vaxF2ieeeF in: ccp4-7.0-src/checkout/libccp4/ccp4/vmslibrary.c See line: out.c[0] = buffer[i].c[1] - (uint8)1; /* subtracts 2 from exponent */ i.e. subtract 2 from exponent -> division by 4. Cheers -- Ian On Tue, 13 Nov 2018 at 19:52, Zhijie Li <[email protected]<mailto:[email protected]>> wrote: If somebody is going to send these files by email, please send one to me too. Thanks in advance. I actually prefer to get a MTZ file because the miller indices would serve as good clues for understanding the encodings. Even the first 1024 bytes of an MTZ would do (data array starts at byte 80 in MTZ). In my life I had only seen ieee754. According to what I can find, VAX has an exponent bias of 128 (ieee754 uses 127). Then it seems to me that when converting from vax to ieee a division of 2 is involved. However all procedures I have seen use a division of 4, which is quite puzzling to me. A real data file containing meaningful numbers (eg., HKL indices) would be very helpful. Thanks in advance. Zhijie > On Nov 13, 2018, at 2:21 PM, Johan Hattne > <[email protected]<mailto:[email protected]>> wrote: > > Related by not exactly on topic: would anybody on the list be able to share > old map files (not MTZ:s) with Convex, Cray, Fujitsu, or VAX reals/strings? > I’d be interested to see what those files actually look(ed) like. > > // Best wishes; Johan > >> On Nov 9, 2018, at 18:38, Zhijie Li >> <[email protected]<mailto:[email protected]>> wrote: >> >> Hi all, >> >> On linux there are a few good GUI HEX editors. Here I’d like to recommend >> BLESS, which conveniently displays all possible numerical interpretations of >> the four bytes under cursor. It also allows the user to switch between big >> endian or little endian through a checkbox. Unfortunately all floats are >> assumed to be IEEE754, therefore VAX floats won’t be interpreted correctly. >> ( The simplest way to convert vax to ieee float would be to write a little >> program to do some bit operations. I’d be happy to take that as my weekend >> project) >> >> >> BTW, along the line of space efficiency, I can’t help noticing that the >> miller indices are saved as float32 in mtz, as all other numbers in mtz. >> This certainly have made mtz format a beautiful homogeneous data format ;). >> In this particular case, if we have doubts about the reliability of the >> machine stamp, trying to restore the miller indices would be a good way to >> test hypotheses. >> >> Zhijie >> >>> On Nov 9, 2018, at 9:04 PM, James Holton >>> <[email protected]<mailto:[email protected]>> >>> wrote: >>> >>> As a beamline scientist I must say I am glad that diffraction image data is >>> not usually stored as ASCII text. In fact, I am slowly warming to the idea >>> of storing it as not just binary, but compressed formats. Problem, I'm >>> sure will be that it won't be long before we forget how to decompress >>> them, as most of the algorithms we are using aren't all that widespread. >>> Probably around the same time future generations will curse us for using >>> ASCII instead of unicode, which is a 16-bit standard. I'm sure we will be >>> reviled for limiting ourselves so, just to save a factor of two in disk >>> space. >>> In situations like this I always use the unix "od" command. It makes >>> everything "human readable" by converting the bytes into strings you can >>> read. Then it is just a matter of figuring out what the bytes are. >>> Unfortunately, "od" only decodes floats on the native platform, so if the >>> mtz is from another platform (Windows vs Linux, for example), then you >>> might need to do some swapping. Thus far, I have encountered files that >>> require one of a few swapping strategies in order to make them work: >>> >>> 1 2 3 4 - no swapping >>> >>> 4 3 2 1 - reverse all bytes >>> >>> 3 4 1 2 - swap words and swap bytes within the words >>> 2 1 4 3 - reverse of previous >>> >>> 2-1 1 4 3 - same as last, but if not all zero, decrement byte #2 before >>> swapping >>> 3 4 1 2+1 - same as 3412, but if not all zero increment byte #2 before >>> swapping >>> I'm sure there are other combinations, but the oldest MTZ I have is only >>> from 1996. >>> >>> -James Holton >>> MAD Scientist >>> >>> >>>> On 11/9/2018 4:47 AM, Eleanor Dodson wrote: >>>> Anyone any idea what to do about this?? Created in 1992!! >>>> Seems unreadable.. >>>> >>>> No CTYP lines input for file: 1 >>>> Indices output even if all data items flagged "missing" >>>> Warning, NOT all LABOUT data lines given >>>> Warning: Machine stamp corrupted? Assuming native format. >>>>>>>>>> CCP4 library signal library_file:End of File (Error) >>>> >>>> >>>> To unsubscribe from the CCP4BB list, click the following link: >>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 >>>> >>> >>> >>> To unsubscribe from the CCP4BB list, click the following link: >>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 >>> >> >> To unsubscribe from the CCP4BB list, click the following link: >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 >> > > Research Specialist @ Gonen Lab > ____________________________________________________ > UCLA * 615 Charles E. Young Drive South > BSRB #347 * Los Angeles, CA 90095 > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 ________________________________ To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
