On Fri, Aug 10, 2012 at 1:47 PM, David Smiley (@MITRE.org) <dsmi...@mitre.org> wrote: > Information I've read vary on exactly what is the accuracy of float > vs double but at a kilometer there's no question a double is overkill.
Back of the envelope: 23 mantissa bits + 1 implied bit == 24 effective mantissa bits in a 32 bit float. 40,000 km circumference / (2^24) = .0024 km (i.e. our resolution at the equator is 2.4m at best - there will be some lost unused space at the beginning and end of the +-180 number-line). Is that in line with what you've read? -Yonik http://lucidworks.com
