On 3/15/21 8:18 AM, Richard Henderson wrote:
> On 3/15/21 5:33 AM, Eric Blake wrote:
>> On 3/13/21 3:48 PM, Richard Henderson wrote:
>>> On 3/11/21 2:07 PM, Eric Blake wrote:
+ /*
+ * Add in a fudge-factor (2^53 when double is IEEE format) for
+ * all scales
On 3/15/21 5:33 AM, Eric Blake wrote:
On 3/13/21 3:48 PM, Richard Henderson wrote:
On 3/11/21 2:07 PM, Eric Blake wrote:
+ /*
+ * Add in a fudge-factor (2^53 when double is IEEE format) for
+ * all scales less than P (2^50), so that things like
+ * 12.345M with
On 3/13/21 3:48 PM, Richard Henderson wrote:
> On 3/11/21 2:07 PM, Eric Blake wrote:
>> + /*
>> + * Add in a fudge-factor (2^53 when double is IEEE format) for
>> + * all scales less than P (2^50), so that things like
>> + * 12.345M with unit 1000 produce 12345000
On Thu, Mar 11, 2021 at 02:07:02PM -0600, Eric Blake wrote:
> Not all floating point fractions are precise. For example, the two
> nearest 32-bit IEEE float values for 0.345 are 0.34498808 and
> 0.3452861, with the lower one being closer. When our scaling unit
> is 1000, that in turn can
On 3/11/21 2:07 PM, Eric Blake wrote:
+/*
+ * Add in a fudge-factor (2^53 when double is IEEE format) for
+ * all scales less than P (2^50), so that things like
+ * 12.345M with unit 1000 produce 12345000 instead of
+ * 12344999.
+ */
+if
Not all floating point fractions are precise. For example, the two
nearest 32-bit IEEE float values for 0.345 are 0.34498808 and
0.3452861, with the lower one being closer. When our scaling unit
is 1000, that in turn can produce instances of double rounding (our
first when truncating to
