https://bugs.documentfoundation.org/show_bug.cgi?id=105657
Bug ID: 105657
Summary: Option 'Precision as shown' working absurdly with
formats from the 'Fraction' category..
Product: LibreOffice
Version: 3.3.0 release
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: medium
Component: Calc
Assignee: [email protected]
Reporter: [email protected]
(Ex ante: The option 'Precision as shown' is a dangerous tool to do things that
should be done thoroughly using explicit rounding sloppily when in a lazy mood
. The option should never have been introduced. I strongly discourage everybody
to use it. However, it exists, and thus it should work as reliably and sensibly
as possible. Unfortunately the option was also entitled to work on cells whose
numeric values are formatted the 'Fraction' way. This may be considered insane
from the beginning. It's a fact nonetheless.)
The precision a fraction can achieve is limited by the maximum denominator
allowed.
The worst case of the length s of a range of values within the interval from 0
through 1 that cannot be distinguished by fractions with a maximum denominator
of M allowed should be: s = 1/(2*(M - 1)) .
For the often used
M = 9 this is 1/16
M = 99 1/196
M = 999 1/1996 ...
To get things more familiar - and keeping the secure side - we may consider an
inaccuracy of roughly 10^(-D) if D is the number of the nines in 9, 99, 999
..., or, turned the other way, if the maximum denominator is (10^D - 1).
OK. Let's count the question marks in the formal denominator of the format code
and we have a "precision" applicable if 'Precision as shown' is selected.
What Calc seemingly does in fact is to count the number N of positions in the
formal numerator and to set the "precison" for the cell to 10^(-N).
The 'Precision as shown' actually is applied to the intermediary result in
advance of setting theCell.Value (and of formatting this for theCell.String).
Thus we may enter the formula =1/3 in a cell formatted with the code # ?/???
and get shown 2/7 if 'Precision as shown' is ON.
This behaviour is inherited from OOo. It should be changed nonetheless.
--
You are receiving this mail because:
You are the assignee for the bug._______________________________________________
Libreoffice-bugs mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
