Hi Freddie,
If I understand correctly, your question can be reduced to "Why is the
precision in the following minimal example so low?" with the minimal example
\documentclass{article}
\usepackage{pgfplots}
\begin{document}
\pgfkeys{/pgf/fpu}
%\pgfmathfloatsetextprecision{3}%
\pgfmathparse{ -313.3008711--313.3008819}\pgfmathresult
\pgfmathparse{ 9+9}\pgfmathresult
\end{document}
the outcome of that example is
1Y2.0e-5]
1Y1.8e1]
where "1" = "+", "Y" means "here starts the mantissa" and "e" means
"here starts the exponent".
Please scream if that is not what you had in mind...
IF that is your question, than it is a well-known issue: it is
cancellation of significant digits and is inherent to every processing
unit which operates with limited precision. However, it is completely
independent of pgf(plots): it happens for *every* numerical program.
You are loosing (almost all) significant digits and you boost the
relative error.
Take, for example
#include <iostream>
using namespace std;
int main(int argc, char** argv)
{
float A = -313.3008711f;
float B = -313.3008819f;
float C = A-B;
double Ad = -313.3008711;
double Bd = -313.3008819;
double Cd = Ad-Bd;
cout.precision(16);
cout << scientific;
cout << "float : " << A << " - " << B << " =" << C <<endl;
cout << "double: " << Ad << " - " << Bd << " =" << Cd <<endl;
return 0;
}
which has the outcome
float : -3.1330087280273438e+02 - -3.1330087280273438e+02
=0.0000000000000000e+00
double: -3.1330087109999999e+02 - -3.1330088189999998e+02
=1.0799999984101305e-05
With that in mind, we can say that this example of the floating point
unit has a precision between that of float and double.
In fact, if you activate the line
\pgfmathfloatsetextprecision{3}%
in my experiment, you get 1Y1.0e-5] for your experiment. However, this
will try to compute "9+9" by means of "9000+9000" which is beyond TeX's
capacities. (The default (and maximum) value is
\pgfmathfloatsetextprecision{2}).
Solution: there are two solutions:
(a) use a different mathematical engine which has higher precision. This
is not a real solution as it does not fix the systematical problem.
(b) try to rephrase the equation such that you do not need to subtract
numbers which are almost equal. Might be difficult, I agree.
Best regards
Christian
Am 13.09.2012 21:02, schrieb Frederik Heber:
> Hello,
>
> I am typesetting a table with pgfplotstable and stumbling over some
> weird rounding mistakes, seemingly on the part of pgfplotstable. Any
> help is appreciated.
>
> I am calculating some "energy" that should converge to the correct value
> at some specific "level" (see example below). In order to asses this
> convergence I calculate the absolute difference between the "energy" on
> this specific "level" against the "energy" on current one "level".
> I notice some weird rounding mistakes in the table values that I cannot
> explain: The "diff" value of "level" 6 should read (at least) 1.e-5 not
> 2e-5, if not even -313.3008711--313.3008819 = 1.08e-5.
>
> I am using pgf version 2.10-cvs and updated to pgfplots 2012/08/23
> v1.6.1 (git show 1.6.1 ) (downloaded today from sourceforge), both
> version numbers are taken directly from the pdflatex log.
>
> On a further note: I searched the (excellent) manual, stumbling over
> /pgfkeys{/pgf/fpu=true}. However, as the fpu is always activated when
> using "expr" this does not seem to be the issue?
>
> Here is the minimum example (sorry for the messy table creations but I
> don't want to bother you with files, it's the same result anyway):
> ======================= SNIP ===========================
> \documentclass{article}
>
> \usepackage{tikz}
> \usepackage{pgfplots}
> \usepackage{pgfplotstable}
>
> \begin{document}
>
> Here is a minimum example:
> % just create a list of example values
> \pgfplotstableset{
> create on use/a/.style={create col/set list={1,2,3,4,5,6,7}},
> create on use/b/.style={create col/set
> list={-321.4430756,-313.2926882,-313.3037961,-313.3012453,-313.3008931,-313.3008819,-313.3008711}},
> }
> % create the table
> \pgfplotstablenew[columns={a,b}]{7}\supertable
> % get the minimum value from level "7" (index starts at 0)
> \pgfplotstablegetelem{6}{b}\of\supertable
> \let\amin=\pgfplotsretval
> % prepare creation of one more row "diff" that is the absolute
> difference between the minimum value and the current one
> \pgfplotstableset{
> create on use/diff/.style={
> create col/expr={
> abs(\amin-\thisrow{b})
> }
> },
> }
> % create the complete table with the difference column
> \pgfplotstablenew[columns={a,b,diff}]{7}\moresupertable
> % print the table
> \pgfplotstabletypeset[
> columns={a,b,diff},%
> columns/a/.style={
> column name={level},
> precision=6,
> },
> columns/b/.style={
> column name={energy},
> precision=6,
> },
> columns/diff/.style={
> column name={diff},
> precision=6,
> },
> ]{\moresupertable}
>
> \end{document}
>
> ======================= SNIP ===========================
>
> Regards and thanks,
>
> Frederik
------------------------------------------------------------------------------
How fast is your code?
3 out of 4 devs don\\\'t know how their code performs in production.
Find out how slow your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219672;13503038;z?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Pgfplots-features mailing list
Pgfplots-features@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pgfplots-features
