Re: [Pgfplots-features] rounding error problem in pgfplotstable

2012-09-18 Thread Christian Feuersaenger
He Freddie, glad it could help - perhaps it is useful for others as well. It might even be possible to use the results to increase the precision a little bit... but not in the next weeks :) Best regards Christian PS Good luck for your thesis! Am 17.09.2012 13:26, schrieb Frederik Heber: > H

Re: [Pgfplots-features] rounding error problem in pgfplotstable

2012-09-17 Thread Frederik Heber
Hello Christian, many thanks for taking some of your time to look into my question! I was aware of the limited precision problem as the manual states a relative precision of the fpu of 1e-4 to 1e-6 (p.32). However, I was wondering why the "wrong" result of 2e-5 and not (correctly rounded) 1e-5 oc

Re: [Pgfplots-features] rounding error problem in pgfplotstable

2012-09-15 Thread Christian Feuersaenger
Hi Freddie, If I understand correctly, your question can be reduced to "Why is the precision in the following minimal example so low?" with the minimal example \documentclass{article} \usepackage{pgfplots} \begin{document} \pgfkeys{/pgf/fpu} %\pgfmathfloatsetextprecision{3}% \pgfmathparse{