[
https://issues.apache.org/jira/browse/CASSANDRA-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547567#comment-14547567
]
Stefania commented on CASSANDRA-9224:
-------------------------------------
Thanks for the comments, I've created a new patch for 2.1, see link attached.
I'm also using python 2.7.6, on Ubuntu trusty. The tests are calibrated for the
current default precision of 5, for which there are no rounding errors on my
box.
For the single precision float (val2) I had to increase the float precision to
about 7 or 8 in order to start getting rounding errors and for the double
precision (val1) I started getting rounding errors with a precision of around
15 or 16. I think this is expected.
Are you getting rounding errors only with a precision of 16 or even lower? And
do you want to support a precision higher than 5? If so, I don't think it makes
much sense to support more than 6 for floats, but for doubles we can increase
to say about 12 (but we'd have to enhance the patch further to separate the
formatting of floats and doubles).
> Figure out a better default float precision rule for cqlsh
> ----------------------------------------------------------
>
> Key: CASSANDRA-9224
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9224
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Tyler Hobbs
> Assignee: Stefania
> Labels: cqlsh
> Fix For: 3.x, 2.1.x, 2.2.x
>
>
> We currently use a {{DEFAULT_FLOAT_PRECISION}} of 5 in cqlsh with formatting
> {{'%.*g' % (float_precision, val)}}. In practice, this is way too low. For
> example, 12345.5 will show up as 123456. Since the float precision is used
> for cqlsh's COPY TO, it's particularly important that we maintain as much
> precision as is practical by default.
> There are some other tricky considerations, though. If the precision is too
> high, python will do something like this:
> {noformat}
> > '%.25g' % (12345.5555555555555555,)
> '12345.55555555555474711582'
> {noformat}
> That's not terrible, but it would be nice to avoid if we can.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
