[
https://issues.apache.org/jira/browse/CALCITE-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615558#comment-16615558
]
Julian Hyde commented on CALCITE-2546:
--------------------------------------
It was a mistake for me to send the full precision of the {{double}} values to
JSON. (It almost always is, because the last few bits are noise. Same goes for
printing floating-point values and saving them in a database.)
Truncating to 3 decimal places is no more arbitrary than truncating to the
precision of one's 64 bit floating point unit, and a lot less likely to give
random failures. Rather than using floor and pow, a better fix would be to send
BigDecimal values to JSON.
If you agree, [~vladimirsitnikov], I'll make that change and commit.
> testProfileScott2 in ProfilerTest.java fail with java.lang.AssertionError
> -------------------------------------------------------------------------
>
> Key: CALCITE-2546
> URL: https://issues.apache.org/jira/browse/CALCITE-2546
> Project: Calcite
> Issue Type: Bug
> Components: core
> Environment: uname -a
> Linux pts00607-vm2 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:30:22 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
> Reporter: Alisha Prabhu
> Assignee: Julian Hyde
> Priority: Major
> Labels: ppc64le
> Attachments: Error.txt
>
>
> Maven command used in the core module : mvn
> -Dtest=ProfilerTest#testProfileScott2 test
> The values expectedCardinality and surprise for columns:[HIREDATE,COMM] in
> the test case differ on power platform.
> Expected values :
> expectedCardinality:12.682618485430247,surprise:0.4344728973121492
> On ppc64le :
> expectedCardinality:12.682618485430254,surprise:0.43447289731214944
> The test case has passed on x86_64.
> The attached Error.txt file has the error log seen on power.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
