[
https://issues.apache.org/jira/browse/LUCENE-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226503#comment-15226503
]
Nicholas Knize edited comment on LUCENE-7179 at 4/5/16 3:51 PM:
----------------------------------------------------------------
bq. I think if we are changing it to use 64-bits, then we should also fix the
current rounding and overflow issues too?
In the Lucene-7165 patch the encoding tolerance is changed to be consistent
1e-7. Indeed the instability occurs > 1e-7. You can see the issue present
itself in {{scaleLon}} when the 32bit quantization is "truncated" to a long.
Overflow is not a problem because the positive values are mapped using {{(0x1L
<< 32) - 1}} so 360.0 correctly maps to 0xFFFFFFFF / 4294967295. Not sure what
you mean by "fix the rounding issue"?
was (Author: nknize):
bq. I think if we are changing it to use 64-bits, then we should also fix the
current rounding and overflow issues too?
In the Lucene-7165 patch the encoding tolerance is changed to be consistent
1e-7. Indeed the instability occurs > 1e-7. You can see the issue present
itself in {{scaleLon}} when the 32bit quantization is "truncated" to a long.
Overflow is not a problem because the positive values are mapped using {{0x1L
<< 32) - 1}} so 360.0 correctly maps to 0xFFFFFFFF / 4294967295. Not sure what
you mean by "fix the rounding issue"?
> GeoPoint and LatLonPoint test data should quantize once
> -------------------------------------------------------
>
> Key: LUCENE-7179
> URL: https://issues.apache.org/jira/browse/LUCENE-7179
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Nicholas Knize
>
> {{LatLonPoint}} and {{GeoPointField}} tests pre quantizes test data to ensure
> consistency with indexed (encoded) data. The pre quantized data then becomes
> indexed, undergoing another quantization. To guarantee numerical stability
> this should be changed such that the test data is quantized after indexing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
