[
https://issues.apache.org/jira/browse/OLINGO-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891980#comment-15891980
]
Svante von Erichsen commented on OLINGO-1087:
---------------------------------------------
OK, I think I get it.
First, any mathematical notion of mantissa and exponent seems misleading for
this (otherwise my first idea could seem to be right).
Instead, only look at the printed representation in radix 10, omitting all
leading zeroes before the decimal point and all trailing zeroes after it.
If scale is a nonnegative integer, it is the maximum count of digits after the
decimal point in that printed representation.
If scale is "variable" or omitted, precision is the count of digits in that
printed representation. If scale is a nonnegative integer, then precision
minus scale is the maximum count of digits before the decimal point.
Examples:
0.0302 fits in Precision=4, Scale=4, but not in Precision=3, Scale=4.
320 fits in Precision=3, Scale=0, but not in Precision=3, Scale=1
> Precision in EdmDecimal not calculated correctly
> ------------------------------------------------
>
> Key: OLINGO-1087
> URL: https://issues.apache.org/jira/browse/OLINGO-1087
> Project: Olingo
> Issue Type: Bug
> Components: odata2-core
> Affects Versions: V2 2.0.8
> Reporter: Svante von Erichsen
> Attachments: 0001-OLINGO-1087-Fix-calculation-of-precision.patch
>
>
> The "precision" found in a read number is currently at least the number of
> decimals (omitting trailing zeroes). That seems to be wrong, or at least not
> consistent with at least some server side view of the matter, for numbers
> smaller than 0.1 (the number shown has 35 decimals, but precision 34):
> org.apache.olingo.odata2.api.edm.EdmSimpleTypeException: The metadata
> constraints '[Precision=34]' do not match the literal
> '0.08102359900000000170194880411145277'.
> at
> org.apache.olingo.odata2.core.edm.EdmDecimal.internalValueOfString(EdmDecimal.java:107)
> at
> org.apache.olingo.odata2.core.edm.AbstractSimpleType.valueOfString(AbstractSimpleType.java:91)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonPropertyConsumer.readSimpleProperty(JsonPropertyConsumer.java:236)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonPropertyConsumer.readPropertyValue(JsonPropertyConsumer.java:169)
> ... 69 common frames omitted
> Wrapped by: org.apache.olingo.odata2.api.ep.EntityProviderException: An
> exception of type 'EdmSimpleTypeException' occurred.
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonPropertyConsumer.readPropertyValue(JsonPropertyConsumer.java:171)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonEntryConsumer.handleName(JsonEntryConsumer.java:172)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonEntryConsumer.readEntryContent(JsonEntryConsumer.java:130)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonEntryConsumer.readFeedEntry(JsonEntryConsumer.java:117)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readArrayContent(JsonFeedConsumer.java:153)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.handleName(JsonFeedConsumer.java:122)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readFeedContent(JsonFeedConsumer.java:111)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readFeed(JsonFeedConsumer.java:96)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readFeedStandalone(JsonFeedConsumer.java:63)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonEntityConsumer.readDeltaFeed(JsonEntityConsumer.java:95)
> at
> org.apache.olingo.odata2.core.ep.consumer.JsonEntityConsumer.readFeed(JsonEntityConsumer.java:81)
> at
> org.apache.olingo.odata2.core.ep.JsonEntityProvider.readFeed(JsonEntityProvider.java:309)
> at
> org.apache.olingo.odata2.core.ep.ProviderFacadeImpl.readFeed(ProviderFacadeImpl.java:165)
> at
> org.apache.olingo.odata2.api.ep.EntityProvider.readFeed(EntityProvider.java:676)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
