stevedlawrence commented on a change in pull request #387:
URL: https://github.com/apache/incubator-daffodil/pull/387#discussion_r442290976
##########
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/RuntimeData.scala
##########
@@ -455,10 +455,9 @@ final class SimpleTypeRuntimeData(
// Per http://www.w3.org/TR/xmlschema-2/#rf-totalDigits
// |i| < 10^totalDigits
- val number = new java.math.BigDecimal(scala.math.pow(10.0,
digits.doubleValue()))
- val biNumber = new java.math.BigInteger(number.intValueExact().toString())
+ val number = new java.math.BigDecimal(scala.math.pow(10,
digits)).toBigIntegerExact
val bdData = diNode.dataValueAsBigDecimal.unscaledValue()
- val isDataLessThanNumber = bdData.compareTo(biNumber) < 0
+ val isDataLessThanNumber = bdData.compareTo(number) < 0
isDataLessThanNumber
Review comment:
I thought some more about this, and I think the above logic
overcomplicates things. There's no need to compare against powers of 10. All
that is needed is bd.precision and bd.scala to figure out the total digits.
If scale <= 0, that means the number is of the from XXXXXYYYYY where the
number of X's is the precision and the number of Y's are the negative scale
(and all zeros). For example 1234*10^2 = 123400. In this example precision is 4
and scale is -2. So for this case, the total digits is just ``bd.precision -
bd.scale``.
If scale > 0, then the number is of the form (precision digits) * 10 ^
-scale, and we want whatever is larger of precision and scale. For example, if
precision was 4 and scale was 2, the number would be of the form XX.XX, and so
total digits is just the precision (or 4). On the other hand if precision was 2
and scale was 4, the number would be of the form 0.00XX, so we want scale this
time (4 again). So in this case, the total digits is just
Math.max(bd.precision, bd.scale).
I think was also need to sort of normalize the big decimal by stripping off
trailing zeros. Otherwise something like 10.000 throws off the logic. This is
because this is represented as a precision of 5 and scale of 3 (10000*10^-3),
but those 3 trailing zeros shouldn't count towards totalDigts.
So I think the logic becomes something like this:
```scala
val bd = diNode.dataValueAsBigDecimal.stripTrailingZeros
val totalDigits =
if (bd.scale <= 0) bd.precision - bd.scale
else Math.max(bd.precision, bd.scale)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
