Hi Preston, I think that we should move to the JDK code for casting to double and float. We are using it in other places (e.g. in the XQuery parser) and so if our own implementation produces different results, we are actually inconsistent within VXQuery. If we find out at some point that this is a performance issue, we can re-visit this, but I think that won’t be soon.
Cheers, Till > On Mar 27, 2015, at 11:46 AM, Eldon Carman <[email protected]> wrote: > > Not sure if we discussed this before, but it keeps coming up in our tests. > > Several of the XQTS tests fail for extremely large or small value numbers > being converted to a double form a string. The issue lies in our custom > parse double code. (Its not a simple algorithm to convert a string to a > double without losing precision.) The cast function's argument is a byte > array that hold a UTF8 string. Java offers a function that parses a string > and returns a double. Using the java function would give us greater > precision than our current code but would require converting the byte array > to a string object. > > Do you know of an parseDouble function that could be modified to fit our > use case? > > Or do you think it would be ok to break our object avoidance rule in this > case for better precision? > > Thanks > Preston
