DaveBirdsall commented on a change in pull request #1831: [TRAFODION-3300] Fix
overflow issues with extreme big nums + ROUND
URL: https://github.com/apache/trafodion/pull/1831#discussion_r278206583
##########
File path: core/sql/generator/GenPreCode.cpp
##########
@@ -10033,17 +10033,56 @@ ItemExpr * MathFunc::preCodeGen(Generator *
generator)
if (nodeIsPreCodeGenned())
return this;
+ // for ROUND, if the first operand is a BigNum, don't cast
+ // the children to DOUBLE PRECISION; but do make sure the
+ // second operand is an integer
+ NABoolean castIt = TRUE;
+ if (getOperatorType() == ITM_ROUND)
+ {
+ const NAType &typ0 = child(0)->getValueId().getType();
+ if (((const NumericType &)typ0).isBigNum())
+ {
+ castIt = FALSE;
+
+ if (getArity() > 1)
+ {
+ const NumericType &typ1 = (const NumericType
&)child(1)->getValueId().getType();
+ if (!typ1.isInteger() ||
+ ((typ1.getFSDatatype() != REC_BIN8_SIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN8_UNSIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN16_SIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN16_UNSIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN32_SIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN32_UNSIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN64_SIGNED) &&
+ (typ1.getFSDatatype() != REC_BIN64_UNSIGNED) ) )
+ {
Review comment:
Didn't know about this method. Just took a look at it. It includes
REC_BPINT_UNSIGNED, which I had not added run-time support for. I'll add it and
make this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
