voonhous commented on PR #8380: URL: https://github.com/apache/hudi/pull/8380#issuecomment-1502725116
> Let's give a summary about the changes: > > float -> decimal: HALF_EVEN double -> decimal: HALF_UP, > > > Since SPARK uses HALF_UP rounding when performing a FIX_SCALE_TYPE to DECIMAL casting when there is a lost in scale > > Can you show us the code snippet how Spark does this then? Is the Spark behavior a standard that we need to follow? This is the code that i referred to. In general, the default rounding mode is `HALF_UP`. https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala On top of that, I did a simple verification for fixed types which i included in the jira ticket. I'll paste them over here too: ```shell -- test HALF_UP rounding (verify that it does not use HALF_EVEN) > SELECT CAST(CAST("10.024" AS DOUBLE) AS DECIMAL(4, 2)); 10.02 > SELECT CAST(CAST("10.025" AS DOUBLE) AS DECIMAL(4, 2)); 10.03 > SELECT CAST(CAST("10.026" AS DOUBLE) AS DECIMAL(4, 2)); 10.03 -- test negative HALF_UP rounding (verify that it does not use HALF_EVEN) > SELECT CAST(CAST("-10.024" AS DOUBLE) AS DECIMAL(4, 2)); -10.02 > SELECT CAST(CAST("-10.025" AS DOUBLE) AS DECIMAL(4, 2)); -10.03 > SELECT CAST(CAST("-10.026" AS DOUBLE) AS DECIMAL(4, 2)); -10.03 -- test negative HALF_UP rounding (will return same result as HALF_EVEN) > SELECT CAST(CAST("10.034" AS DOUBLE) AS DECIMAL(4, 2)); 10.03 > SELECT CAST(CAST("10.035" AS DOUBLE) AS DECIMAL(4, 2)); 10.04 > SELECT CAST(CAST("10.036" AS DOUBLE) AS DECIMAL(4, 2)); 10.04 -- test negative HALF_UP rounding (will return same result as HALF_EVEN) > SELECT CAST(CAST("-10.034" AS DOUBLE) AS DECIMAL(4, 2)); -10.03 > SELECT CAST(CAST("-10.035" AS DOUBLE) AS DECIMAL(4, 2)); -10.04 > SELECT CAST(CAST("-10.036" AS DOUBLE) AS DECIMAL(4, 2)); -10.04 ``` As can be seen from the results i pasted above, general casting rules where scale is lost will use a rounding mode of HALF_UP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
