[ https://issues.apache.org/jira/browse/SPARK-28369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-28369: ----------------------------------- Assignee: Marco Gaido > Check overflow in decimal UDF > ----------------------------- > > Key: SPARK-28369 > URL: https://issues.apache.org/jira/browse/SPARK-28369 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.0 > Reporter: Mick Jermsurawong > Assignee: Marco Gaido > Priority: Minor > Fix For: 3.0.0 > > > Udf resulting in overflowing BigDecimal currently returns null. This is > inconsistent with new behavior allow option to check and throw overflow > introduced in https://issues.apache.org/jira/browse/SPARK-23179 > {code:java} > import spark.implicits._ > val tenFold: java.math.BigDecimal => java.math.BigDecimal = > _.multiply(new java.math.BigDecimal("10")) > val tenFoldUdf = udf(tenFold) > val ds = spark > .createDataset(Seq(BigDecimal("12345678901234567890.123"))) > .select(tenFoldUdf(col("value"))) > .as[BigDecimal] > ds.collect shouldEqual Seq(null){code} > The problem is at the {{CatalystTypeConverters}} where {{toPrecision}} gets > converted to null > [https://github.com/apache/spark/blob/13ae9ebb38ba357aeb3f1e3fe497b322dff8eb35/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala#L344-L356] -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org