mickjermsurawong-stripe opened a new pull request #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016 ## What changes were proposed in this pull request? - In [SPARK-23179](https://github.com/apache/spark/pull/20350), an option to throw exception on decimal overflow was introduced. The option was applied only to sql Decimal operation and `RowEncoder` but not `ExpressionEncoder`. - The problem now is round-tripping overflowing java/scala BigDecimal/BigInteger currently return null results. - The serializer encode java/scala BigDecimal to to sql Decimal, which still has the underlying the former. - When writing out to UnsafeRow, `changePrecision` will be false and row has null value. https://github.com/apache/spark/blob/24e1e41648de58d3437e008b187b84828830e238/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java#L202-L206 - This PR adds the option to throw when detecting overflowing BigDecimal/BigInteger. This gives a consistent behavior between decimal arithmetic on sql expression (DecimalPrecision), and getting decimal from dataframe (RowEncoder) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://spark.apache.org/contributing.html before opening a pull request.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
