mgaido91 opened a new pull request #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal URL: https://github.com/apache/spark/pull/23308 ## What changes were proposed in this pull request? Currently, when we infer the schema for scala/java decimals, we return as data type the `SYSTEM_DEFAULT` implementation, ie. the decimal type with precision 38 and scale 18. But this is not right, as we know nothing about the right precision and scale and these values can be not enough to store the data. This problem arises in particular with UDF, where we cast all the input of type `DecimalType` to a `DecimalType(38, 18)`: in case this is not enough, null is returned as input for the UDF. The PR changes the resolution of `BigDecimal`/`Decimal` to the abstract `DecimalType`. Please notice that the same problem is still present for Decimals in arrays, structs and maps, because the related types don't accept an `AbstractDataType`. In a followup work, we can introduce abstract types for them accepting abstract types in order to fix the issue for them too. ## How was this patch tested? added UT
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
