mgaido91 commented on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal URL: https://github.com/apache/spark/pull/23308#issuecomment-447836201 @cloud-fan that's true, but also when we want to infer the type of user provided data we have no idea of the right precision and scale to use. What we are doing is just picking the default one. I'd rather see 2 methods: one for getting a concrete schema and one for the abstract one with the former just calling the latter and then getting its `defaultConcreteDataType`. I think logically this is closer to what we are actually doing. Anyway, I don't think your suggestion is feasible because we still need to keep the implicit casting functionality. What I can do is to update the `inputTypes` on the flight, when building the `ScalaUDF`. Let me do that just to do a small fix for the problem, then we can get back to this as a cleaner refactor and a way to support also array/struct/maps f decimals (because currently that is not feasible) if we agree on this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
