bersprockets opened a new pull request #23653: [SPARK-26711][SQL] Lazily convert string values to BigDecimal during JSON schema inference URL: https://github.com/apache/spark/pull/23653 ## What changes were proposed in this pull request? This PR fixes a bug where JSON schema inference attempts to convert every String value to a BigDecimal regardless of the setting of "prefersDecimal". With that bug, behavior is still correct, but performance is impacted. This PR makes this conversion lazy, so it is only performed if prefersDecimal is set to true. Using Spark with a single executor thread to infer the schema of a single-column, 100M row JSON file, the performance impact is as follows: option | baseline | pr -----|----|----- inferTimestamp=_default_<br>prefersDecimal=_default_ | 12.5 minutes | 6.1 minutes | inferTimestamp=false<br>prefersDecimal=_default_ | 6.5 minutes | 49 seconds | inferTimestamp=false<br>prefersDecimal=true | 6.5 minutes | 6.5 minutes | ## How was this patch tested? I ran JsonInferSchemaSuite and JsonSuite. Also, I ran manual tests to see performance impact (see above).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
