zsxwing opened a new pull request, #38556: URL: https://github.com/apache/spark/pull/38556
### What changes were proposed in this pull request? Currently when `ScalaReflection` returns a deserializer, for a few complex types, such as array, map, udt, etc, it creates functions that may still touch `ScalaReflection` after the deserializer is created. `ScalaReflection` is a performance bottleneck for multiple threads as it holds multiple global locks. We can refactor `ScalaReflection.deserializerFor` to pre-compute everything that needs to touch `ScalaReflection` before creating the deserializer. After this, once the deserializer is created, it can be reused by multiple threads without touching `ScalaReflection.deserializerFor` any more and it will be much faster. ### Why are the changes needed? Optimize `ScalaReflection.deserializerFor` to make deserializers faster under multiple threads. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? This is refactoring `deserializerFor` to optimize the code. Existing tests should already cover the correctness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
