cloud-fan commented on issue #25243: [SPARK-28498][SQL][TEST] clear the states of SparkSession after each test URL: https://github.com/apache/spark/pull/25243#issuecomment-515351496 After some local experiments, it turns out that creating a fresh copy of spark session has significant overhead. The `DataFrameSuite` becomes 10% slower. A fresh copy of spark session means we need to re-create the analyzer, optimizer, session catalog, etc. which are a lot of objects. We also need to re-create temp views of the test data, which is expensive as well. I've changed the implementation to only clear the states of a spark session instead of creating a fresh copy. The overhead is negligible now. review hints: you can hide whitespace changes 
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
