Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
As a next step analysed SQL test flow. Here are the steps:
1. SharedSparkSession.beforeAll called which initialise SparkSession and
SQLContext
2. SparkFunSuite.beforeAll creates a thread snapshot
3. Test code runs
4. SparkFunSuite.afterAll prints out the possible leaks
5. SharedSparkSession.afterAll stops SparkSession
Not sure if I understand right but this will not report false positives.
The only problem what I see here as it's not gonna report SparkSession and
SQLContext related leaks.
As you mentioned before this code should find SparkContext related
threading issues which applies here as well. This is not fulfilled at the
moment and my proposal is to fix it this way:
1. SparkFunSuite.beforeAll creates a thread snapshot
2. SharedSparkSession.beforeAll called which initialise SparkSession and
SQLContext
3. Test code runs
4. SharedSparkSession.afterAll stops SparkSession
5. SparkFunSuite.afterAll prints out the possible leaks
With this change I don't see any false positives and missed threads.
Please share your ideas related this topic.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
