echauchot commented on PR #17406: URL: https://github.com/apache/beam/pull/17406#issuecomment-1114582816
> Usage of `forkEvery 1` is typically a strong sign of poor quality / bad code and should be avoided: > > * It often hides resource leaks, either in code or worse, potentially in the runner itself. > * It significantly impacts performance when running tests. > > This PR introduces new JUnit rules to better manage the lifecyle of `SparkContext` or `SparkContextOptions` and allows to safely reuse those for all tests of a suite where applicable. The previous `ReuseSparkContextRule` leaks the SparkContext and was removed. Besides, with `forkEvery 1` trying to reuse the `SparkContext` is absolutely pointless ... > > There's might be more issues hidden I didn't discover locally ... I'll keep a look. > > This also reduce the runtime of tests to ~30 secs from 1:30+ mins. > > Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: > > * [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). > * [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. > * [ ] Update `CHANGES.md` with noteworthy changes. > * [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). > > See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). > > To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md > > ## GitHub Actions Tests Status (on master branch) > [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) > > See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI. Bad smell indeed ! Thanks for catching this @mosche and reducing the test time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
