[
https://issues.apache.org/jira/browse/SPARK-59821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59821.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59096
[https://github.com/apache/spark/pull/59096]
> Make Maven utility tests safe for concurrent runs
> -------------------------------------------------
>
> Key: SPARK-59821
> URL: https://issues.apache.org/jira/browse/SPARK-59821
> Project: Spark
> Issue Type: Test
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Qianlan Chen
> Assignee: Qianlan Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Maven utility tests are not fully isolated when multiple test JVMs run
> concurrently.
> SparkTestUtils.createCompiledClass writes generated class files to the
> process working directory before moving them to the requested destination.
> Concurrent compilations of the same class name can therefore overwrite or
> move each other's output.
> MavenUtils also uses the fixed dummy/.m2/repository path during tests. One
> test JVM can clean this repository while another JVM is resolving
> dependencies.
> This race is exposed by high-concurrency flakiness runs, where existing
> MavenUtilsSuite cases fail intermittently even though normal master CI runs
> are generally green. It is a test-infrastructure race rather than a Maven
> resolution product bug. The issue predates SPARK-59553 and was only exposed
> while validating that change.
> The tests should:
> * Compile generated classes in invocation-specific temporary directories.
> * Use a JVM-specific but stable temporary Maven repository.
> * Include a regression test that compiles the same class name concurrently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
