[ https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383707#comment-17383707 ]
Hyukjin Kwon commented on SPARK-36218: -------------------------------------- cc [~maropu], [~cloud_fan], [~dongjoon] FYI. Actually, I faced this issue in our internal repo a while ago, and just added a hacky fix by adding an explicit GC: {code} if (tpcdsDataPath.nonEmpty) { tpcdsQueries .filter(_ != "q95") // TODO(SC-75125) .filter(_ != "q75") // TODO(SC-75127) .filter(_ != "q64") // TODO(SC-75126) .foreach { name => val queryString = resourceToString(s"tpcds/$name.sql", classLoader = Thread.currentThread().getContextClassLoader) test(name) { + // SPARK-36218: workaround to prevent unexpected failure related to resource usage. + System.gc() val goldenFile = new File(s"$baseResourcePath/v1_4", s"$name.sql.out") runQuery(queryString, goldenFile) } } tpcdsQueriesV2_7_0 .filter(_ != "q95") // TODO(SC-75125) .filter(_ != "q75") // TODO(SC-75127) .filter(_ != "q64") // TODO(SC-75126) .foreach { name => val queryString = resourceToString(s"tpcds-v2.7.0/$name.sql", classLoader = Thread.currentThread().getContextClassLoader) test(s"$name-v2.7") { + // SPARK-36218: workaround to prevent unexpected failure related to resource usage. + System.gc() val goldenFile = new File(s"$baseResourcePath/v2_7", s"$name.sql.out") runQuery(queryString, goldenFile) } } } else { ignore("skipped because env `SPARK_TPCDS_DATA` is not set") {} } } {code} > Flaky Test: TPC-DS in PR builder > -------------------------------- > > Key: SPARK-36218 > URL: https://issues.apache.org/jira/browse/SPARK-36218 > Project: Spark > Issue Type: Test > Components: SQL, Tests > Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0 > Reporter: Hyukjin Kwon > Priority: Major > > {code} > [info] - q1 (9 seconds, 603 milliseconds) > [info] - q2 (5 seconds, 860 milliseconds) > [info] - q3 (1 second, 777 milliseconds) > [info] - q4 (31 seconds, 951 milliseconds) > [info] - q5 (4 seconds, 561 milliseconds) > [info] - q7 (2 seconds, 471 milliseconds) > [info] - q8 (2 seconds, 74 milliseconds) > [info] - q9 (4 seconds, 402 milliseconds) > [info] - q10 (4 seconds, 618 milliseconds) > /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77: 1659 > Killed "$@" > Error: Process completed with exit code 137. > {code} > It dies in the middle: https://github.com/apache/spark/runs/3109502701 -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org