LuciferYang commented on PR #47369: URL: https://github.com/apache/spark/pull/47369#issuecomment-2235697156
> > from the log we can see > > ``` > > Warning: [1888.608s][warning][gc,alloc] broadcast-exchange-907: Retried waiting for GCLocker too often allocating 33554434 words > > ``` > > > > > > > > > > > > > > > > > > > > > > > > If this is the reason for the test failures, I think we should first investigate the root cause of the GCLocker's activity to avoid covering up any unknown performance bottlenecks. > > I have only encountered this `once` so far, and this is just a record. Additionally, I believe it originated from `TPCDSQueryTestSuite`. I compared it slightly with `TPCDSQueryTestSuite` and found that there are two major differences between it and `TPCDSQueryTestSuite`: `joinConfs` and `System.gc() // Workaround for GitHub Actions memory limitation, see also SPARK-37368` > > https://github.com/apache/spark/blob/3b4c423fec08b3740c20257e130c7b016a538e79/sql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala#L223-L224 SPARK-37368 was not added System.gc due to the activity of GCLocker. IIRC, it was to address excessive memory usage leading to the test process being killed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
