vanzin commented on issue #23801: [SPARK-26891][YARN] Fixing flaky test in YarnSchedulerBackendSuite URL: https://github.com/apache/spark/pull/23801#issuecomment-464173703 Hmm, I think I understand where this is coming from. But if that's the case, then the following test in the same suite is also flaky, for the same reason. I think the problem is here: ``` val driverEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME, createDriverEndpoint()) ``` When the test instantiates a `YarnSchedulerBackend`, that line is overwriting the endpoint in the running `SparkContext`, so that any messages now are sent to the test scheduler instead. That's where the multi-threaded access comes from. So I think the best thing here would be to avoid having a live `SparkContext` if possible. If not, then figure out another way to avoid the situation above.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
