JoshRosen commented on code in PR #49350:
URL: https://github.com/apache/spark/pull/49350#discussion_r1908107301
##########
core/src/test/scala/org/apache/spark/deploy/ExternalShuffleServiceSuite.scala:
##########
@@ -45,37 +42,11 @@ import org.apache.spark.util.{ThreadUtils, Utils}
* we hash files into folders.
*/
class ExternalShuffleServiceSuite extends ShuffleSuite with BeforeAndAfterAll
with Eventually {
- var server: TransportServer = _
- var transportContext: TransportContext = _
- var rpcHandler: ExternalBlockHandler = _
-
- protected def initializeHandlers(): Unit = {
- val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle",
numUsableCores = 2)
- rpcHandler = new ExternalBlockHandler(transportConf, null)
- transportContext = new TransportContext(transportConf, rpcHandler)
- server = transportContext.createServer()
-
- conf.set(config.SHUFFLE_MANAGER, "sort")
- conf.set(config.SHUFFLE_SERVICE_ENABLED, true)
- conf.set(config.SHUFFLE_SERVICE_PORT, server.getPort)
Review Comment:
I'm guessing that previously all of this complexity was needed in order to
be able to bind to an ephemeral port and that now we can remove it now that
there's a test RPC to get the bind port?
It looks like the old approach of stubbing out these components dates back
to the original external shuffle service commit:
https://github.com/apache/spark/blame/103f513231d31cfa475aa34ce4defc63acc3cab6/core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala#L46
The new approach seems cleaner because there's less of a gap between the
test and production codepaths and thus less surface over which we could forget
to plumb in arguments or otherwise diverge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
