Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r210148558
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -180,7 +183,42 @@ private[spark] abstract class BasePythonRunner[IN,
OUT](
dataOut.writeInt(partitionIndex)
// Python version of driver
PythonRDD.writeUTF(pythonVer, dataOut)
+ // Init a GatewayServer to port current BarrierTaskContext to
Python side.
+ val isBarrier = context.isInstanceOf[BarrierTaskContext]
+ val secret = if (isBarrier) {
+ Utils.createSecret(env.conf)
+ } else {
+ ""
+ }
+ val gatewayServer: Option[GatewayServer] = if (isBarrier) {
+ Some(new GatewayServer.GatewayServerBuilder()
+ .entryPoint(context.asInstanceOf[BarrierTaskContext])
+ .authToken(secret)
+ .javaPort(0)
+ .callbackClient(GatewayServer.DEFAULT_PYTHON_PORT,
GatewayServer.defaultAddress(),
+ secret)
+ .build())
--- End diff --
Not yet. So I asked to hold it back for now since another gateway here
looks the last choice, and was wondering if we can avoid to target 2.4.0. If
this blocks, please go ahead. Will check it later on this weekends.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
