RexXiong commented on code in PR #2346:
URL:
https://github.com/apache/incubator-celeborn/pull/2346#discussion_r1512045144
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -850,6 +852,22 @@ private[celeborn] class Master(
logInfo(s"Offered extra $offerSlotsExtraSize slots for $shuffleKey")
}
+ if (authEnabled) {
+ // Pass application registration information to the workers
+ val pbApplicationMeta = PbApplicationMeta.newBuilder()
+ .setAppId(requestSlots.applicationId)
+ .setSecret(secretRegistry.getSecretKey(requestSlots.applicationId))
+ .build()
+ val transportMessage =
+ new TransportMessage(MessageType.APPLICATION_META,
pbApplicationMeta.toByteArray)
+
+ slots.keySet().asScala.foreach { worker =>
+ logDebug(s"Sending app registration info to
${worker.host}:${worker.internalPort}")
+ internalRpcEnvInUse.setupEndpointRef(
Review Comment:
IMO there may be a large number of workers needing to connect for the first
time, and to avoid blocking the requestSlots requests, maybe better move the
setupEndpoint process to somewhere, such as when the worker register. WDYT?
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/InternalRpcEndpoint.scala:
##########
@@ -28,16 +29,19 @@ import org.apache.celeborn.common.rpc._
*/
private[celeborn] class InternalRpcEndpoint(
override val rpcEnv: RpcEnv,
- val conf: CelebornConf)
+ val conf: CelebornConf,
+ val secretRegistry: SecretRegistry)
extends RpcEndpoint with Logging {
override def onDisconnected(address: RpcAddress): Unit = {
logDebug(s"Client $address got disconnected.")
}
override def receive: PartialFunction[Any, Unit] = {
- // TODO: [CELEBORN-1234] Handle the application secret from the Master
- case _ => throw new CelebornException(self + " not implemented")
+ case pb: PbApplicationMeta =>
+ val appId = pb.getAppId
+ val secret = pb.getSecret
+ logInfo(s"Received application meta for $appId from the coordinator")
Review Comment:
from the master~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
