eejbyfeldt commented on code in PR #40808:
URL: https://github.com/apache/spark/pull/40808#discussion_r1189849661
##########
core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala:
##########
@@ -51,6 +51,7 @@ import org.apache.spark.util.Utils
private[spark] class NettyBlockTransferService(
conf: SparkConf,
securityManager: SecurityManager,
+ serializerManager: SerializerManager,
Review Comment:
Yeah, I think I think I see what you are aiming for now.
Seems that the classloader also depend on the `userClassPath` passed to the
executor constructor here:
https://github.com/apache/spark/blob/27b0c5ec12e7fcb8e442aabe448b825a400fa755/core/src/main/scala/org/apache/spark/executor/Executor.scala#L67
and exactly where that comes from seems to depend on the scheduler backend.
For example for Yarn it seems to be this function
https://github.com/apache/spark/blob/27b0c5ec12e7fcb8e442aabe448b825a400fa755/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1524-L1538
so I don't see how to easily construct and correctly configure the
serializer directly here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
