yugan95 commented on PR #56542: URL: https://github.com/apache/spark/pull/56542#issuecomment-4727139308
@HeartSaVioR Thanks for the follow-up questions. To clarify the shard infrastructure: it is fully inlined in the executor process — just an additional Netty server and a few RPC endpoints started inside `SparkEnv`. There is no separate process, no external service, and no additional deployment step. When `spark.shard.enabled=false` (the default), none of this code is initialized and there is zero overhead. The reason it defaults to off is that the Netty server opens an additional port on each executor. Keeping it opt-in felt more appropriate rather than starting a new listener by default. I understand the concern about scope and I'm open to drafting a SPIP. Will wait to hear from the SQL area maintainers as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
