I did test my change: https://github.com/radekg/spark/commit/b21aae1468169ee0a388d33ba6cebdb17b895956#diff-0c89b4a60c30a7cd2224bb64d93da942R125 It seems to do what I want it to do, however, not quite sure about the overall impact.
I’d appreciate if someone who knows the NettyRpcEnv details could point to some documentation. Kind regards, Radek Gruchalski ra...@gruchalski.com (mailto:ra...@gruchalski.com) (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Thursday, 26 November 2015 at 20:45, Rad Gruchalski wrote: > Dear all, > > I am currently looking at modifying NettyRpcEnv for this PR: > https://github.com/apache/spark/pull/9608 > The functionality which I’m trying to achieve is the following: if there is a > configuration property spark.driver.advertisedPort, make executors reply to > advertisedPort instead of spark.driver.port. This would enable NettyRpcEnv > work correctly on Mesos with Docker bridge networking (something what is > working spot on for AkkaRcpEnv). > > I’ve spent some time looking at the new NettyRpcEnv and I think I know what > is happening but seeking for confirmation. > > The master RpcEndpointAddress appears to be shipped to the executor as part > of a serialized message, when the executors are requested, inside of the > NettyRpcEndpointRef. In order to make my PR work, I need to change the > RcpEndpointAddress shipped to the executors on the master. > I think the best candidate is: > > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala#L125 > > Am I correct that this value becomes the RcpEndpointAddress? If so, my naive > implementation would be like this: > > if (server != null) RpcAddress(host, > conf.getInt(“spark.driver.advertisedPort”, server.getPort())) else null > > however, I am not sure what impact such change would have, about to test my > changes as soon as the code from master branch builds. > > If that is not the correct place to make such change, what would be the best > place to investigate? Is there an overview of NettyRcpEnv available anywhere? > > > > > > > > > > > Kind regards, > Radek Gruchalski > ra...@gruchalski.com (mailto:ra...@gruchalski.com) > (mailto:ra...@gruchalski.com) > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) > > Confidentiality: > This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor must > you copy or show it to anyone; please delete/destroy and inform the sender > immediately. > > > >
