Hi Haiyang, you are right, YARN takes over the resource management, bot I constantly got Exception ConnectionRefused on mentioned port. So, I suppose some spark internal communications are done via this port... but I don't know what exactly and how can I change it...
Thank you, Konstantin Kudryavtsev On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <haiyangfu...@gmail.com> wrote: > Hi Konstantin, > Would you please post some more details? Error info or exception from the > log on what situation？when you run spark job on yarn cluster mode ,yarn > will take over all the resource management. > > > On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev < > kudryavtsev.konstan...@gmail.com> wrote: > >> Hi Larry, >> >> I'm afraid this is standalone mode, I'm interesting in YARN >> >> Also, I don't see port-in-trouble 33007 which i believe related to Akka >> >> Thank you, >> Konstantin Kudryavtsev >> >> >> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xia...@sjtu.edu.cn> wrote: >> >>> Hi Konstantin, >>> >>> I think you can find it at >>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security >>> and you can specify port for master or worker at conf/spark-env.sh >>> >>> Larry >>> >>> >>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote: >>> >>> Hi there, >>> >>> I'm trying to run Spark on YARN cluster and face with issued that some >>> ports are closed, particularly port 33007 (I suppose it's used by Akka) >>> >>> Could you please provide me with a list of all ports required for >>> Spark? >>> Also, is it possible to set up these ports? >>> >>> Thank you in advance, >>> Konstantin Kudryavtsev >>> >>> >>> >> >
