Also Konstantin do you have a firewall between your Spark services? If that's what's causing these issues, then you may be interested in the ability to configure every port a Spark service listens on -- https://issues.apache.org/jira/browse/SPARK-2157
On Thu, Jul 31, 2014 at 8:47 AM, Haiyang Fu <haiyangfu...@gmail.com> wrote: > Hi Konstantin, > > Could you please post your first container's stderr log here which is > always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071 > are all random socket ports determined by each job. So 33007 maybe just > a temporary port for data transferation. The deeper reason for 33007 > ConnectionRefused maybe something else.would you please check If it is > always 33007 when you submit your job again ? > > > haiy > > > On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev < > kudryavtsev.konstan...@gmail.com> wrote: > >> Hi Haiyang, >> >> you are right, YARN takes over the resource management, bot I constantly >> got Exception ConnectionRefused on mentioned port. So, I suppose some spark >> internal communications are done via this port... but I don't know what >> exactly and how can I change it... >> >> Thank you, >> Konstantin Kudryavtsev >> >> >> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <haiyangfu...@gmail.com> >> wrote: >> >>> Hi Konstantin, >>> Would you please post some more details? Error info or exception from >>> the log on what situation？when you run spark job on yarn cluster mode ,yarn >>> will take over all the resource management. >>> >>> >>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev < >>> kudryavtsev.konstan...@gmail.com> wrote: >>> >>>> Hi Larry, >>>> >>>> I'm afraid this is standalone mode, I'm interesting in YARN >>>> >>>> Also, I don't see port-in-trouble 33007 which i believe related to >>>> Akka >>>> >>>> Thank you, >>>> Konstantin Kudryavtsev >>>> >>>> >>>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xia...@sjtu.edu.cn> wrote: >>>> >>>>> Hi Konstantin, >>>>> >>>>> I think you can find it at >>>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security >>>>> and you can specify port for master or worker at conf/spark-env.sh >>>>> >>>>> Larry >>>>> >>>>> >>>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote: >>>>> >>>>> Hi there, >>>>> >>>>> I'm trying to run Spark on YARN cluster and face with issued that >>>>> some ports are closed, particularly port 33007 (I suppose it's used >>>>> by Akka) >>>>> >>>>> Could you please provide me with a list of all ports required for >>>>> Spark? >>>>> Also, is it possible to set up these ports? >>>>> >>>>> Thank you in advance, >>>>> Konstantin Kudryavtsev >>>>> >>>>> >>>>> >>>> >>> >> >
