Also Konstantin do you have a firewall between your Spark services? If
that's what's causing these issues, then you may be interested in the
ability to configure every port a Spark service listens on --
https://issues.apache.org/jira/browse/SPARK-2157
On Thu, Jul 31, 2014 at 8:47 AM, Haiyang Fu
Hi Konstantin,
Could you please post your first container's stderr log here which is
always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
are all random socket ports determined by each job. So 33007 maybe just a
temporary port for data transferation. The deeper reason for
Hi Haiyang,
you are right, YARN takes over the resource management, bot I constantly
got Exception ConnectionRefused on mentioned port. So, I suppose some spark
internal communications are done via this port... but I don't know what
exactly and how can I change it...
Thank you,
Konstantin Kudryav
Sorry, I don't have experience with YARN.
I checked the YARN page
http://spark.apache.org/docs/latest/running-on-yarn.html
And for configuration, it refers to
http://spark.apache.org/docs/latest/configuration.html " Most of the
configs are the same for Spark on YARN as for other deployment mod
Hi Konstantin,
Would you please post some more details? Error info or exception from the
log on what situation？when you run spark job on yarn cluster mode ,yarn
will take over all the resource management.
On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
kudryavtsev.konstan...@gmail.com>
Hi Larry,
I'm afraid this is standalone mode, I'm interesting in YARN
Also, I don't see port-in-trouble 33007 which i believe related to Akka
Thank you,
Konstantin Kudryavtsev
On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao wrote:
> Hi Konstantin,
>
> I think you can find it at
> https://spar
Hi Konstantin,
I think you can find it at
https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
and you can specify port for master or worker at conf/spark-env.sh
Larry
On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
Hi there,
I'm trying to run
Hi there,
I'm trying to run Spark on YARN cluster and face with issued that some
ports are closed, particularly port 33007 (I suppose it's used by Akka)
Could you please provide me with a list of all ports required for Spark?
Also, is it possible to set up these ports?
Thank you in advance,
Kons
