Re: Getting the IP address of Spark Driver in yarn-cluster mode

Thanks Steve. Here is the Python pseudo code that got it working for me: import time; import urllib2 nodes= ({'worker1_hostname':'worker1_ip', ... }) YARN_app_queue = 'default' YARN_address = 'http://YARN_IP:8088' YARN_app_startedTimeBegin = str(int(time.time() - 3600)) # We allow

Re: Getting the IP address of Spark Driver in yarn-cluster mode

On 24 Oct 2016, at 19:34, Masood Krohy > wrote: Hi everyone, Is there a way to set the IP address/hostname that the Spark Driver is going to be running on when launching a program through spark-submit in yarn-cluster mode (PySpark

Getting the IP address of Spark Driver in yarn-cluster mode

Hi everyone, Is there a way to set the IP address/hostname that the Spark Driver is going to be running on when launching a program through spark-submit in yarn-cluster mode (PySpark 1.6.0)? I do not see an option for this. If not, is there a way to get this IP address after the Spark app has