For the record, this is what I came up with (ignoring the configurable port
for now):
spark/sbin/start-master.sh
master_ui_response_code=0
while [ "$master_ui_response_code" -ne 200 ]; do
sleep 1
master_ui_response_code="$(
curl --head --silent --output /dev/null \
--write-out "%{http_code}" localhost:8080
)"done
spark/sbin/start-slaves.sh
Turns out that the master typically takes 3-4 seconds to come up.
That’s 15 seconds saved. Hurray for yak shaving!
Nick
On Sun, Apr 12, 2015 at 5:56 PM Nicholas Chammas <nicholas.cham...@gmail.com>
wrote:
> Oh, good point. So I guess I should be able to query the master via code
> like this before any slaves are started.
>
> On Sat, Apr 11, 2015 at 7:52 PM Ted Yu <yuzhih...@gmail.com> wrote:
>
>> From SparkUI.scala :
>>
>> def getUIPort(conf: SparkConf): Int = {
>> conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
>> }
>> Better retrieve effective UI port before probing.
>>
>> Cheers
>>
>> On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
>> nicholas.cham...@gmail.com> wrote:
>>
>>> So basically, to tell if the master is ready to accept slaves, just poll
>>> http://master-node:4040 for an HTTP 200 response?
>>>
>>>
>>> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
>>> shiva...@eecs.berkeley.edu> wrote:
>>>
>>> > Yeah from what I remember it was set defensively. I don't know of a
>>> good
>>> > way to check if the master is up though. I guess we could poll the
>>> Master
>>> > Web UI and see if we get a 200/ok response
>>> >
>>> > Shivaram
>>> >
>>> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>>> > nicholas.cham...@gmail.com> wrote:
>>> >
>>> >> Check this out
>>> >> <
>>> >>
>>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>>> >> >
>>> >> (from spark-ec2):
>>> >>
>>> >> # Start Master$BIN_FOLDER/start-master.sh
>>> >
>>> >
>>> >> # Pause
>>> >> sleep 20
>>> >> # Start Workers$BIN_FOLDER/start-slaves.sh
>>> >>
>>> >> I know this was probably done defensively, but is there a more direct
>>> way
>>> >> to know when the master is ready?
>>> >>
>>> >> Nick
>>> >>
>>> >>
>>> >
>>>
>>
>>
