Below is trace from trying to access with ~/path. I also did the echo as
per Nick (see the last line), looks ok to me. This is my development box
with Spark 1.2.0 running CentOS 6.5, Python 2.6.6
[pete.zybrick@pz-lt2-ipc spark-1.2.0]$ ec2/spark-ec2
--key-pair=spark-streaming-kp
--identity-file=~/.pzkeys/spark-streaming-kp.pem --region=us-east-1 login
pz-spark-cluster
Searching for existing cluster pz-spark-cluster...
Found 1 master(s), 3 slaves
Logging into master ec2-54-152-95-129.compute-1.amazonaws.com...
Warning: Identity file ~/.pzkeys/spark-streaming-kp.pem not accessible: No
such file or directory.
Permission denied (publickey).
Traceback (most recent call last):
File "ec2/spark_ec2.py", line 1082, in <module>
main()
File "ec2/spark_ec2.py", line 1074, in main
real_main()
File "ec2/spark_ec2.py", line 1007, in real_main
ssh_command(opts) + proxy_opt + ['-t', '-t', "%s@%s" % (opts.user,
master)])
File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-o',
'StrictHostKeyChecking=no', '-i', '~/.pzkeys/spark-streaming-kp.pem', '-t',
'-t', u'r...@ec2-54-152-95-129.compute-1.amazonaws.com']' returned non-zero
exit status 255
[pete.zybrick@pz-lt2-ipc spark-1.2.0]$ echo ~/.pzkeys/spark-streaming-kp.pem
/home/pete.zybrick/.pzkeys/spark-streaming-kp.pem
On Wed, Jan 28, 2015 at 3:49 PM, Charles Feduke <charles.fed...@gmail.com>
wrote:
> Yeah, I agree ~ should work. And it could have been [read: probably was]
> the fact that one of the EC2 hosts was in my known_hosts (don't know, never
> saw an error message, but the behavior is no error message for that state),
> which I had fixed later with Pete's patch. But the second execution when
> things worked with an absolute path could have worked because the random
> hosts that came up on EC2 were never in my known_hosts.
>
>
> On Wed Jan 28 2015 at 3:45:36 PM Nicholas Chammas <
> nicholas.cham...@gmail.com> wrote:
>
>> Hmm, I can’t see why using ~ would be problematic, especially if you
>> confirm that echo ~/path/to/pem expands to the correct path to your
>> identity file.
>>
>> If you have a simple reproduction of the problem, please send it over.
>> I’d love to look into this. When I pass paths with ~ to spark-ec2 on my
>> system, it works fine. I’m using bash, but zsh handles tilde expansion the
>> same as bash.
>>
>> Nick
>>
>>
>> On Wed Jan 28 2015 at 3:30:08 PM Charles Feduke <charles.fed...@gmail.com>
>> wrote:
>>
>>> It was only hanging when I specified the path with ~ I never tried
>>> relative.
>>>
>>> Hanging on the waiting for ssh to be ready on all hosts. I let it sit
>>> for about 10 minutes then I found the StackOverflow answer that suggested
>>> specifying an absolute path, cancelled, and re-run with --resume and the
>>> absolute path and all slaves were up in a couple minutes.
>>>
>>> (I've stood up 4 integration clusters and 2 production clusters on EC2
>>> since with no problems.)
>>>
>>> On Wed Jan 28 2015 at 12:05:43 PM Nicholas Chammas <
>>> nicholas.cham...@gmail.com> wrote:
>>>
>>>> Ey-chih,
>>>>
>>>> That makes more sense. This is a known issue that will be fixed as part
>>>> of SPARK-5242 <https://issues.apache.org/jira/browse/SPARK-5242>.
>>>>
>>>> Charles,
>>>>
>>>> Thanks for the info. In your case, when does spark-ec2 hang? Only when
>>>> the specified path to the identity file doesn't exist? Or also when you
>>>> specify the path as a relative path or with ~?
>>>>
>>>> Nick
>>>>
>>>>
>>>> On Wed Jan 28 2015 at 9:29:34 AM ey-chih chow <eyc...@hotmail.com>
>>>> wrote:
>>>>
>>>>> We found the problem and already fixed it. Basically, spark-ec2
>>>>> requires ec2 instances to have external ip addresses. You need to specify
>>>>> this in the ASW console.
>>>>> ------------------------------
>>>>> From: nicholas.cham...@gmail.com
>>>>> Date: Tue, 27 Jan 2015 17:19:21 +0000
>>>>> Subject: Re: spark 1.2 ec2 launch script hang
>>>>> To: charles.fed...@gmail.com; pzybr...@gmail.com; eyc...@hotmail.com
>>>>> CC: user@spark.apache.org
>>>>>
>>>>>
>>>>> For those who found that absolute vs. relative path for the pem file
>>>>> mattered, what OS and shell are you using? What version of Spark are you
>>>>> using?
>>>>>
>>>>> ~/ vs. absolute path shouldn’t matter. Your shell will expand the ~/
>>>>> to the absolute path before sending it to spark-ec2. (i.e. tilde
>>>>> expansion.)
>>>>>
>>>>> Absolute vs. relative path (e.g. ../../path/to/pem) also shouldn’t
>>>>> matter, since we fixed that for Spark 1.2.0
>>>>> <https://issues.apache.org/jira/browse/SPARK-4137>. Maybe there’s
>>>>> some case that we missed?
>>>>>
>>>>> Nick
>>>>>
>>>>> On Tue Jan 27 2015 at 10:10:29 AM Charles Feduke <
>>>>> charles.fed...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Absolute path means no ~ and also verify that you have the path to the
>>>>> file correct. For some reason the Python code does not validate that the
>>>>> file exists and will hang (this is the same reason why ~ hangs).
>>>>> On Mon, Jan 26, 2015 at 10:08 PM Pete Zybrick <pzybr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Try using an absolute path to the pem file
>>>>>
>>>>>
>>>>>
>>>>> > On Jan 26, 2015, at 8:57 PM, ey-chih chow <eyc...@hotmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
>>>>> > modified the script according to
>>>>> >
>>>>> > https://github.com/grzegorz-dubicki/spark/commit/5dd8458d2ab
>>>>> 9753aae939b3bb33be953e2c13a70
>>>>> >
>>>>> > But the script was still hung at the following message:
>>>>> >
>>>>> > Waiting for cluster to enter 'ssh-ready'
>>>>> > state.............................................
>>>>> >
>>>>> > Any additional thing I should do to make it succeed? Thanks.
>>>>> >
>>>>> >
>>>>> > Ey-Chih Chow
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > View this message in context: http://apache-spark-user-list.
>>>>> 1001560.n3.nabble.com/spark-1-2-ec2-launch-script-hang-tp21381.html
>>>>> > Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>> >
>>>>> > ------------------------------------------------------------
>>>>> ---------
>>>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>> > For additional commands, e-mail: user-h...@spark.apache.org
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
