[jira] [Commented] (SPARK-7900) Reduce number of tagging calls in spark-ec2
[
https://issues.apache.org/jira/browse/SPARK-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571460#comment-14571460
]
Nicholas Chammas commented on SPARK-7900:
-
I'm marking this as a duplicate of [SPARK-4983].
Reduce number of tagging calls in spark-ec2
---
Key: SPARK-7900
URL: https://issues.apache.org/jira/browse/SPARK-7900
Project: Spark
Issue Type: Bug
Components: EC2
Affects Versions: 1.4.0
Reporter: Nicholas Chammas
Priority: Minor
spark-ec2 currently tags each instance with its own name:
https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692
Quite often, one of these tagging calls will fail:
{code}
Launching instances...
Launched 10 slaves in us-west-2a, regid = r-89656e83
Launched master in us-west-2a, regid = r-07646f0d
Waiting for AWS to propagate instance metadata...
Traceback (most recent call last):
File ../spark/ec2/spark_ec2.py, line 1395, in module
main()
File ../spark/ec2/spark_ec2.py, line 1387, in main
real_main()
File ../spark/ec2/spark_ec2.py, line 1222, in real_main
(master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
File ../spark/ec2/spark_ec2.py, line 667, in launch_cluster
value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 80, in
add_tag
self.add_tags({key: value}, dry_run)
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 97, in
add_tags
dry_run=dry_run
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py, line 4202,
in create_tags
return self.get_status('CreateTags', params, verb='POST')
File /path/spark/ec2/lib/boto-2.34.0/boto/connection.py, line 1223, in
get_status
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
?xml version=1.0 encoding=UTF-8?
ResponseErrorsErrorCodeInvalidInstanceID.NotFound/CodeMessageThe
instance ID 'i-d3b72524' does not
exist/Message/Error/ErrorsRequestIDf0936ab5-4d10-46d1-a35d-cefaf8a68adc/RequestID/Response
{code}
This is presumably a problem with AWS metadata taking time to become
available on all the servers that spark-ec2 hits as it makes the several
tagging calls.
Instead of retrying the tagging calls, we should just reduce them to 2
calls--one for the master, one for the slaves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7900) Reduce number of tagging calls in spark-ec2
[
https://issues.apache.org/jira/browse/SPARK-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563202#comment-14563202
]
Shivaram Venkataraman commented on SPARK-7900:
--
I am not sure how we can reduce the number of calls to 2 -- Don't we need to
tag every slave instance ?
Reduce number of tagging calls in spark-ec2
---
Key: SPARK-7900
URL: https://issues.apache.org/jira/browse/SPARK-7900
Project: Spark
Issue Type: Bug
Components: EC2
Affects Versions: 1.4.0
Reporter: Nicholas Chammas
Priority: Minor
spark-ec2 currently tags each instance with its own name:
https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692
Quite often, one of these tagging calls will fail:
{code}
Launching instances...
Launched 10 slaves in us-west-2a, regid = r-89656e83
Launched master in us-west-2a, regid = r-07646f0d
Waiting for AWS to propagate instance metadata...
Traceback (most recent call last):
File ../spark/ec2/spark_ec2.py, line 1395, in module
main()
File ../spark/ec2/spark_ec2.py, line 1387, in main
real_main()
File ../spark/ec2/spark_ec2.py, line 1222, in real_main
(master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
File ../spark/ec2/spark_ec2.py, line 667, in launch_cluster
value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 80, in
add_tag
self.add_tags({key: value}, dry_run)
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 97, in
add_tags
dry_run=dry_run
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py, line 4202,
in create_tags
return self.get_status('CreateTags', params, verb='POST')
File /path/spark/ec2/lib/boto-2.34.0/boto/connection.py, line 1223, in
get_status
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
?xml version=1.0 encoding=UTF-8?
ResponseErrorsErrorCodeInvalidInstanceID.NotFound/CodeMessageThe
instance ID 'i-d3b72524' does not
exist/Message/Error/ErrorsRequestIDf0936ab5-4d10-46d1-a35d-cefaf8a68adc/RequestID/Response
{code}
This is presumably a problem with AWS metadata taking time to become
available on all the servers that spark-ec2 hits as it makes the several
tagging calls.
Instead of retrying the tagging calls, we should just reduce them to 2
calls--one for the master, one for the slaves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7900) Reduce number of tagging calls in spark-ec2
[
https://issues.apache.org/jira/browse/SPARK-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563208#comment-14563208
]
Nicholas Chammas commented on SPARK-7900:
-
The name tags are optional, but we can tag them all at once with a single call.
The only downside is that they get the same name: something like
{{cluster-name-slave}}.
These names are provided just so that they show up on the EC2 web console. They
serve no other purpose.
Reduce number of tagging calls in spark-ec2
---
Key: SPARK-7900
URL: https://issues.apache.org/jira/browse/SPARK-7900
Project: Spark
Issue Type: Bug
Components: EC2
Affects Versions: 1.4.0
Reporter: Nicholas Chammas
Priority: Minor
spark-ec2 currently tags each instance with its own name:
https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692
Quite often, one of these tagging calls will fail:
{code}
Launching instances...
Launched 10 slaves in us-west-2a, regid = r-89656e83
Launched master in us-west-2a, regid = r-07646f0d
Waiting for AWS to propagate instance metadata...
Traceback (most recent call last):
File ../spark/ec2/spark_ec2.py, line 1395, in module
main()
File ../spark/ec2/spark_ec2.py, line 1387, in main
real_main()
File ../spark/ec2/spark_ec2.py, line 1222, in real_main
(master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
File ../spark/ec2/spark_ec2.py, line 667, in launch_cluster
value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 80, in
add_tag
self.add_tags({key: value}, dry_run)
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 97, in
add_tags
dry_run=dry_run
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py, line 4202,
in create_tags
return self.get_status('CreateTags', params, verb='POST')
File /path/spark/ec2/lib/boto-2.34.0/boto/connection.py, line 1223, in
get_status
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
?xml version=1.0 encoding=UTF-8?
ResponseErrorsErrorCodeInvalidInstanceID.NotFound/CodeMessageThe
instance ID 'i-d3b72524' does not
exist/Message/Error/ErrorsRequestIDf0936ab5-4d10-46d1-a35d-cefaf8a68adc/RequestID/Response
{code}
This is presumably a problem with AWS metadata taking time to become
available on all the servers that spark-ec2 hits as it makes the several
tagging calls.
Instead of retrying the tagging calls, we should just reduce them to 2
calls--one for the master, one for the slaves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7900) Reduce number of tagging calls in spark-ec2
[
https://issues.apache.org/jira/browse/SPARK-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563053#comment-14563053
]
Nicholas Chammas commented on SPARK-7900:
-
An alternative approach would be to just [increase the wait
time|https://github.com/apache/spark/blob/e838a25bdb5603ef05e779225704c972ce436145/ec2/spark_ec2.py#L681-L683]
to 30 or 60 seconds before trying to tag instances.
[~shivaram] / [~joshrosen]: Any preference on an approach?
Reduce number of tagging calls in spark-ec2
---
Key: SPARK-7900
URL: https://issues.apache.org/jira/browse/SPARK-7900
Project: Spark
Issue Type: Bug
Components: EC2
Affects Versions: 1.4.0
Reporter: Nicholas Chammas
Priority: Minor
spark-ec2 currently tags each instance with its own name:
https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692
Quite often, one of these tagging calls will fail:
{code}
Launching instances...
Launched 10 slaves in us-west-2a, regid = r-89656e83
Launched master in us-west-2a, regid = r-07646f0d
Waiting for AWS to propagate instance metadata...
Traceback (most recent call last):
File ../spark/ec2/spark_ec2.py, line 1395, in module
main()
File ../spark/ec2/spark_ec2.py, line 1387, in main
real_main()
File ../spark/ec2/spark_ec2.py, line 1222, in real_main
(master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
File ../spark/ec2/spark_ec2.py, line 667, in launch_cluster
value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 80, in
add_tag
self.add_tags({key: value}, dry_run)
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 97, in
add_tags
dry_run=dry_run
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py, line 4202,
in create_tags
return self.get_status('CreateTags', params, verb='POST')
File /path/spark/ec2/lib/boto-2.34.0/boto/connection.py, line 1223, in
get_status
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
?xml version=1.0 encoding=UTF-8?
ResponseErrorsErrorCodeInvalidInstanceID.NotFound/CodeMessageThe
instance ID 'i-d3b72524' does not
exist/Message/Error/ErrorsRequestIDf0936ab5-4d10-46d1-a35d-cefaf8a68adc/RequestID/Response
{code}
This is presumably a problem with AWS metadata taking time to become
available on all the servers that spark-ec2 hits as it makes the several
tagging calls.
Instead of retrying the tagging calls, we should just reduce them to 2
calls--one for the master, one for the slaves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7900) Reduce number of tagging calls in spark-ec2
[
https://issues.apache.org/jira/browse/SPARK-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563225#comment-14563225
]
Shivaram Venkataraman commented on SPARK-7900:
--
I personally don't mind all the slaves getting the same name tag -- As long as
it doesn't affect other functionality in spark_ec2.py.
Reduce number of tagging calls in spark-ec2
---
Key: SPARK-7900
URL: https://issues.apache.org/jira/browse/SPARK-7900
Project: Spark
Issue Type: Bug
Components: EC2
Affects Versions: 1.4.0
Reporter: Nicholas Chammas
Priority: Minor
spark-ec2 currently tags each instance with its own name:
https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692
Quite often, one of these tagging calls will fail:
{code}
Launching instances...
Launched 10 slaves in us-west-2a, regid = r-89656e83
Launched master in us-west-2a, regid = r-07646f0d
Waiting for AWS to propagate instance metadata...
Traceback (most recent call last):
File ../spark/ec2/spark_ec2.py, line 1395, in module
main()
File ../spark/ec2/spark_ec2.py, line 1387, in main
real_main()
File ../spark/ec2/spark_ec2.py, line 1222, in real_main
(master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
File ../spark/ec2/spark_ec2.py, line 667, in launch_cluster
value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 80, in
add_tag
self.add_tags({key: value}, dry_run)
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py, line 97, in
add_tags
dry_run=dry_run
File /path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py, line 4202,
in create_tags
return self.get_status('CreateTags', params, verb='POST')
File /path/spark/ec2/lib/boto-2.34.0/boto/connection.py, line 1223, in
get_status
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
?xml version=1.0 encoding=UTF-8?
ResponseErrorsErrorCodeInvalidInstanceID.NotFound/CodeMessageThe
instance ID 'i-d3b72524' does not
exist/Message/Error/ErrorsRequestIDf0936ab5-4d10-46d1-a35d-cefaf8a68adc/RequestID/Response
{code}
This is presumably a problem with AWS metadata taking time to become
available on all the servers that spark-ec2 hits as it makes the several
tagging calls.
Instead of retrying the tagging calls, we should just reduce them to 2
calls--one for the master, one for the slaves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
