Is spark-ec2 intended for spinning up production Spark clusters?

I think the answer is no.

However, the docs for spark-ec2
<https://spark.apache.org/docs/latest/ec2-scripts.html> very much leave
that possibility open, and indeed I see many people asking questions or
opening issues that stem from some production use case they are trying to
fit spark-ec2 to.

Here's the latest example
<https://issues.apache.org/jira/browse/SPARK-6900?focusedCommentId=14504236&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14504236>
of
someone using spark-ec2 to power their (presumably) production service.

Shouldn't we actively discourage people from using spark-ec2 in this way?

I understand there's no stopping people from doing what they want with it,
and certainly the questions and issues we receive about spark-ec2 are still
valid, even if they stem from discouraged use cases.

>From what I understand, spark-ec2 is intended for quick experimentation,
one-off jobs, prototypes, and so forth.

If that's the case, it's best to stress this in the docs.

Nick

Reply via email to