Github user thvasilo commented on the pull request:
https://github.com/apache/spark/pull/4916#issuecomment-77581802
@nchammas The instances do come up with the correct type, the problem is
more the one described [here]
(https://issues.apache.org/jira/browse/SPARK-5838?focusedCommentId=14349004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14349004)
In short, while the instances themselves are OK, their setup is done
assuming the default instance type, m1.large.
This means that the machines are assumed to have 2 disks, and that leads to
problems that are described in in issue
[5838](https://issues.apache.org/jira/browse/SPARK-5838), where machines that
have one disk end up having shuffle spills in the in the small (8GB) snapshot
partitions that quickly fills up and results in failing jobs due to "No space
left on device" errors.
I will update issue 6188 to include this information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
