[
https://issues.apache.org/jira/browse/SPARK-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Florian Verhein updated SPARK-5641:
-----------------------------------
Description:
*Updated - no longer via deploy.generic, no substitutions*
Essentially, give users an easy way to rcp a directory structure to the
master's / as part of the cluster launch, at a useful point in the workflow
(before setup.sh is called on the master).
Useful if binary files need to be uploaded. E.g. I use this for rpm transfer to
install extra stuff at cluster deployment time.
However note that it could also be used to override / add to either:
- what's on the image
- what gets cloned from spark-ec2 (e.g. add new module)
was:
Useful if binary files need to be uploaded. E.g. I use this for rpm transfer to
install extra stuff at cluster deployment time.
However note that it could also be used to override either:
- what's on the image
- what gets cloned from spark-ec2 (since deploy_files runs afterwards)
The idea is that the user can just dump the files into ec2/deploy.generic/.
This can be implemented by modifying deploy_files so that it simply copies the
file (if it is of certain types), rather than treating it as a text file and
attempting to replace template variables.
Detecting binary files is non-trivial. So the proposal is to have a list of
file extensions that will trigger simple file copying.
> Allow spark_ec2.py to copy arbitrary files to cluster
> -----------------------------------------------------
>
> Key: SPARK-5641
> URL: https://issues.apache.org/jira/browse/SPARK-5641
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Florian Verhein
> Priority: Minor
>
> *Updated - no longer via deploy.generic, no substitutions*
> Essentially, give users an easy way to rcp a directory structure to the
> master's / as part of the cluster launch, at a useful point in the workflow
> (before setup.sh is called on the master).
> Useful if binary files need to be uploaded. E.g. I use this for rpm transfer
> to install extra stuff at cluster deployment time.
> However note that it could also be used to override / add to either:
> - what's on the image
> - what gets cloned from spark-ec2 (e.g. add new module)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
