[
https://issues.apache.org/jira/browse/SPARK-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Florian Verhein updated SPARK-5641:
-----------------------------------
Description:
Useful if binary files need to be uploaded. E.g. I use this for rpm transfer to
install extra stuff at cluster deployment time.
However note that it could also be used to override either:
- what's on the image
- what gets cloned from spark-ec2 (since deploy_files runs afterwards)
The idea is that the user can just dump the files into ec2/deploy.generic/.
This can be implemented by modifying deploy_files so that it simply copies the
file (if it is of certain types), rather than treating it as a text file and
attempting to replace template variables.
Detecting binary files is non-trivial. So the proposal is to have a list of
file extensions that will trigger simple file copying.
was:
Useful if binary files need to be uploaded. E.g. I use this for rpm transfer to
install extra stuff at cluster deployment time.
Could also be used to override what's on the image, etc.
The idea is that the user can just dump the files into deploy.generic.
This can be implemented by modifying deploy_templates so that it simply copies
the file (if it is of certain types), rather than treating it as a text file
and replacing template variables.
> Allow spark_ec2.py to copy arbitrary files to cluster via deploy.generic
> ------------------------------------------------------------------------
>
> Key: SPARK-5641
> URL: https://issues.apache.org/jira/browse/SPARK-5641
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Florian Verhein
> Priority: Minor
>
> Useful if binary files need to be uploaded. E.g. I use this for rpm transfer
> to install extra stuff at cluster deployment time.
> However note that it could also be used to override either:
> - what's on the image
> - what gets cloned from spark-ec2 (since deploy_files runs afterwards)
> The idea is that the user can just dump the files into ec2/deploy.generic/.
> This can be implemented by modifying deploy_files so that it simply copies
> the file (if it is of certain types), rather than treating it as a text file
> and attempting to replace template variables.
> Detecting binary files is non-trivial. So the proposal is to have a list of
> file extensions that will trigger simple file copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
