Thank you Andrew for you reply!
I am very intested in having this feature. It is possible to run PySpark on
AWS EMR in client mode(https://aws.amazon.com/articles/4926593393724923),
but that kills the whole idea of running batch jobs in EMR on PySpark.
Could you please (help to) create a task(with some details of possible
implementation) for this feature? I'd like to implement that but I'm too
new to Spark to know how to do it in a good way...
-Vladimir
On Tue, Jan 20, 2015 at 8:40 PM, Andrew Or and...@databricks.com wrote:
Hi Vladimir,
Yes, as the error messages suggests, PySpark currently only supports local
files. This does not mean it only runs in local mode, however; you can
still run PySpark on any cluster manager (though only in client mode). All
this means is that your python files must be on your local file system.
Until this is supported, the straightforward workaround then is to just
copy the files to your local machine.
-Andrew
2015-01-20 7:38 GMT-08:00 Vladimir Grigor vladi...@kiosked.com:
Hi all!
I found this problem when I tried running python application on Amazon's
EMR yarn cluster.
It is possible to run bundled example applications on EMR but I cannot
figure out how to run a little bit more complex python application which
depends on some other python scripts. I tried adding those files with
'--py-files' and it works fine in local mode but it fails and gives me
following message when run in EMR:
Error: Only local python files are supported:
s3://pathtomybucket/mylibrary.py.
Simplest way to reproduce in local:
bin/spark-submit --py-files s3://whatever.path.com/library.py main.py
Actual commands to run it in EMR
#launch cluster
aws emr create-cluster --name SparkCluster --ami-version 3.3.1
--instance-type m1.medium --instance-count 2 --ec2-attributes
KeyName=key20141114 --log-uri s3://pathtomybucket/cluster_logs
--enable-debugging --use-default-roles --bootstrap-action
Name=Spark,Path=s3://pathtomybucket/bootstrap-actions/spark/install-spark,Args=[-s,
http://pathtomybucket/bootstrap-actions/spark
,-l,WARN,-v,1.2,-b,2014121700,-x]
#{
# ClusterId: j-2Y58DME79MPQJ
#}
#run application
aws emr add-steps --cluster-id j-2Y58DME79MPQJ --steps
ActionOnFailure=CONTINUE,Name=SparkPy,Jar=s3://eu-west-1.elasticmapreduce/libs/script-runner/script-runner.jar,Args=[/home/hadoop/spark/bin/spark-submit,--deploy-mode,cluster,--master,yarn-cluster,--py-files,s3://pathtomybucket/tasks/demo/main.py,main.py]
#{
#StepIds: [
#s-2UP4PP75YX0KU
#]
#}
And in stderr of that step I get Error: Only local python files are
supported: s3://pathtomybucket/tasks/demo/main.py.
What is the workaround or correct way to do it? Using hadoop's distcp to
copy dependency files from s3 to nodes as another pre-step?
Regards, Vladimir
