Thanks Andrew.
We cannot include Spark in our Java project due to dependency issues. The
Spark will not be exposed to clients.
What we want todo is to put spark tarball (in worst case) into HDFS, so
through our java app which runs in local mode, launch spark-submit script
with user python files.
So
Hi Elkhan,
Spark submit depends on several things: the launcher jar (1.3.0+ only), the
spark-core jar, and the spark-yarn jar (in your case). Why do you want to
put it in HDFS though? AFAIK you can't execute scripts directly from HDFS;
you need to copy them to a local file system first. I don't se
Hi all,
If I want to ship spark-submit script to HDFS. and then call it from HDFS
location for starting Spark job, which other files/folders/jars need to be
transferred into HDFS with spark-submit script ?
Due to some dependency issues, we can include Spark in our Java
application, so instead we
