Re: Spark-submit and multiple files
You MUST put --py-files BEFORE main.py, as mentioned in another threads. On Fri, Mar 20, 2015 at 1:47 AM, Guillaume Charhon wrote: > Hi Davies, > > I am already using --py-files. The system does use the other file. The error > I am getting is not trivial. Please check the error log. > > > > On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu wrote: >> >> You could submit additional Python source via --py-files , for example: >> >> $ bin/spark-submit --py-files work.py main.py >> >> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez >> wrote: >> > Hello guys, >> > >> > I am having a hard time to understand how spark-submit behave with >> > multiple >> > files. I have created two code snippets. Each code snippet is composed >> > of a >> > main.py and work.py. The code works if I paste work.py then main.py in a >> > pyspark shell. However both snippets do not work when using spark submit >> > and >> > generate different errors. >> > >> > Function add_1 definition outside >> > http://www.codeshare.io/4ao8B >> > https://justpaste.it/jzvj >> > >> > Embedded add_1 function definition >> > http://www.codeshare.io/OQJxq >> > https://justpaste.it/jzvn >> > >> > I am trying a way to make it work. >> > >> > Thank you for your support. >> > >> > >> > >> > -- >> > View this message in context: >> > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >> > - >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark-submit and multiple files
I tried your program in yarn-client mode and it worked with no
exception. This is the command I used:
spark-submit --master yarn-client --py-files work.py main.py
(Spark 1.2.1)
On 20.3.2015. 9:47, Guillaume Charhon wrote:
Hi Davies,
I am already using --py-files. The system does use the other file. The
error I am getting is not trivial. Please check the error log.
On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <mailto:dav...@databricks.com>> wrote:
You could submit additional Python source via --py-files , for
example:
$ bin/spark-submit --py-files work.py main.py
On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez
mailto:guilla...@databerries.com>> wrote:
> Hello guys,
>
> I am having a hard time to understand how spark-submit behave
with multiple
> files. I have created two code snippets. Each code snippet is
composed of a
> main.py and work.py. The code works if I paste work.py then
main.py in a
> pyspark shell. However both snippets do not work when using
spark submit and
> generate different errors.
>
> Function add_1 definition outside
> http://www.codeshare.io/4ao8B
> https://justpaste.it/jzvj
>
> Embedded add_1 function definition
> http://www.codeshare.io/OQJxq
> https://justpaste.it/jzvn
>
> I am trying a way to make it work.
>
> Thank you for your support.
>
>
>
> --
> View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
> Sent from the Apache Spark User List mailing list archive at
Nabble.com.
>
>
-
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
> For additional commands, e-mail: user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
>
Re: Spark-submit and multiple files
Hi Davies, I am already using --py-files. The system does use the other file. The error I am getting is not trivial. Please check the error log. On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu wrote: > You could submit additional Python source via --py-files , for example: > > $ bin/spark-submit --py-files work.py main.py > > On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez > wrote: > > Hello guys, > > > > I am having a hard time to understand how spark-submit behave with > multiple > > files. I have created two code snippets. Each code snippet is composed > of a > > main.py and work.py. The code works if I paste work.py then main.py in a > > pyspark shell. However both snippets do not work when using spark submit > and > > generate different errors. > > > > Function add_1 definition outside > > http://www.codeshare.io/4ao8B > > https://justpaste.it/jzvj > > > > Embedded add_1 function definition > > http://www.codeshare.io/OQJxq > > https://justpaste.it/jzvn > > > > I am trying a way to make it work. > > > > Thank you for your support. > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > >
Re: Spark-submit and multiple files
You could submit additional Python source via --py-files , for example: $ bin/spark-submit --py-files work.py main.py On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez wrote: > Hello guys, > > I am having a hard time to understand how spark-submit behave with multiple > files. I have created two code snippets. Each code snippet is composed of a > main.py and work.py. The code works if I paste work.py then main.py in a > pyspark shell. However both snippets do not work when using spark submit and > generate different errors. > > Function add_1 definition outside > http://www.codeshare.io/4ao8B > https://justpaste.it/jzvj > > Embedded add_1 function definition > http://www.codeshare.io/OQJxq > https://justpaste.it/jzvn > > I am trying a way to make it work. > > Thank you for your support. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Spark-submit and multiple files
Hello guys, I am having a hard time to understand how spark-submit behave with multiple files. I have created two code snippets. Each code snippet is composed of a main.py and work.py. The code works if I paste work.py then main.py in a pyspark shell. However both snippets do not work when using spark submit and generate different errors. Function add_1 definition outside http://www.codeshare.io/4ao8B https://justpaste.it/jzvj Embedded add_1 function definition http://www.codeshare.io/OQJxq https://justpaste.it/jzvn I am trying a way to make it work. Thank you for your support. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
