You MUST put --py-files BEFORE main.py, as mentioned in another threads. On Fri, Mar 20, 2015 at 1:47 AM, Guillaume Charhon <guilla...@databerries.com> wrote: > Hi Davies, > > I am already using --py-files. The system does use the other file. The error > I am getting is not trivial. Please check the error log. > > > > On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <dav...@databricks.com> wrote: >> >> You could submit additional Python source via --py-files , for example: >> >> $ bin/spark-submit --py-files work.py main.py >> >> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez <guilla...@databerries.com> >> wrote: >> > Hello guys, >> > >> > I am having a hard time to understand how spark-submit behave with >> > multiple >> > files. I have created two code snippets. Each code snippet is composed >> > of a >> > main.py and work.py. The code works if I paste work.py then main.py in a >> > pyspark shell. However both snippets do not work when using spark submit >> > and >> > generate different errors. >> > >> > Function add_1 definition outside >> > http://www.codeshare.io/4ao8B >> > https://justpaste.it/jzvj >> > >> > Embedded add_1 function definition >> > http://www.codeshare.io/OQJxq >> > https://justpaste.it/jzvn >> > >> > I am trying a way to make it work. >> > >> > Thank you for your support. >> > >> > >> > >> > -- >> > View this message in context: >> > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
