Hi Avishek, As of Spark 1.0, PySpark does in fact run on YARN.
-Sandy On Fri, Aug 8, 2014 at 12:47 PM, Avishek Saha <avishek.s...@gmail.com> wrote: > So I think I have a better idea of the problem now. > > The environment is YARN client and IIRC PySpark doesn't run on YARN > cluster. > > So my client is heavily loaded which causes iy loose a lot of e executors > which might be part of the problem. > > Btw any plans in supporting PySpark in YARN clusters mode? > On Aug 7, 2014 3:04 PM, "Davies Liu" <dav...@databricks.com> wrote: > >> What is the environment ? YARN or Mesos or Standalone? >> >> It will be more helpful if you could show more loggings. >> >> On Wed, Aug 6, 2014 at 7:25 PM, Avishek Saha <avishek.s...@gmail.com> >> wrote: >> > Hi, >> > >> > I get a lot of executor lost error for "saveAsTextFile" with PySpark >> > and Hadoop 2.4. >> > >> > For small datasets this error occurs but since the dataset is small it >> > gets eventually written to the file. >> > For large datasets, it takes forever to write the final output. >> > >> > Any help is appreciated. >> > Avishek >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > >> >
