Hi, can you share more details. do you have any exceptions from the driver? or executors?
best, On Jul 18, 2017 02:49, "saatvikshah1994" <saatvikshah1...@gmail.com> wrote: > Hi, > > I have a pyspark App which when provided a huge amount of data as input > throws the error explained here sometimes: > https://stackoverflow.com/questions/32340639/unable-to-understand-error- > sparklistenerbus-has-already-stopped-dropping-event. > All my code is running inside the main function, and the only slightly > peculiar thing I am doing in this app is using a custom PySpark ML > Transformer(Modified from > https://stackoverflow.com/questions/32331848/create-a- > custom-transformer-in-pyspark-ml). > Could this be the issue? How can I debug why this is happening? > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Spark-UI-crashes-on-Large-Workloads-tp28873.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
