Hi, I have a pyspark App which when provided a huge amount of data as input throws the error explained here sometimes: https://stackoverflow.com/questions/32340639/unable-to-understand-error-sparklistenerbus-has-already-stopped-dropping-event. All my code is running inside the main function, and the only slightly peculiar thing I am doing in this app is using a custom PySpark ML Transformer(Modified from https://stackoverflow.com/questions/32331848/create-a-custom-transformer-in-pyspark-ml). Could this be the issue? How can I debug why this is happening?
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-crashes-on-Large-Workloads-tp28873.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
