Can you look in the datanode logs and see whats going on? Most likely, you
are hitting the ulimit on open file handles.
Thanks
Best Regards
On Wed, Jul 8, 2015 at 10:55 AM, Pankaj Arora pankaj.ar...@guavus.com
wrote:
Hi,
I am running long running application over yarn using spark and I am
@spark.apache.orgmailto:dev@spark.apache.org
dev@spark.apache.orgmailto:dev@spark.apache.org
Subject: Re: Spark job hangs when History server events are written to hdfs
As such we do not open any files by ourselves. EventLoggingListener opens the
file to write down the events in json format for history server
As such we do not open any files by ourselves. EventLoggingListener opens
the file to write down the events in json format for history server. But it
uses the same writer(PrintWriter object) and eventually the same output
stream (which boils down to DFSOutputStream for us). It
seems
Hi,
I am running long running application over yarn using spark and I am facing
issues while using spark’s history server when the events are written to hdfs.
It seems to work fine for some time and in between I see following exception.
2015-06-01 00:00:03,247 [SparkListenerBus] ERROR
