Re: Spark Streaming having trouble writing checkpoint

2015-12-14 Thread Robert Towne
I forgot to include the data node logs for this time period: 2015-12-14 00:14:52,836 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: server51:50010:DataXceiver error processing unknown operation src: /127.0.0.1:39442 dst: /127.0.0.1:50010 java.io.EOFException at

Spark Streaming having trouble writing checkpoint

2015-12-14 Thread Robert Towne
I have a Spark Streaming app (1.5.2 compile for hadoop 2.6) that occasionally has problem writing its checkpoint file. This is a YARN (yarn cluster) app running as user mapred. What I see in my streaming app logs are: App log App log 15/12/15 00:14:08 server51: