java.io.OptionalDataException during task deserialization

2017-08-01 Thread Viacheslav Krot
Hi, I have a weird issue - spark streaming application fails once/twice a day with java.io.OptionalDataException. It happens when deserializing task. The problem appeared after migration from spark 1.6 to spark 2.2, cluster runs in latest CDH distro, YARN mode. I have no clue what to blame and w

java.io.OptionalDataException during task deserialization

2017-08-01 Thread Viacheslav Krot
Hi, I have a weird issue - spark streaming application fails once/twice a day with java.io.OptionalDataException. It happens when deserializing task. The problem appeared after migration from spark 1.6 to spark 2.2, cluster runs in latest CDH distro, YARN mode. I have no clue what to blame and w