Re: Spark Streaming: Custom Receiver OOM consistently

Thanks ! On Mon, May 22, 2017 at 5:58 PM kant kodali wrote: > Well there are few things here. > > 1. What is the Spark Version? > cdh 1.6 2. You said there is OOM error but what is the cause that appears in the > log message or stack trace? OOM can happen for various

Re: Spark Streaming: Custom Receiver OOM consistently

Well there are few things here. 1. What is the Spark Version? 2. You said there is OOM error but what is the cause that appears in the log message or stack trace? OOM can happen for various reasons and JVM usually specifies the cause in the error message. 3. What is the driver and executor

Re: Spark Streaming: Custom Receiver OOM consistently

thanks Alonso, Sorry, but there are some security reservations. But we can assume the receiver, is equivalent to writing a JMS based custom receiver, where we register a message listener and for each message delivered by JMS will be stored by calling store method of listener. Something like :

Re: Spark Streaming: Custom Receiver OOM consistently

could you share the code? Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman 2017-05-20 7:54 GMT+02:00 Manish Malhotra : > Hello, >

Spark Streaming: Custom Receiver OOM consistently

Hello, have implemented Java based custom receiver, which consumes from messaging system say JMS. once received message, I call store(object) ... Im storing spark Row object. it run for around 8 hrs, and then goes OOM, and OOM is happening in receiver nodes. I also tried to run multiple