Re: Spark custom streaming receiver not storing data reliably?

2015-07-06 Thread Ajit Bhingarkar
Jorn, Thanks for your response. I am pasting below a snippet of code which shows drools integration when facts/events are picked up after reading through a File (FileReader-readLine()), it works as expected and I have tested it for wide range of record data in a File. Same code doesn't work

Spark custom streaming receiver not storing data reliably?

2015-07-05 Thread Ajit Bhingarkar
Hi, I am trying to integrate Drools rules API with Spark so that the solution could solve few CEP centric use cases. When I read data from a local file (simple FileWriter - readLine()), I see that all my rules are reliably fired and everytime I get the results as expected. I have tested with

Re: Spark custom streaming receiver not storing data reliably?

2015-07-05 Thread Jörn Franke
Can you provide the result set you are using and specify how you integrated the drools engine? Drools basically is based on a large shared memory. Hence, if you have several tasks in Shark they end up using different shared memory areas. A full integration of drools requires some sophisticated