Re: Nifi & Spark receiver performance configuration

2015-10-08 Thread Mark Payne
its the number of flow file sent to at most the swap threshold? > - Why nifi waits swap.in.period to send batch of flowfile? > > It's not I'm not happy with nifi perf and/or the spark receiver, but the > configuration or doc should be more clear on tuning. > > Regards. > De

RE: Nifi & Spark receiver performance configuration

2015-10-08 Thread Aurélien DEHAY
with nifi perf and/or the spark receiver, but the configuration or doc should be more clear on tuning. Regards. De : Bryan Bende Envoyé : jeudi 8 octobre 2015 16:52 À : users@nifi.apache.org Objet : Re: Nifi & Spark receiver performance configuration Hello,

Re: Nifi & Spark receiver performance configuration

2015-10-08 Thread Bryan Bende
Hello, When you say you were unhappy with the performance, can you give some more information about what was not performing well? Was the NiFi Spark Receiver not pulling messages in fast enough and they were queuing up in NiFi? Was NiFi not producing messages as fast as you expected? What kind of

Nifi & Spark receiver performance configuration

2015-10-08 Thread Aurélien DEHAY
Hello. I’m doing some experimentations on Apache Nifi to see where we can use it. One idea is to use nifi to feed a spark cluster. So I’m doing some simple test (GenerateFlowFile => spark output port and a simple word count on spark side. I was pretty unhappy with the performance out of t