Hello, When you say you were unhappy with the performance, can you give some more information about what was not performing well?
Was the NiFi Spark Receiver not pulling messages in fast enough and they were queuing up in NiFi? Was NiFi not producing messages as fast as you expected? What kind of environment were you running this? All on a local machine for testing? -Bryan On Thu, Oct 8, 2015 at 6:52 AM, Aurélien DEHAY <[email protected]> wrote: > Hello. > > > > I’m doing some experimentations on Apache Nifi to see where we can use it. > > > > One idea is to use nifi to feed a spark cluster. So I’m doing some simple > test (GenerateFlowFile => spark output port and a simple word count on > spark side. > > > > I was pretty unhappy with the performance out of the box, so I looked on > the net and found almost nothing. > > > > So I looked at nifi.properties, and found that some of the following > properties have a huge impact on how many messages / second were processed > to Spark : > > > > nifi.queue.swap.threshold=20000 > > nifi.swap.in.period=1 sec > > nifi.swap.in.threads=1 > > nifi.swap.out.period=1 sec > > nifi.swap.out.threads=4 > > > > The documentation seems unclear on this point for output ports, is anyone > have a pointer for me ? > > > > Thanks. > > > > Aurélien. >
