Hi I am looking for way where I can make use of spark streaming in Nifi. I see couple of post where SiteToSite tcp connection is used for spark streaming application but I thinking it will be good If I can launch Spark streaming from Nifi custom processor.
PublishKafka will publish message into Kafka followed by Nifi Spark streaming processor will read from Kafka Topic. I can launch Spark streaming application from custom Nifi processor using Spark Streaming launcher API but biggest challenge is that it will create spark streaming context for each flow file which can be costly operation. Does any one suggest storing spark streaming context in controller service ? or any better approach for running spark streaming application with Nifi ? Thanks and Regards, Shashi
