Re: org.apache.nifi.spark.NiFiReceiver and spark spark.streaming.receiver.maxRate

[Andrew Grande]

A typical production setup is to use Kafka in the middle.

Andrew

On Mon, Sep 18, 2017, 3:02 AM Margus Roo  wrote:

> Hi
>
> I need to take flow with Spark streaming from Nifi port. As we know
> Spark supports spark.streaming.receiver.maxRate and
> spark.streaming.receiver.backpressure
>
> Seems that org.apache.nifi.spark.NiFiReceiver does support it at all.
>
> https://github.com/apache/nifi/tree/master/nifi-external/nifi-spark-receiver/src/main/java/org/apache/nifi/spark
> - code is quite old too.
>
> My question is - what is the recommended aproach today getting stream
> from nifi with Spark? Is
>
> https://github.com/apache/nifi/tree/master/nifi-external/nifi-spark-receiver/src/main/java/org/apache/nifi/spark
> the best we have? If it is then what is the best aproach to integrate
> spark maxRate or backpressure in it?
>
> --
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> https://www.facebook.com/allan.tuuring
> +372 51 48 780
>
>

org.apache.nifi.spark.NiFiReceiver and spark spark.streaming.receiver.maxRate

[Margus Roo]

Hi

I need to take flow with Spark streaming from Nifi port. As we know 
Spark supports spark.streaming.receiver.maxRate and 
spark.streaming.receiver.backpressure


Seems that org.apache.nifi.spark.NiFiReceiver does support it at all. 
https://github.com/apache/nifi/tree/master/nifi-external/nifi-spark-receiver/src/main/java/org/apache/nifi/spark 
- code is quite old too.


My question is - what is the recommended aproach today getting stream 
from nifi with Spark? Is 
https://github.com/apache/nifi/tree/master/nifi-external/nifi-spark-receiver/src/main/java/org/apache/nifi/spark 
the best we have? If it is then what is the best aproach to integrate 
spark maxRate or backpressure in it?


--
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
https://www.facebook.com/allan.tuuring
+372 51 48 780