See a lot of Spark Streaming receiver based approach code there, which makes me a bit worried about scalability.
Nonetheless. API wise cant we just so dstream.rdd.forEach? And issue these writes using the WriteClient api? On Sat, Sep 21, 2019 at 4:16 AM Taher Koitawala <taher...@gmail.com> wrote: > Hi Vinoth, > Nifi has the capability to pass data to a custom spark job. > However that is done through a StreamingContext, not sure if we can build > something on this. I'm trying to wrap my head around how to fit the > StreamingContext in our existing code. > > Here is an example: > https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark > > Regards, > Taher Koitawala > > On Wed, Sep 18, 2019, 8:27 PM Vinoth Chandar <vin...@apache.org> wrote: > > > Not too familiar wth Nifi myself. Would this still target an use-case > like > > what pratyaksh mentioned? > > For delta streamer specifically, we are moving more and more towards > > continuous mode, where > > Hudi writing and compaction are amanged by a single long running spark > > application. > > > > Would Nifi also help us manage compactions when working with Hudi > > datasource or just writing plain spark Hudi pipelines? > > > > On 2019/09/18 08:18:44, Taher Koitawala <taher...@gmail.com> wrote: > > > That's another way of doing things. I want to know if someone wrote > > > something like PutParquet. Which directly can write data to Hudi. > AFAIK I > > > don't think anyone has. > > > > > > That will really be powerful. > > > > > > On Wed, Sep 18, 2019, 1:37 PM Pratyaksh Sharma <pratyaks...@gmail.com> > > > wrote: > > > > > > > Hi Taher, > > > > > > > > In the initial phase of our CDC pipeline, we were using Hudi with > Nifi. > > > > Nifi was being used to read Binlog file of mysql and to push that > data > > to > > > > some Kafka topic. This topic was then getting consumed by > > DeltaStreamer. So > > > > Nifi was indirectly involved in that flow. > > > > > > > > On Wed, Sep 18, 2019 at 10:29 AM Taher Koitawala <taher...@gmail.com > > > > > > wrote: > > > > > > > > > Hi All, > > > > > Just wanted to know has anyone tried to write data to > Hudi > > > > with a > > > > > Nifi flow? > > > > > > > > > > Perhaps may be just a csv file on local to Hudi dataset? If not > then > > lets > > > > > try that! > > > > > > > > > > Regards, > > > > > Taher Koitawala > > > > > > > > > > > > > > >
