I think we will have to make a Nifi Processor. The Nifi processor should host all what do with Spark to write data. We will have to scope out the work on this and compactions.
Regards, Taher Koitawala On Wed, Sep 18, 2019, 8:30 PM Suneel Marthi <smar...@apache.org> wrote: > Adding Nifi dev@ to this thread. > > > On Wed, Sep 18, 2019 at 10:57 AM Vinoth Chandar <vin...@apache.org> wrote: > > > Not too familiar wth Nifi myself. Would this still target an use-case > like > > what pratyaksh mentioned? > > For delta streamer specifically, we are moving more and more towards > > continuous mode, where > > Hudi writing and compaction are amanged by a single long running spark > > application. > > > > Would Nifi also help us manage compactions when working with Hudi > > datasource or just writing plain spark Hudi pipelines? > > > > On 2019/09/18 08:18:44, Taher Koitawala <taher...@gmail.com> wrote: > > > That's another way of doing things. I want to know if someone wrote > > > something like PutParquet. Which directly can write data to Hudi. > AFAIK I > > > don't think anyone has. > > > > > > That will really be powerful. > > > > > > On Wed, Sep 18, 2019, 1:37 PM Pratyaksh Sharma <pratyaks...@gmail.com> > > > wrote: > > > > > > > Hi Taher, > > > > > > > > In the initial phase of our CDC pipeline, we were using Hudi with > Nifi. > > > > Nifi was being used to read Binlog file of mysql and to push that > data > > to > > > > some Kafka topic. This topic was then getting consumed by > > DeltaStreamer. So > > > > Nifi was indirectly involved in that flow. > > > > > > > > On Wed, Sep 18, 2019 at 10:29 AM Taher Koitawala <taher...@gmail.com > > > > > > wrote: > > > > > > > > > Hi All, > > > > > Just wanted to know has anyone tried to write data to > Hudi > > > > with a > > > > > Nifi flow? > > > > > > > > > > Perhaps may be just a csv file on local to Hudi dataset? If not > then > > lets > > > > > try that! > > > > > > > > > > Regards, > > > > > Taher Koitawala > > > > > > > > > > > > > > >
