Not too familiar wth Nifi myself. Would this still target an use-case like what pratyaksh mentioned? For delta streamer specifically, we are moving more and more towards continuous mode, where Hudi writing and compaction are amanged by a single long running spark application.
Would Nifi also help us manage compactions when working with Hudi datasource or just writing plain spark Hudi pipelines? On 2019/09/18 08:18:44, Taher Koitawala <taher...@gmail.com> wrote: > That's another way of doing things. I want to know if someone wrote > something like PutParquet. Which directly can write data to Hudi. AFAIK I > don't think anyone has. > > That will really be powerful. > > On Wed, Sep 18, 2019, 1:37 PM Pratyaksh Sharma <pratyaks...@gmail.com> > wrote: > > > Hi Taher, > > > > In the initial phase of our CDC pipeline, we were using Hudi with Nifi. > > Nifi was being used to read Binlog file of mysql and to push that data to > > some Kafka topic. This topic was then getting consumed by DeltaStreamer. So > > Nifi was indirectly involved in that flow. > > > > On Wed, Sep 18, 2019 at 10:29 AM Taher Koitawala <taher...@gmail.com> > > wrote: > > > > > Hi All, > > > Just wanted to know has anyone tried to write data to Hudi > > with a > > > Nifi flow? > > > > > > Perhaps may be just a csv file on local to Hudi dataset? If not then lets > > > try that! > > > > > > Regards, > > > Taher Koitawala > > > > > >
