Hi Vinoth,
Nifi has the capability to pass data to a custom spark job.
However that is done through a StreamingContext, not sure if we can build
something on this. I'm trying to wrap my head around how to fit the
StreamingContext in our existing code.
Here is an example:
https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
Regards,
Taher Koitawala
On Wed, Sep 18, 2019, 8:27 PM Vinoth Chandar <vin...@apache.org> wrote:
> Not too familiar wth Nifi myself. Would this still target an use-case like
> what pratyaksh mentioned?
> For delta streamer specifically, we are moving more and more towards
> continuous mode, where
> Hudi writing and compaction are amanged by a single long running spark
> application.
>
> Would Nifi also help us manage compactions when working with Hudi
> datasource or just writing plain spark Hudi pipelines?
>
> On 2019/09/18 08:18:44, Taher Koitawala <taher...@gmail.com> wrote:
> > That's another way of doing things. I want to know if someone wrote
> > something like PutParquet. Which directly can write data to Hudi. AFAIK I
> > don't think anyone has.
> >
> > That will really be powerful.
> >
> > On Wed, Sep 18, 2019, 1:37 PM Pratyaksh Sharma <pratyaks...@gmail.com>
> > wrote:
> >
> > > Hi Taher,
> > >
> > > In the initial phase of our CDC pipeline, we were using Hudi with Nifi.
> > > Nifi was being used to read Binlog file of mysql and to push that data
> to
> > > some Kafka topic. This topic was then getting consumed by
> DeltaStreamer. So
> > > Nifi was indirectly involved in that flow.
> > >
> > > On Wed, Sep 18, 2019 at 10:29 AM Taher Koitawala <taher...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > > Just wanted to know has anyone tried to write data to Hudi
> > > with a
> > > > Nifi flow?
> > > >
> > > > Perhaps may be just a csv file on local to Hudi dataset? If not then
> lets
> > > > try that!
> > > >
> > > > Regards,
> > > > Taher Koitawala
> > > >
> > >
> >
>
