Hi Martin, As far as I understood the recent discussions on this matter, it would require to run a Spark standalone job in the NiFi JVM to actually write data in Delta Lake. Mike may have better ideas but, to me, it sounds like a tedious process.
Thanks, Pierre Le lun. 6 avr. 2020 à 21:40, Martin Ebert <[email protected]> a écrit : > Hi Joe, > I don't know if this is the right channel for this. But I've been talking > to the Tech Lead for Delta at Databricks: "I’m very pro any other OSS > projects that want to integrate with Delta. Feel free to put those > committers in touch with me if they are looking to build integrations." > > A general delta integration would be a great gain for the Nifi community if > one would go into the exchange here. As far as I could follow this, Mike > Thomson has dealt with delta a bit. > > Best, > Martin > > Joe Witt <[email protected]> schrieb am Fr., 27. März 2020, 21:42: > > > Martin, > > > > If I follow that blog post correctly Databricks (a vendor) in partnership > > with several other vendors created a set of integrations to feed data > into > > Delta Lake. It might make sense then for Databricks to partner with the > > appropriate vendor around Apache NiFi to have that same kind of > > engagement/collaboration/etc.. > > > > Here though - in the Apache NiFi community it is just simply about the > > community establishing some JIRAs and what it wants to do as it relates > to > > feeding data from Apache NiFi into Delta Lake and knocking it out. You > seem > > to have been learning a lot about NiFi for the past couple months. And I > > suspect you already know a good bit about Delta Lake. You're perhaps in > > the best position to help identify some things you think we should do. > If > > that is right I'm looking forward to hearing some ideas you have. There > is > > clearly something we can do here - or maybe already it works... But lets > > find out. You'll likely have to help lead/push/advocate for that but > with > > some momentum I think you'll find folks help. > > > > Thanks > > > > On Fri, Mar 27, 2020 at 3:38 PM Martin Ebert <[email protected]> > > wrote: > > > > > Hi community, > > > are there any plans to adopt this auto loader functionality: > > > > > > > > > https://databricks.com/blog/2020/02/24/introducing-databricks-ingest-easy-data-ingestion-into-delta-lake.html > > > > > > What are the plans for NiFi 2020? > > > > > > Best, > > > Martin > > > > > >
