Hi any rfc/ongoing efforts on the reverse delta streamer ? We have a use case to do hudi => Kafka and would enjoy building a more general tool.
However we need a rfc basis to start some effort in the right way On April 12, 2023 3:08:22 AM UTC, Vinoth Chandar <mail.vinoth.chan...@gmail.com> wrote: >Cool. lets draw up a RFC for this? @pratyaksh - do you want to start one, >given you expressed interest? > >On Mon, Apr 10, 2023 at 7:32 PM Léo Biscassi <leo.bisca...@gmail.com> wrote: > >> +1 >> This would be great! >> >> Cheers, >> >> On Mon, Apr 3, 2023 at 3:00 PM Pratyaksh Sharma <pratyaks...@gmail.com> >> wrote: >> >> > Hi Vinoth, >> > >> > I am aligned with the first reason that you mentioned. Better to have a >> > separate tool to take care of this. >> > >> > On Mon, Apr 3, 2023 at 9:01 PM Vinoth Chandar < >> > mail.vinoth.chan...@gmail.com> >> > wrote: >> > >> > > +1 >> > > >> > > I was thinking that we add a new utility and NOT extend DeltaStreamer >> by >> > > adding a Sink interface, for the following reasons >> > > >> > > - It will make it look like a generic Source => Sink ETL tool, which is >> > > actually not our intention to support on Hudi. There are plenty of good >> > > tools for that out there. >> > > - the config management can get bit hard to understand, since we >> overload >> > > ingest and reverse ETL into a single tool. So break it off at use-case >> > > level? >> > > >> > > Thoughts? >> > > >> > > David: PMC does not have control over that. Please see unsubscribe >> > > instructions here. https://hudi.apache.org/community/get-involved >> > > Love to keep this thread about reverse streamer discussion. So kindly >> > fork >> > > another thread if you want to discuss unsubscribing. >> > > >> > > On Fri, Mar 31, 2023 at 1:47 AM Davidiam <david.rosa...@gmail.com> >> > wrote: >> > > >> > > > Hello Vinoth, >> > > > >> > > > Can you please unsubscribe me? I have been trying to unsubscribe for >> > > > months without success. >> > > > >> > > > Kind Regards, >> > > > David >> > > > >> > > > Sent from Outlook for Android<https://aka.ms/AAb9ysg> >> > > > ________________________________ >> > > > From: Vinoth Chandar <vin...@apache.org> >> > > > Sent: Friday, March 31, 2023 5:09:52 AM >> > > > To: dev <dev@hudi.apache.org> >> > > > Subject: [DISCUSS] Hudi Reverse Streamer >> > > > >> > > > Hi all, >> > > > >> > > > Any interest in building a reverse streaming tool, that does the >> > reverse >> > > of >> > > > what the DeltaStreamer tool does? It will read Hudi table >> incrementally >> > > > (only source) and write out the data to a variety of sinks - Kafka, >> > JDBC >> > > > Databases, DFS. >> > > > >> > > > This has come up many times with data warehouse users. Often times, >> > they >> > > > want to use Hudi to speed up or reduce costs on their data ingestion >> > and >> > > > ETL (using Spark/Flink), but want to move the derived data back into >> a >> > > data >> > > > warehouse or an operational database for serving. >> > > > >> > > > What do you all think? >> > > > >> > > > Thanks >> > > > Vinoth >> > > > >> > > >> > >> >> >> -- >> *Léo Biscassi* >> Blog - https://leobiscassi.com >> >> - >>
