+1 I was thinking that we add a new utility and NOT extend DeltaStreamer by adding a Sink interface, for the following reasons
- It will make it look like a generic Source => Sink ETL tool, which is actually not our intention to support on Hudi. There are plenty of good tools for that out there. - the config management can get bit hard to understand, since we overload ingest and reverse ETL into a single tool. So break it off at use-case level? Thoughts? David: PMC does not have control over that. Please see unsubscribe instructions here. https://hudi.apache.org/community/get-involved Love to keep this thread about reverse streamer discussion. So kindly fork another thread if you want to discuss unsubscribing. On Fri, Mar 31, 2023 at 1:47 AM Davidiam <david.rosa...@gmail.com> wrote: > Hello Vinoth, > > Can you please unsubscribe me? I have been trying to unsubscribe for > months without success. > > Kind Regards, > David > > Sent from Outlook for Android<https://aka.ms/AAb9ysg> > ________________________________ > From: Vinoth Chandar <vin...@apache.org> > Sent: Friday, March 31, 2023 5:09:52 AM > To: dev <dev@hudi.apache.org> > Subject: [DISCUSS] Hudi Reverse Streamer > > Hi all, > > Any interest in building a reverse streaming tool, that does the reverse of > what the DeltaStreamer tool does? It will read Hudi table incrementally > (only source) and write out the data to a variety of sinks - Kafka, JDBC > Databases, DFS. > > This has come up many times with data warehouse users. Often times, they > want to use Hudi to speed up or reduce costs on their data ingestion and > ETL (using Spark/Flink), but want to move the derived data back into a data > warehouse or an operational database for serving. > > What do you all think? > > Thanks > Vinoth >
