I think this will be a good opportunity to plan better in terms of abstraction too which is needed for the Flink and Beam engines we might use.
Regards, Taher Koitawala On Sun, Sep 22, 2019, 3:37 PM leesf <[email protected]> wrote: > +1. > Happy to see DeltaStreamer becomes more and more powerful. Also, we need to > pay some attention to the layout and organization of these connectors as > more and more data sources introduced to HUDI like vinoyang suggested. > > Best, > Leesf > > Bhavani Sudha Saktheeswaran <[email protected]> 于2019年9月22日周日 > 下午12:18写道： > > > +1 to adding more connectors to DeltStreamer and making them as much > > pluggable modules as possible like Vino Yang suggested. > > > > > > On Sat, Sep 21, 2019 at 7:12 PM vino yang <[email protected]> wrote: > > > > > + 1 to introduce these connectors. It's nice to see that Hudi's > ecosystem > > > is growing. As Hudi connects to more and more systems, it is necessary > to > > > introduce separate modules to place these connectors. This can lead to > > > module relayout or code refactoring. Of course, all this needs to be > > > discussed in more depth. Best， Vino On 09/21/2019 18:59, Vinay Patil > > wrote: > > > Hi Taher, Basically this can be proposal to support Kinesis and > DynamoDb > > > stream support can be enabled by reusing this source code. Flink has > > > provided support for DynamoDb Streams by reusing Kinesis Streams > classes. > > > Regards, Vinay Patil On Sat, Sep 21, 2019 at 4:26 PM Taher Koitawala < > > > [email protected]> wrote: > That would be a great addition Vinay. How > > > about adding Kinesis as well? > > Regards, > Taher Koitawala > > On > Sat, > > > Sep 21, 2019, 4:20 PM Vinay Patil <[email protected]> wrote: > > > > > > > Hi Team, > > > > The DynamoDb streams contains the CDC data when > enabled > > on > > > a DynamoDb > > table, we can add a source for DeltaStreamer which will > > > enable us to read > > this data and write it back either to Hudi > dataset > > or > > > to another sink. > > > > > > Thoughts on adding this support in Hudi ? > > > > > > > > > > > > Regards, > > Vinay Patil > > > > > >
