Hi Vinoth, I have provided the answers to your questions.
> *should we just integrate to Kinesis? If DynamoDB will pump its changes into Kinesis* *anyway, why should we aware of DynanoDB directly?* - Yes, we should first integrate with Kinesis. As I mentioned once the stream is Enabled on DynamoDb table , the CDC data can be accessed from the shards in real time. So adding support for DynamoDb streams will be a subtask of Kinesis. > If DynamoDB will pump its changes into Kinesis anyway, why should we aware of DynamoDB directly? - Yes, we don't need to talk to DynamoDB table directly but with the streams enabled on it [1] > does kinesis streams have schemas mapped from DynamoDB already or should we be implementing a DynamoDBSchemaProvider as well? - IMO, we don't need to be aware about the schema here, we will be getting only the CDC data in this stream[1] and the schema can be different for each record ( adding or removing a column) 1. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html Regards, Vinay Patil On Sun, Sep 22, 2019 at 6:28 PM Vinoth Chandar <[email protected]> wrote: > +1 For now we can keep this in hudi-utilities itself IMO. > > As for the connector or Deltastreamer Source to be specific, should we just > integrate to Kinesis? If DynamoDB will pump its changes into Kinesis > anyway, why should we aware of DynanoDB directly? > Also we may need to rethink how we are going to maintain the schema? does > kinesis streams have schemas mapped from DynamoDB already or should we be > implementing a DynamoDBSchemaProvider as well? > > This would be a really great addition. But also can see how challenging it > can be (which is fun :)) > > On Sun, Sep 22, 2019 at 4:09 AM Taher Koitawala <[email protected]> > wrote: > > > I think this will be a good opportunity to plan better in terms of > > abstraction too which is needed for the Flink and Beam engines we might > > use. > > > > Regards, > > Taher Koitawala > > > > On Sun, Sep 22, 2019, 3:37 PM leesf <[email protected]> wrote: > > > > > +1. > > > Happy to see DeltaStreamer becomes more and more powerful. Also, we > need > > to > > > pay some attention to the layout and organization of these connectors > as > > > more and more data sources introduced to HUDI like vinoyang suggested. > > > > > > Best, > > > Leesf > > > > > > Bhavani Sudha Saktheeswaran <[email protected]> 于2019年9月22日周日 > > > 下午12:18写道： > > > > > > > +1 to adding more connectors to DeltStreamer and making them as much > > > > pluggable modules as possible like Vino Yang suggested. > > > > > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM vino yang <[email protected]> > > wrote: > > > > > > > > > + 1 to introduce these connectors. It's nice to see that Hudi's > > > ecosystem > > > > > is growing. As Hudi connects to more and more systems, it is > > necessary > > > to > > > > > introduce separate modules to place these connectors. This can lead > > to > > > > > module relayout or code refactoring. Of course, all this needs to > be > > > > > discussed in more depth. Best， Vino On 09/21/2019 18:59, Vinay > Patil > > > > wrote: > > > > > Hi Taher, Basically this can be proposal to support Kinesis and > > > DynamoDb > > > > > stream support can be enabled by reusing this source code. Flink > has > > > > > provided support for DynamoDb Streams by reusing Kinesis Streams > > > classes. > > > > > Regards, Vinay Patil On Sat, Sep 21, 2019 at 4:26 PM Taher > Koitawala > > < > > > > > [email protected]> wrote: > That would be a great addition Vinay. > > How > > > > > about adding Kinesis as well? > > Regards, > Taher Koitawala > > On > > > Sat, > > > > > Sep 21, 2019, 4:20 PM Vinay Patil <[email protected]> > wrote: > > > > > > > > > > > Hi Team, > > > > The DynamoDb streams contains the CDC data when > > > enabled > > > > on > > > > > a DynamoDb > > table, we can add a source for DeltaStreamer which > > will > > > > > enable us to read > > this data and write it back either to Hudi > > > dataset > > > > or > > > > > to another sink. > > > > > > Thoughts on adding this support in > Hudi > > ? > > > > > > > > > > > > > > > > > > Regards, > > Vinay Patil > > > > > > > > > > > > >
