Re: Hudi Concurrent Ingestion with Spark Streaming

2020-09-17 Thread nishith agarwal
Great! -Nishith On Thu, Sep 17, 2020 at 10:28 AM tanu dua wrote: > Thank you so much Nisheth. I understand now how it’s going to work. > > On Wed, 16 Sep 2020 at 11:15 PM, nishith agarwal > wrote: > > > Tanu, > > > > > > > > I'm assuming you're talking about multiple kafka partitions from a

Re: Hudi Concurrent Ingestion with Spark Streaming

2020-09-17 Thread tanu dua
Thank you so much Nisheth. I understand now how it’s going to work. On Wed, 16 Sep 2020 at 11:15 PM, nishith agarwal wrote: > Tanu, > > > > I'm assuming you're talking about multiple kafka partitions from a single > > Spark Streaming job. In this case, your job can read from > > multiple

Re: Hudi Concurrent Ingestion with Spark Streaming

2020-09-16 Thread nishith agarwal
Tanu, I'm assuming you're talking about multiple kafka partitions from a single Spark Streaming job. In this case, your job can read from multiple partitions but at the end, this data should be written to a single table. The dataset/rdd resulting from reading multiple partitions is passed as a