Hi Nicholas, We discussed about incremental mode for arbitrary queries at our end and we came up with an idea. We can put a filtering criteria using where clause and provide placeholder for incrementing conditions.
Thanks, Rushiraj On Mon, Jan 6, 2020 at 4:50 PM rushiraj chavan <rushirajcha...@gmail.com> wrote: > > Hi Nicholas, > > I guess it can be made pluggable. Sorry I didn't understand what you > meant by ... SPI.... > > Purushotham(pushpavant...@gmail.com) and I are working on JDBC Delta > Streamer. It is WIP. > > Thanks, > Rushiraj > > On Mon, Jan 6, 2020 at 4:20 PM 蒋晓峰 <programg...@163.com> wrote: > > > > Hi Rushiraj, > > As you said, Is the idea that provide way to pass custom parameters to the > > arbitrary queries pluggable? I think that user could scale the implement by > > SPI. And Has the work of JDBC Delta Streamer already done? > > > > > > Bests, > > Nicholas > > > > At 2020-01-06 18:05:28, "rushiraj chavan" <rushirajcha...@gmail.com> wrote: > > >Hi Nicholas, > > > > > >We haven't given enough thought to it. At a high level, it looks bit > > >hard to generalise as we won't have any control > > > over the arbitrary queries. In that case, the burden would be on the > > >user to configure complex queries to have > > > incremental nature. We would also need to provide a way to pass > > >custom parameters to the arbitrary queries which > > >would be defined by the user. It is feasible but needs significant > > >design work. We will make note of this in our future work. > > > > > >I hope that makes sense. > > > > > >Thanks, > > >Rushiraj > > > > > >On Mon, Jan 6, 2020 at 2:40 PM 蒋晓峰 <programg...@163.com> wrote: > > >> > > >> Hi Purushotham, > > >> About arbitrary queries (multi table complex queries), why support > > >> this only in Bulk Mode? What concern about this? > > >> Thanks, > > >> Nicholas > > >> > > >> > > >> At 2020-01-06 15:41:31, "Purushotham Pushpavanthar" > > >> <pushpavant...@gmail.com> wrote: > > >> >Hi everyone, > > >> > > > >> >We are working on introducing JDBC Delta Streamer as > > >> >one of the sources for HUDI. We've drafted initial version of design in > > >> >RFC-14. > > >> >Kindly review and let us know your thoughts. > > >> > > > >> >I'm initiating this thread to discuss few comments raised by Vinoth. > > >> > > > >> > 1. As discussed on the RFC page, we are going to support compound > > >> > incremental columns. > > >> > 2. About arbitrary queries (multi table complex queries), we are > > >> > planning to support this only in Bulk Mode. > > >> > > > >> > > > >> >[1] https://issues.apache.org/jira/browse/HUDI-251 > > >> >[2] > > >> >https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller > > >> > > > >> >Regards, > > >> >Purushotham Pushpavanth
