Hi Rushiraj, As you said, Is the idea that provide way to pass custom parameters to the arbitrary queries pluggable? I think that user could scale the implement by SPI. And Has the work of JDBC Delta Streamer already done?
Bests, Nicholas At 2020-01-06 18:05:28, "rushiraj chavan" <[email protected]> wrote: >Hi Nicholas, > >We haven't given enough thought to it. At a high level, it looks bit >hard to generalise as we won't have any control > over the arbitrary queries. In that case, the burden would be on the >user to configure complex queries to have > incremental nature. We would also need to provide a way to pass >custom parameters to the arbitrary queries which >would be defined by the user. It is feasible but needs significant >design work. We will make note of this in our future work. > >I hope that makes sense. > >Thanks, >Rushiraj > >On Mon, Jan 6, 2020 at 2:40 PM 蒋晓峰 <[email protected]> wrote: >> >> Hi Purushotham, >> About arbitrary queries (multi table complex queries), why support this >> only in Bulk Mode? What concern about this? >> Thanks, >> Nicholas >> >> >> At 2020-01-06 15:41:31, "Purushotham Pushpavanthar" >> <[email protected]> wrote: >> >Hi everyone, >> > >> >We are working on introducing JDBC Delta Streamer as >> >one of the sources for HUDI. We've drafted initial version of design in >> >RFC-14. >> >Kindly review and let us know your thoughts. >> > >> >I'm initiating this thread to discuss few comments raised by Vinoth. >> > >> > 1. As discussed on the RFC page, we are going to support compound >> > incremental columns. >> > 2. About arbitrary queries (multi table complex queries), we are >> > planning to support this only in Bulk Mode. >> > >> > >> >[1] https://issues.apache.org/jira/browse/HUDI-251 >> >[2] >> >https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller >> > >> >Regards, >> >Purushotham Pushpavanth
