Re: Recursive Split Detection + same split optimization

2023-07-10 Thread Benoit Tailhades
Thank you Hang for your answer. Regarding your proposal 2, implementing such logic will prevent parallelizing on TM, since from the 1st ID, I will fetch n IDs, but with this approach, all IDs will finally be managed by the same TM. However, I am not totally satisfied with the 1st choice which is

Re: Recursive Split Detection + same split optimization

2023-07-10 Thread Hang Ruan
Hi, Benoit. A split enumerator responsible for discovering the source splits, and assigning them to the reader. It seems like that your connector discovering splits in TM and assigning them in JM. I think there are 2 choices: 1. If you need the enumerator to assign splits, you have to send the

Recursive Split Detection + same split optimization

2023-07-09 Thread Benoit Tailhades
Hello Everyone, I am trying to implement a custom source where split detection is an expensive operation and I would like to benefit from the split reader results to build my next splits. Basically, my source is taking as input an id from my external system, let's name it ID1. >From ID1, I can