DanielLeens commented on issue #10929: URL: https://github.com/apache/seatunnel/issues/10929#issuecomment-4530993272
Thanks, this response example is very helpful. With this terminal-page sample, the problem is now much clearer: the API signals end-of-data by keeping the same non-empty `scrollId`, returning `data = []`, and setting `isLastData = true`. The current cursor path in `HttpSourceReader` still only stops when the next cursor becomes empty, so this kind of response will indeed be treated as "keep going", even though the page is already terminal. I still would not make `readSize < batch_size` the universal stop rule for all cursor-based APIs, because some cursor APIs can legally return a short page before the real end. A safer direction would be: 1. add a no-progress guard for repeated cursor / repeated request state 2. evaluate support for a configurable response end flag, for cases like your `isLastData = true` 3. keep the existing empty-cursor behavior as another valid termination path So this should stay open as a real bug in the cursor-pagination termination logic. Thanks again for providing a concrete terminal response example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
