kalencaya commented on issue #1270:
URL:
https://github.com/apache/incubator-seatunnel/issues/1270#issuecomment-1053830971
thanks for your sharing.
My data sync works are always limited by some http interfaces ratelimiter,
especially for incremental sync.
A classical http interface which supports incremental sync contains below
request params or variant:
```
{
"startTime": "yyyy-MM-dd HH:mm:ss",
"endTime": "yyyy-MM-dd HH:mm:ss",
"pageIndex": 1,
"pageSize": 50
}
```
For ratelimiter and request params limitions, I have to controll concurrency
and handle annoying network failure to get accurate and quick sync result. So,
my core work is split sync task, then execute concurrently, update sync
offset(latest synced time interval), I also belives flink source framework can
do that process better.
Http authorization requires every request provides credentials, and I have
more than 2500 credential account. Unbalanced hot data can appear in any
credential account and any time, I had started a sync job for every credential
account and every http interface, which just for incremental scenarios. How to
handle numerous http interface credentials now really blocks me.
Now, I'm taking advantage of akka excellent concurrency and xxl-job to
refactoring sync framework, you can contact me by wechat which is also
kalencaya.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
