tmljob commented on issue #1270:
URL:
https://github.com/apache/incubator-seatunnel/issues/1270#issuecomment-1054021091
> thanks for your sharing.
>
> My data sync works are always limited by some http interfaces ratelimiter,
especially for incremental sync. A classical http interface which supports
incremental sync contains below request params or variant:
>
> ```
> {
> "startTime": "yyyy-MM-dd HH:mm:ss",
> "endTime": "yyyy-MM-dd HH:mm:ss",
> "pageIndex": 1,
> "pageSize": 50
> }
> ```
>
> For ratelimiter and request params limitions, I have to controll
concurrency and handle annoying network failure to get accurate and quick sync
result. So, my core work is split sync task, then execute concurrently, update
sync offset(latest synced time interval), I also belives flink source framework
can do that process better.
>
> Http authorization requires every request provides credentials, and I have
more than 2500 credential account. Unbalanced hot data can appear in any
credential account and any time, I had started a sync job for every credential
account and every http interface, which just for incremental scenarios. How to
handle numerous http interface credentials now really blocks me.
>
> Now, I'm taking advantage of akka excellent concurrency and xxl-job to
refactoring sync framework, you can contact me by wechat which is also
kalencaya.
I can't find you through this WeChat account, please see if you can add me
through this account: **tmljob123**, I will pull up a WeChat group and discuss
with another brother.
@kalencaya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
