vagetablechicken edited a comment on issue #3143: Non blocking OlapTableSink URL: https://github.com/apache/incubator-doris/pull/3143#issuecomment-611406948 @imay @morningman Sorry about my change is a bit late. I've changed yeild() to sleep(). And the default interval time is 10ms. Why 10ms, it's calculated through the avg time of add batch RPCs. If the case is large, it get faster obviously, so I focus on little cases this time. I tested two little cases, and got the logs about add batch time. e.g. `node add batch time(ms)/wait lock time(ms)/num: {10003:(76703)(0)(4336)} {10002:(87961)(0)(4921)} {10006:(80490)(0)(4570)} {10005:(71098)(0)(3984)} {10004:(87736)(0)(4686)}` So `avg time` = `add batch time` / `num`. And if you think the network time delay is not negligible, you can add it. In my tests, the avg times are tens of milliseconds. So for the faster sink, I set it to 10ms. And the test result(interval=10ms) is | test sample | origin ver(avg) | nonblocking ver(avg) | |-|-|-| | {"ScannedRows":28582315,"TaskNumber":1,"FileNumber":3,"FileSize":569042681} | 4min37s | 3min27s | |{"ScannedRows":100967318,"TaskNumber":1,"FileNumber":9,"FileSize":3096583144} | 12min27s | 8min27s |
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
