arekusuri edited a comment on issue #2942: GOBBLIN-1101(DSS-25241): Enhance bulk api retry for ExceedQuota URL: https://github.com/apache/incubator-gobblin/pull/2942#issuecomment-607071075 > This affects resource utilization and latency, so I think users may want to control that based on their workload and priorities. For example, they may want the ETL job to fail immediately and not retry or to have a longer polling interval to reduce contention with higher priority jobs. @htran1 thanks for your above comment. Add sleep in the retry logic is to avoid resource consuming peak. The sleep duration should be a random number. Nobody knows what is the right number to set up. (to do a lot of experiments may help find out a optimal number) It is not depending anything like table size. Flow job developer would not be able to figure out what number to set up. Leave users the key can only confuse them more. > they may want the ETL job to fail immediately Very good question. We have key "salesforce.fetchRetryLimit", if set to 0, there won't be retry. Default retry limit is 5.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
