arekusuri commented on a change in pull request #2942: GOBBLIN-1101(DSS-25241):
Enhance bulk api retry for ExceedQuota
URL: https://github.com/apache/incubator-gobblin/pull/2942#discussion_r400580522
##########
File path:
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/BulkResultIterator.java
##########
@@ -62,24 +65,51 @@ private void initHeader() {
}
private List<String> nextLineWithRetry() {
- Exception exception = null;
- for (int i = 0; i < retryLimit; i++) {
+ Throwable rootCause = null;
+ int executeCount = 0;
+ while (executeCount < retryLimit + 1) {
+ executeCount ++;
try {
if (this.csvReader == null) {
- this.csvReader = openAndSeekCsvReader(null);
+ this.csvReader = openAndSeekCsvReader(rootCause);
}
List<String> line = this.csvReader.nextRecord();
this.lineCount++;
return line;
} catch (InputStreamCSVReader.CSVParseException e) {
throw new RuntimeException(e); // don't retry if it is parse error
- } catch (Exception e) { // if it is any other exception, retry may
resolve the issue.
- exception = e;
- log.info("***Retrying***: {} - {}", fileIdVO, e.getMessage());
- this.csvReader = openAndSeekCsvReader(e);
+ } catch (OpenAndSeekException e) {
+ rootCause = e.getCause();
+ // Each organization is allowed 10 concurrent long-running requests.
If the limit is reached,
+ // any new synchronous Apex request results in a runtime exception.
+ if (e.isCurrentExceptionExceedQuta()) {
+ log.warn("--Caught ExceededQuota: " + e.getMessage());
+ threadSleep(5 * 60 * 1000); // 5 minutes
+ executeCount --; // if the current exception is Quota Exceeded, keep
trying forever
+ }
+ log.info("***Retrying***1: {} - {}", fileIdVO, e.getMessage());
+ this.csvReader = null; // in next loop, call openAndSeekCsvReader
+ } catch (Exception e) {
+ // Retry may resolve other exceptions.
+ rootCause = e;
+ threadSleep(1 * 60 * 1000); // 1 minute
Review comment:
This is meant to be random duration as well.
This kind of failure will be retried only limited times.
User wouldn't know how long to set up.
Only way to find a better duration is test and get an optimal value. However
this is not a critical value. we don't have to really find out the most optimal
value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
