[
https://issues.apache.org/jira/browse/GOBBLIN-1202?focusedWorklogId=449469&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-449469
]
ASF GitHub Bot logged work on GOBBLIN-1202:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 22/Jun/20 19:53
Start Date: 22/Jun/20 19:53
Worklog Time Spent: 10m
Work Description: arekusuri commented on a change in pull request #3049:
URL: https://github.com/apache/incubator-gobblin/pull/3049#discussion_r443789518
##########
File path:
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
##########
@@ -417,23 +425,31 @@ private int computeTargetPartitionSize(Histogram
histogram, int minTargetPartiti
* Get a {@link JsonArray} containing the query results
*/
private JsonArray getRecordsForQuery(SalesforceConnector connector, String
query) {
- try {
- String soqlQuery = SalesforceExtractor.getSoqlUrl(query);
- List<Command> commands =
RestApiConnector.constructGetCommand(connector.getFullUri(soqlQuery));
- CommandOutput<?, ?> response = connector.getResponse(commands);
+ RestApiProcessingException exception = null;
+ for (int i = 0; i < workUnitConf.restRetries; i++) {
+ try {
+ String soqlQuery = SalesforceExtractor.getSoqlUrl(query);
+ List<Command> commands =
RestApiConnector.constructGetCommand(connector.getFullUri(soqlQuery));
+ CommandOutput<?, ?> response = connector.getResponse(commands);
+
+ String output;
+ Iterator<String> itr = (Iterator<String>)
response.getResults().values().iterator();
+ if (itr.hasNext()) {
+ output = itr.next();
+ } else {
+ throw new DataRecordException("Failed to get data from salesforce;
REST response has no output");
+ }
- String output;
- Iterator<String> itr = (Iterator<String>)
response.getResults().values().iterator();
- if (itr.hasNext()) {
- output = itr.next();
- } else {
- throw new DataRecordException("Failed to get data from salesforce;
REST response has no output");
+ return GSON.fromJson(output,
JsonObject.class).getAsJsonArray("records");
+ } catch (RestApiClientException | DataRecordException e) {
+ throw new RuntimeException("Fail to get data from salesforce", e);
+ } catch (RestApiProcessingException e) {
+ exception = e;
+ log.info(String.format("Caught RestApiProcessingException, retry(%s)
rest query: %s", i+1, query));
Review comment:
good catch, fixed!
# And added wait duration
# refactored variable name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 449469)
Time Spent: 0.5h (was: 20m)
> Add retry for REST API call
> ---------------------------
>
> Key: GOBBLIN-1202
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1202
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Alex Li
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> SFDC objects have index on their column - *SystemModstamp*
> This index could be in disk. When we execute
> {code:java}
> Select count(systemmodstamp) from table_name group by day_only(systemmodstamp)
> {code}
> If the index is in disk, it needs to load. It would be timeout.
> Retry would result it.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
