chamikaramj commented on a change in pull request #12106:
URL: https://github.com/apache/beam/pull/12106#discussion_r451179908
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
##########
@@ -793,6 +795,14 @@ public void deleteDataset(String projectId, String
datasetId)
String.format(
"BigQuery insertAll error, retrying: %s",
ApiErrorExtractor.INSTANCE.getErrorMessage(e)));
+ GoogleJsonError.ErrorInfo errorInfo =
getErrorInfo(e);
+ if (errorInfo == null) {
+ return insert.execute().getInsertErrors();
+ }
+ if (!ApiErrorExtractor.INSTANCE.rateLimited(e)
+ &&
!errorInfo.getReason().equals("quotaExceeded")) {
Review comment:
I would move this to a String constant. Also probably add a reference to
here: https://cloud.google.com/bigquery/docs/error-messages
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
##########
@@ -793,6 +795,14 @@ public void deleteDataset(String projectId, String
datasetId)
String.format(
"BigQuery insertAll error, retrying: %s",
ApiErrorExtractor.INSTANCE.getErrorMessage(e)));
+ GoogleJsonError.ErrorInfo errorInfo =
getErrorInfo(e);
+ if (errorInfo == null) {
+ return insert.execute().getInsertErrors();
Review comment:
Why are we performing an extra retry here ?
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
##########
@@ -893,6 +903,20 @@ public void deleteDataset(String projectId, String
datasetId)
ignoreInsertIds);
}
+ private GoogleJsonError.ErrorInfo getErrorInfo(IOException e) {
+ GoogleJsonResponseException jsonCause = null;
+ Throwable eCause = e;
+ if (eCause instanceof GoogleJsonResponseException) {
Review comment:
How do these exceptions extend both GoogleJsonResponseException and
IOException ?
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
##########
@@ -893,6 +903,20 @@ public void deleteDataset(String projectId, String
datasetId)
ignoreInsertIds);
}
+ private GoogleJsonError.ErrorInfo getErrorInfo(IOException e) {
+ GoogleJsonResponseException jsonCause = null;
+ Throwable eCause = e;
+ if (eCause instanceof GoogleJsonResponseException) {
+ jsonCause = (GoogleJsonResponseException) eCause;
+ } else {
+ return null;
+ }
+ GoogleJsonError jsonError = jsonCause.getDetails();
Review comment:
Please add a unit test for this method.
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
##########
@@ -793,6 +795,14 @@ public void deleteDataset(String projectId, String
datasetId)
String.format(
"BigQuery insertAll error, retrying: %s",
ApiErrorExtractor.INSTANCE.getErrorMessage(e)));
+ GoogleJsonError.ErrorInfo errorInfo =
getErrorInfo(e);
+ if (errorInfo == null) {
+ return insert.execute().getInsertErrors();
+ }
+ if (!ApiErrorExtractor.INSTANCE.rateLimited(e)
+ &&
!errorInfo.getReason().equals("quotaExceeded")) {
+ return insert.execute().getInsertErrors();
Review comment:
Ditto regarding the extra retry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
