stankiewicz commented on code in PR #36832:
URL: https://github.com/apache/beam/pull/36832#discussion_r2535383345
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageApiInsertError.java:
##########
@@ -43,6 +53,40 @@ public String getErrorMessage() {
return errorMessage;
}
+ @Nullable
+ public String getTableUrn() {
+ return tableUrn;
+ }
+
+ @Nullable
+ public String getProjectId() {
+ return getParsedPart(1);
Review Comment:
once you have tableReference, parsing is not needed.
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageApiInsertErrorCoder.java:
##########
@@ -42,12 +42,15 @@ public void encode(BigQueryStorageApiInsertError value,
OutputStream outStream)
throws IOException {
TABLE_ROW_CODER.encode(value.getRow(), outStream);
STRING_CODER.encode(value.getErrorMessage(), outStream);
+ STRING_CODER.encode(value.getTableUrn(), outStream);
}
@Override
public BigQueryStorageApiInsertError decode(InputStream inStream)
throws CoderException, IOException {
return new BigQueryStorageApiInsertError(
- TABLE_ROW_CODER.decode(inStream), STRING_CODER.decode(inStream));
+ TABLE_ROW_CODER.decode(inStream),
+ STRING_CODER.decode(inStream),
+ STRING_CODER.decode(inStream));
Review Comment:
use BigQueryHelpers.parseTableSpec(STRING_CODER.decode(inStream))
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageApiInsertErrorCoder.java:
##########
@@ -42,12 +42,15 @@ public void encode(BigQueryStorageApiInsertError value,
OutputStream outStream)
throws IOException {
TABLE_ROW_CODER.encode(value.getRow(), outStream);
STRING_CODER.encode(value.getErrorMessage(), outStream);
+ STRING_CODER.encode(value.getTableUrn(), outStream);
Review Comment:
use BigQueryHelpers.toTableSpec(tableReference)
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageApiInsertError.java:
##########
@@ -43,6 +53,40 @@ public String getErrorMessage() {
return errorMessage;
}
+ @Nullable
+ public String getTableUrn() {
+ return tableUrn;
+ }
+
+ @Nullable
+ public String getProjectId() {
+ return getParsedPart(1);
+ }
+
+ @Nullable
+ public String getDatasetId() {
+ return getParsedPart(3);
+ }
+
+ @Nullable
+ public String getTableId() {
+ return getParsedPart(5);
+ }
+
+ @Nullable
+ private String getParsedPart(int index) {
Review Comment:
use build in features/helpers eg BigQueryHelpers.parseTableSpec(tableSpec)
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageApiInsertError.java:
##########
@@ -25,13 +25,23 @@ public class BigQueryStorageApiInsertError {
private @Nullable String errorMessage;
+ private @Nullable String tableUrn;
+
+ private @Nullable String[] parsedParts;
+
public BigQueryStorageApiInsertError(TableRow row) {
- this.row = row;
+ this(row, null, null);
}
public BigQueryStorageApiInsertError(TableRow row, @Nullable String
errorMessage) {
+ this(row, errorMessage, null);
+ }
+
+ public BigQueryStorageApiInsertError(
+ TableRow row, @Nullable String errorMessage, @Nullable String tableUrn) {
Review Comment:
maybe better to pass com. google. api. services. bigquery. model
TableReference?
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiConvertMessages.java:
##########
@@ -186,10 +186,15 @@ public void processElement(
badRecordRouter.route(o, element, elementCoder, e, "Unable to
convert value to TableRow");
return;
}
+ String tableUrn = null;
+ TableDestination tableDestination =
dynamicDestinations.getTable(element.getKey());
+ if (tableDestination != null) {
+ tableUrn =
tableDestination.getTableUrn(pipelineOptions.as(BigQueryOptions.class));
+ }
o.get(failedWritesTag)
.output(
new BigQueryStorageApiInsertError(
- failsafeTableRow, conversionException.toString()));
+ failsafeTableRow, conversionException.toString(),
tableUrn));
Review Comment:
tableDestination.getTableReference()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
