Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
ahmedabu98 merged PR #31659: URL: https://github.com/apache/beam/pull/31659 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
Amar3tto commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2284676887 Run Java_GCP_IO_Direct PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
Amar3tto commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2284524270 > @Amar3tto can we add a comment in CHANGES.md calling out this new feature? Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
ahmedabu98 commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2276206894 @Amar3tto can we add a comment in CHANGES.md calling out this new feature? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
ahmedabu98 commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2271583547 CC @johnjcasey @AnandInguva I believe this does what y'all wanted to do in #31106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
ahmedabu98 commented on code in PR #31659:
URL: https://github.com/apache/beam/pull/31659#discussion_r1705736964
##
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java:
##
@@ -3050,7 +3051,312 @@ public void
testStreamingInsertsExtendedErrorRetrieval() throws Exception {
}
@Test
- public void testStorageApiErrors() throws Exception {
+ public void testStorageApiErrorsWriteProto() throws Exception {
+assumeTrue(useStorageApi);
+final Method method =
+useStorageApiApproximate ? Method.STORAGE_API_AT_LEAST_ONCE :
Method.STORAGE_WRITE_API;
+
+final int failFrom = 10;
+
+Function getPrimitive =
+(Integer i) ->
+Proto3SchemaMessages.Primitive.newBuilder()
+.setPrimitiveDouble(i)
+.setPrimitiveFloat(i)
+.setPrimitiveInt32(i)
+.setPrimitiveInt64(i)
+.setPrimitiveUint32(i)
+.setPrimitiveUint64(i)
+.setPrimitiveSint32(i)
+.setPrimitiveSint64(i)
+.setPrimitiveFixed32(i)
+.setPrimitiveFixed64(i)
+.setPrimitiveBool(true)
+.setPrimitiveString(Integer.toString(i))
+.setPrimitiveBytes(
+
ByteString.copyFrom(Integer.toString(i).getBytes(StandardCharsets.UTF_8)))
+.build();
+List goodRows =
Review Comment:
nit: --> s/goodRows/inputRows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
Amar3tto commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2268581243 @ahmedabu98 Could you please review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
Abacn commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2225967719 Didn't get a chance to go through the change. I see the author of the FR @sarinasij is responsive. Would it be possible for you testing the change with your use case, to see if this unblocks you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
Amar3tto commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2225761162 @Abacn Could you please help with the review process? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods [beam]
Amar3tto commented on PR #31659: URL: https://github.com/apache/beam/pull/31659#issuecomment-2223186487 R: @ahmedabu98 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
