liferoad commented on PR #35051: URL: https://github.com/apache/beam/pull/35051#issuecomment-3092519057
> Maybe it's too much to add an integration test here. But we can try running it locally with modified code that forces an end of stream error. Can you try adding an increment to the offset in the line below and running a pipeline locally to see if your new code path gets triggered? e.g. `return datasetService.flush(streamId, offset + 50); ` > > https://github.com/apache/beam/blob/52e42d72df16dace19674508d448bf6f0f92a4f6/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiFlushAndFinalizeDoFn.java#L157 Good idea. Manually changed the offset and ran this: ``` ./gradlew :sdks:java:io:google-cloud-platform:integrationTest --tests "org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageWriteIT.testBigQueryStorageWrite3MProto" -PgcpProject=apache-beam-testing -PgcpTempRoot=gs://temp-storage-for-end-to-end-tests -i > test_output.log ``` `testBigQueryStorageWrite3MProto` failed as expected and also I added the log to test `isOffsetBeyondEndOfStreamError` is called with ``` private boolean isOffsetBeyondEndOfStreamError(Throwable t) { LOG.info("Checking for offset beyond end of stream error."); if (t == null) { return false; } ``` ``` Jul 19, 2025 2:51:21 PM org.apache.beam.sdk.io.gcp.bigquery.StorageApiFlushAndFinalizeDoFn isOffsetBeyondEndOfStreamError INFO: Checking for offset beyond end of stream error. Jul 19, 2025 2:51:21 PM org.apache.beam.sdk.io.gcp.bigquery.StorageApiFlushAndFinalizeDoFn lambda$process$1 WARNING: Flush of stream projects/apache-beam-testing/datasets/big_query_storage_write_it_1752951070961_24/tables/storage_write_1752951073468/streams/Cic2YmJiYjNlNS0wMDAwLTIwMmQtYWI0NS1kNGY1NDdmZDExZDQ6czQ to offset 1499 failed because the offset is beyond the end of the stream. This typically means the stream was finalized or truncated by BQ. The operation will not be retried on this stream. Error: com.google.api.gax.rpc.OutOfRangeException: io.grpc.StatusRuntimeException: OUT_OF_RANGE: Offset 1549 is beyond the end of the stream Entity: projects/apache-beam-testing/datasets/big_query_storage_write_it_1752951070961_24/tables/storage_write_1752951073468/streams/Cic2YmJiYjNlNS0wMDAwLTIwMmQtYWI0NS1kNGY1NDdmZDExZDQ6czQ ``` ``` LOG.warn( "Flush of stream {} to offset {} failed because the offset is beyond the end of the stream. "` `` ` is also shown correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
