[
https://issues.apache.org/jira/browse/BEAM-13974?focusedWorklogId=733688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-733688
]
ASF GitHub Bot logged work on BEAM-13974:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Feb/22 03:22
Start Date: 28/Feb/22 03:22
Worklog Time Spent: 10m
Work Description: baeminbo commented on a change in pull request #16906:
URL: https://github.com/apache/beam/pull/16906#discussion_r815556047
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWritesShardedRecords.java
##########
@@ -404,22 +428,27 @@ public String toString() {
"Got error " + failedContext.getError() + " closing " +
failedContext.streamName);
clearClients.accept(contexts);
appendFailures.inc();
- if (statusCode.equals(Code.OUT_OF_RANGE) ||
statusCode.equals(Code.ALREADY_EXISTS)) {
+ // This means that the offset we have stored does not match the
current end of
+ // the stream in the Storage API. Usually this happens because a
crash or a bundle
+ // failure
+ // happened after an append but before the worker could
checkpoint it's
+ // state. The records that were appended in a failed bundle will
be retried,
+ // meaning that the unflushed tail of the stream must be
discarded to prevent
+ // duplicates.
+ boolean offsetMismatch =
+ statusCode.equals(Code.OUT_OF_RANGE) ||
statusCode.equals(Code.ALREADY_EXISTS);
+ // This implies that the stream doesn't exist or has already
been finalized. In this
+ // case we have no
+ // choice but to create a new stream.
+ boolean streamDoesntExist =
statusCode.equals(Code.INVALID_ARGUMENT);
Review comment:
I guess `FAILED_PRECONDITIONS` can be returned when a WriteStream is
finalized by idleness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 733688)
Time Spent: 40m (was: 0.5h)
> Idle streams in BiQuery Storage API
> -----------------------------------
>
> Key: BEAM-13974
> URL: https://issues.apache.org/jira/browse/BEAM-13974
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Reuven Lax
> Assignee: Reuven Lax
> Priority: P2
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
