[
https://issues.apache.org/jira/browse/BEAM-13162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-13162:
---------------------------------
Priority: P3 (was: P2)
> BigQueryIO Storage Write API method - suppress logging API errors for known
> use cases.
> ---------------------------------------------------------------------------------------
>
> Key: BEAM-13162
> URL: https://issues.apache.org/jira/browse/BEAM-13162
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.33.0
> Reporter: Sergei Lilichenko
> Priority: P3
> Labels: stale-P2
>
> Many BigQuery Storage API errors get logged at ERROR level even though they
> are expected and handled by the Beam's transform. They should be suppressed,
> or output at DEBUG/INFO level to differentiate from abnormal behavior.
> Example:
> {code}
> {
> "jsonPayload":
> { "message": "Got error io.grpc.StatusRuntimeException: ALREADY_EXISTS: The
> offset is within stream, expected offset 52125, received 51264 Entity:
> projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM
> closing
> projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM",
> "step": "Save Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write
> Records", "worker": "data-processing-streaming-10250846-kps9-harness-hnj3",
> "stage": "P6", "job": "2021-10-25_08_46_20-4863073108555028756", "thread":
> "258", "work": "3800026c09938fac-5cf2f68daacdd", "logger":
> "org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords" }
> ,
> "resource": {
> "type": "dataflow_step",
> "labels":
> { "project_id": "event-processing-demo", "step_id": "Save Rows to
> BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records", "job_id":
> "2021-10-25_08_46_20-4863073108555028756", "region": "us-central1",
> "job_name": "data-processing-streaming-storage-write-api-200-2" }
> },
> "timestamp": "2021-10-25T15:57:43.617Z",
> "severity": "ERROR",
> "labels":
> { "compute.googleapis.com/resource_type": "instance",
> "dataflow.googleapis.com/job_name":
> "data-processing-streaming-storage-write-api-200-2",
> "dataflow.googleapis.com/log_type": "supportability",
> "compute.googleapis.com/resource_id": "6365565973148954518",
> "dataflow.googleapis.com/job_id": "2021-10-25_08_46_20-4863073108555028756",
> "dataflow.googleapis.com/region": "us-central1",
> "compute.googleapis.com/resource_name":
> "data-processing-streaming-10250846-kps9-harness-hnj3" }
> ,
> "logName":
> "projects/event-processing-demo/logs/dataflow.googleapis.com%2Fworker",
> "receiveTimestamp": "2021-10-25T15:57:47.557344849Z"
> }
> {code}
> Additional exceptions that fit this category:
> * Got error {{io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is
> closed due to com.google.api.gax.rpc.UnavailableException:
> io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY.
> HTTP/2 error code: NO_ERROR, debug data: server_shutting_down closing}}
> * Got error {{com.google.api.gax.rpc.AbortedException:
> io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is
> restarted}}. This is expected and client is advised to reconnect.
> * Got error {{io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is
> closed due to com.google.api.gax.rpc.AbortedException:
> io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is
> restarted}}. This is expected and client is advised to reconnect.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
