[
https://issues.apache.org/jira/browse/BEAM-13931?focusedWorklogId=727128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-727128
]
ASF GitHub Bot logged work on BEAM-13931:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Feb/22 16:19
Start Date: 15/Feb/22 16:19
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #16838:
URL: https://github.com/apache/beam/pull/16838#issuecomment-1040477581
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 727128)
Time Spent: 2h 10m (was: 2h)
> BigQueryIO is sending rows that are too large to Deadletter Queue even on
> RETRY_ALWAYS
> --------------------------------------------------------------------------------------
>
> Key: BEAM-13931
> URL: https://issues.apache.org/jira/browse/BEAM-13931
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.35.0, 2.36.0
> Reporter: Pablo Estrada
> Assignee: Pablo Estrada
> Priority: P1
> Fix For: 2.37.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Note that BQ does not support requests over a certain size, and rows that go
> past the size may be output into a dead-letter queue that they can get back
> with
> [BigQueryIO.Write.Result.getFailedInsertsWithErr|https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html#getFailedInsertsWithErr--]
> A change went into Beam that outputs rows into the BQIO DLQ even if they're
> meant to be retried indefinitely.
> [https://github.com/apache/beam/commit/1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc#diff-26dbe8f625f702ae3edacdbc02b12acc6e423542fe16835229e22ef8eb4e109cR979-R989]
>
>
> A workaround is to set this pipeline option to a larger amount:
> [https://github.com/apache/beam/blob/1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java#L70]
>
> Currently it's 64KB, which is relatively small. Setting it to 1MB or 5MB or
> so should work around this issue (it should be larger than the maximum row
> size) - gRPC should support up to 10MB request sizes.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
