[
https://issues.apache.org/jira/browse/BEAM-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Amato updated BEAM-12139:
------------------------------
Description:
When this API yields errors specific to failed inserts for a row.
Rows are selected [here for
retrying|https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L967],
using the errorIndex which is returned from the error.
retryRows.add(rowsToPublish.get(errorIndex));
However, this errorIndex is not valid to index rowsToPublish. So it looks like
the wrong rows are being selected to be retried.
*Why can't you use errorIndex to index rowsToPublish?*
because rowsToPublish contains all of the rows which were passed into insertAll.
These are then batched into a smaller list of
["rows"|https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L875]
, where multpile API calls are made to bigquery to insert the rows.
The errors returned actually refer to the list of rows passed into the call
made to BigQuery, so they are only valid indices for "rows". Thus, they are not
valid indices for "rowsToPublish".
Note: These lists have a different number of rows: rowsToPublish.size() >
rows.size()
was:
When this API yields errors specific to failed inserts for a row.
Rows are selected [here for
retrying|https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L967],
using the errorIndex which is returned from the error.
retryRows.add(rowsToPublish.get(errorIndex));
However, this errorIndex is not valid to index rowsToPublish. So it looks like
the wrong rows are being selected to be retried.
Why can't you use errorIndex to index rowsToPublish?
rowsToPublish contains all of the rows which were passed into insertAll.
These are then batched into a smaller list of
["rows"|https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L875]
, where multpile API calls are made to bigquery to insert the rows.
The errors returned actually refer to the list of rows passed into the call
made to BigQuery, so they are only valid indices for "rows". They are not valid
indices for "rowsToPublish".
Note: These lists have a different number of rows: rowsToPublish.size() >
rows.size()
> Suspected data loss (and/or duplicates) bug in BigQueyrServicesImpl
> -------------------------------------------------------------------
>
> Key: BEAM-12139
> URL: https://issues.apache.org/jira/browse/BEAM-12139
> Project: Beam
> Issue Type: Test
> Components: io-java-gcp
> Reporter: Alex Amato
> Priority: P2
>
> When this API yields errors specific to failed inserts for a row.
> Rows are selected [here for
> retrying|https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L967],
> using the errorIndex which is returned from the error.
> retryRows.add(rowsToPublish.get(errorIndex));
> However, this errorIndex is not valid to index rowsToPublish. So it looks
> like the wrong rows are being selected to be retried.
> *Why can't you use errorIndex to index rowsToPublish?*
> because rowsToPublish contains all of the rows which were passed into
> insertAll.
> These are then batched into a smaller list of
> ["rows"|https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L875]
> , where multpile API calls are made to bigquery to insert the rows.
> The errors returned actually refer to the list of rows passed into the call
> made to BigQuery, so they are only valid indices for "rows". Thus, they are
> not valid indices for "rowsToPublish".
> Note: These lists have a different number of rows: rowsToPublish.size() >
> rows.size()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
