ahmedabu98 commented on PR #27434:
URL: https://github.com/apache/beam/pull/27434#issuecomment-1629689305
Ahh I think I see what's going on here... there seems to be a race condition
between copy jobs at different triggers. Here, the second trigger's copy jobs
start running before the first trigger's copy jobs. The second trigger's copy
jobs run and succeed in writing data to the table; then the first trigger's
copy jobs run (with `WRITE_EMPTY`) and an error is thrown. As mentioned in the
previous comment, the first trigger is associated with whatever dispositions
are set by the user. All triggers after that are fixed to `WRITE_APPEND` and
`CREATE_NEVER`.
Example below: In the following logs,
`beam_bq_job_COPY_...510c8e70fecb762c7_00000-0` is copy jobs ID of the first
trigger, `beam_bq_job_COPY_...510c8e70fecb762c7_00001-0` is copy jobs ID of the
second trigger. The suffix of these names is
`<tableHash>_<paneNumber>-<retryIndex>`. We see copy jobs at pane 1 starting
earlier than pane 0. Not sure what is causing this to happen.
```
INFO: Job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0
pending. retrying.
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job
beam_bq_job_TEMP_TABLE_LOAD_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_05014b97910b4635a89409bd024d057d_1af13eb133f5d1e510c8e70fecb762c7_00001_00000-0
succeeded. Statistics: null
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_TEMP_TABLE_LOAD_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_05014b97910b4635a89409bd024d057d_1af13eb133f5d1e510c8e70fecb762c7_00001_00000-0
completed successfully.
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0
pending. retrying.
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.WriteRename
lambda$startCopy$57215ec2$1
INFO: Starting copy job for table GenericData{classInfo=[datasetId,
projectId, tableId], {datasetId=dataset-id, projectId=project-id,
tableId=table-id}} using GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0,
projectId=project-id}}, job id iteration 0
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0
started
Jul 10, 2023 4:12:03 PM org.apache.beam.sdk.io.gcp.bigquery.WriteRename
lambda$startCopy$57215ec2$1
INFO: Starting copy job for table GenericData{classInfo=[datasetId,
projectId, tableId], {datasetId=dataset-id, projectId=project-id,
tableId=table-id}} using GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0,
projectId=project-id}}, job id iteration 0
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0
started
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0
succeeded. Statistics: null
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00001-0
completed successfully.
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers getRetryJobId
INFO: job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0
is failed. Checking the next job id
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers getRetryJobId
INFO: job id
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-1
not found, so retrying with that id
Jul 10, 2023 4:12:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
WARNING: Load job
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-0
failed, will retry: {
"errorResult" : { },
"state" : "FAILED"
}. Next job id
beam_bq_job_COPY_bigqueryiowritetest0teststreamingfileloadswithtemptablestoexistingnullschematable010ahmedabualsaud07102012017ee3285b_df4fd51901aa4fb2a9786d4d4eab1b25_1af13eb133f5d1e510c8e70fecb762c7_00000-1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
