pranjal5215 commented on issue #28219: URL: https://github.com/apache/beam/issues/28219#issuecomment-2423624052
Hi @kennknowles @Abacn , I am using Batch load job (moving data from GCS- Cloud Storage to BigQuery) , and Job shows successfully run, but writes only a small portion of data to BigQuery table, Typically 2- 10 rows out of 1071 (expected), While searching more, I stumbled upon above discussion here. From diagnostics it seems to be similar issue , Getting a lot of 409s at bigquery write step (screenshot attached) , Also most BigQuery jobs seems to have same IDs (attached screenshot) Is this the same issue, Happy to share more details if required on this <img width="1678" alt="image" src="https://github.com/user-attachments/assets/97955dba-1dfa-4d98-a934-df8108bd4980";> <img width="1310" alt="image" src="https://github.com/user-attachments/assets/3b52a2bf-08d9-4499-b281-2660888828d3";> Another version of this job with exact same DynamicDestination configuration is running successfully with streaming setup - (Basically that pipeline is moving data from Kafka to BigQuery) Java SDK version: 2.60.0 2.57.0 Runner: GCP Dataflow Tried to run with both below- but job still writes only partial output to BQ 1. Dataflow runner v2 2. Disabling runner v2 JobID looks like: beam_bq_job_LOAD_debugbqload_952f816652304fca8d830ed781542535_3739ddb1bbdb83c1edfec375a931a79a_00001_00000-0 Have given job name like below ``` options.setJobName("debug-bq-load"); ``` ``` WriteResult result = bigQueryRows.apply("WriteToBigQuery", BigQueryIO.<BigQueryRow>write() .to(new MyDynamicDestination(bigqueryDatasetName)) .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED) .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_EMPTY) .withFormatFunction(bigQueryRow -> bigQueryRow != null ? bigQueryRow.getFormattedTableRow() : null) .withMethod(BigQueryIO.Write.Method.FILE_LOADS) ); ``` Also pasting dataflow job describe output here ``` gcloud dataflow jobs describe 2024-10-18_11_41_06-2506487362125660927 --region=asia-south1 --format=json { "createTime": "2024-10-18T18:41:07.476926Z", "currentState": "JOB_STATE_DONE", "currentStateTime": "2024-10-18T18:44:52.815102Z", "environment": { "shuffleMode": "SERVICE_BASED", "userAgent": { "container.base_repository": "gcr.io/cloud-dataflow/v1beta3", "fnapi.container.version": "2.60.0", "fnapi.environment.major.version": "8", "java.vendor": "Oracle Corporation", "java.version": "18.0.1.1", "legacy.container.version": "2.60.0", "legacy.environment.major.version": "8", "name": "Apache Beam SDK for Java", "os.arch": "aarch64", "os.name": "Mac OS X", "os.version": "14.6.1", "version": "2.60.0" }, "version": { "job_type": "JAVA_BATCH_AUTOSCALING", "major": "8" } }, "id": "2024-10-18_11_41_06-2506487362125660927", "jobMetadata": { "sdkVersion": { "sdkSupportStatus": "SUPPORTED", "version": "2.60.0", "versionDisplayName": "Apache Beam SDK for Java" } }, "location": "asia-south1", "name": "gcs_to_bq", "projectId": "livewel-prod", "stageStates": [ { "currentStateTime": "2024-10-18T18:43:34.734Z", "executionStageName": "s74.org.apache.beam.sdk.values.PCollection.<init>:397#bed3177b52ae68e-c181", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:34.922Z", "executionStageName": "s75.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:18.147Z", "executionStageName": "F204", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.589Z", "executionStageName": "s81-close-shuffle148", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:37.382Z", "executionStageName": "s109-close-shuffle51", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.439Z", "executionStageName": "s54-shuffle-session105", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.618Z", "executionStageName": "s32-shuffle-session77", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.770Z", "executionStageName": "s2840-shuffle-session71", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.954Z", "executionStageName": "s109-shuffle-session48", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.719Z", "executionStageName": "s3913-shuffle-session85", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.589Z", "executionStageName": "success211", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.996Z", "executionStageName": "F185", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.037Z", "executionStageName": "s92-open-shuffle164", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:19.508Z", "executionStageName": "F198", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:17.417Z", "executionStageName": "s22.org.apache.beam.sdk.values.PCollection.<init>:397#1d275f544daf228c", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:13.890Z", "executionStageName": "F187", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:17.380Z", "executionStageName": "F205", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.819Z", "executionStageName": "s45", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.204Z", "executionStageName": "F207", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.732Z", "executionStageName": "s504-shuffle-session99", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.097Z", "executionStageName": "s43-close-shuffle94", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.513Z", "executionStageName": "s43-open-shuffle92", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.061Z", "executionStageName": "s92-shuffle-session163", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.734Z", "executionStageName": "s6-close-shuffle116", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.694Z", "executionStageName": "s86-open-shuffle155", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.554Z", "executionStageName": "s43-shuffle-session91", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:34.894Z", "executionStageName": "s75", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:38.225Z", "executionStageName": "s112.org.apache.beam.sdk.values.PCollection.<init>:397#689338c1ae4b4531", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.096Z", "executionStageName": "s86-close-shuffle157", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:17.707Z", "executionStageName": "s6122-shuffle-session122", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:17.670Z", "executionStageName": "s6122-open-shuffle123", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.145Z", "executionStageName": "s113.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.713Z", "executionStageName": "F195", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.764Z", "executionStageName": "s55.org.apache.beam.sdk.values.PCollection.<init>:397#30b23d9769235bb2", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.683Z", "executionStageName": "s97-open-shuffle173", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.755Z", "executionStageName": "s1731-shuffle-session57", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.059Z", "executionStageName": "s81-open-shuffle146", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.874Z", "executionStageName": "s56.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:34.809Z", "executionStageName": "s73-u182", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.734Z", "executionStageName": "s21-open-shuffle64", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.717Z", "executionStageName": "F192", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.086Z", "executionStageName": "s54-close-shuffle108", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.475Z", "executionStageName": "F200", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:18.662Z", "executionStageName": "s65-close-shuffle131", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.697Z", "executionStageName": "s1731-open-shuffle58", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.384Z", "executionStageName": "s3913-close-shuffle88", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.832Z", "executionStageName": "s56", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.524Z", "executionStageName": "F210", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:34.837Z", "executionStageName": "s74.org.apache.beam.sdk.values.PCollection.<init>:397#bed3177b52ae68e", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.015Z", "executionStageName": "F188", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:27.777Z", "executionStageName": "F202", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.561Z", "executionStageName": "F186", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.824Z", "executionStageName": "s101.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.697Z", "executionStageName": "s504-open-shuffle100", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.782Z", "executionStageName": "s97-close-shuffle175", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:21.444Z", "executionStageName": "start213", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.696Z", "executionStageName": "s6-open-shuffle114", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.351Z", "executionStageName": "F209", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.698Z", "executionStageName": "s2840-open-shuffle72", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.514Z", "executionStageName": "s33.org.apache.beam.sdk.values.PCollection.<init>:397#c22b25ea2180c33a", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.119Z", "executionStageName": "s113", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.084Z", "executionStageName": "s81-shuffle-session145", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.750Z", "executionStageName": "s44.org.apache.beam.sdk.values.PCollection.<init>:397#b06373aead4e1949", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:44:03.052Z", "executionStageName": "s90.org.apache.beam.sdk.values.PCollection.<init>:397#eb37685a4ffcf182", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:18.378Z", "executionStageName": "s65-shuffle-session128", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:19.613Z", "executionStageName": "s67", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.721Z", "executionStageName": "s86-shuffle-session154", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.433Z", "executionStageName": "s2840-close-shuffle74", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.281Z", "executionStageName": "s504-close-shuffle102", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:18.590Z", "executionStageName": "F199", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.003Z", "executionStageName": "F193", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.857Z", "executionStageName": "s45.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:27.814Z", "executionStageName": "s74.org.apache.beam.sdk.values.PCollection.<init>:397#bed3177b52ae68e-c180", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.304Z", "executionStageName": "F208", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.741Z", "executionStageName": "s100.org.apache.beam.sdk.values.PCollection.<init>:397#160896c2d158da1f", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.692Z", "executionStageName": "s3913-open-shuffle86", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.564Z", "executionStageName": "s32-open-shuffle78", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:19.778Z", "executionStageName": "s70-open-shuffle137", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.163Z", "executionStageName": "F201", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.161Z", "executionStageName": "F206", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.797Z", "executionStageName": "s101", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:19.813Z", "executionStageName": "s70-shuffle-session136", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:38.199Z", "executionStageName": "F189", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:21.363Z", "executionStageName": "failure212", "executionStageState": "JOB_STATE_PENDING" }, { "currentStateTime": "2024-10-18T18:43:15.245Z", "executionStageName": "s32-close-shuffle80", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:17.533Z", "executionStageName": "s23.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.729Z", "executionStageName": "F190", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.608Z", "executionStageName": "s1731-close-shuffle60", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.003Z", "executionStageName": "F191", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.573Z", "executionStageName": "s92-close-shuffle166", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.526Z", "executionStageName": "s21-close-shuffle66", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:14.396Z", "executionStageName": "s54-open-shuffle106", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.709Z", "executionStageName": "s97-shuffle-session172", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:37.322Z", "executionStageName": "F194", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:17.494Z", "executionStageName": "s23", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:19.543Z", "executionStageName": "s66.org.apache.beam.sdk.values.PCollection.<init>:397#556f4ebf84d9869b", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:27.864Z", "executionStageName": "s70-close-shuffle139", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:35.511Z", "executionStageName": "F196", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.773Z", "executionStageName": "s21-shuffle-session63", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:19.647Z", "executionStageName": "s67.output", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:41:30.743Z", "executionStageName": "s6-shuffle-session113", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.584Z", "executionStageName": "s34", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:36.929Z", "executionStageName": "s109-open-shuffle49", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:18.216Z", "executionStageName": "s6122-close-shuffle125", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:34.708Z", "executionStageName": "F197", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:18.341Z", "executionStageName": "s65-open-shuffle129", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:15.653Z", "executionStageName": "F203", "executionStageState": "JOB_STATE_DONE" }, { "currentStateTime": "2024-10-18T18:43:16.621Z", "executionStageName": "s34.output", "executionStageState": "JOB_STATE_DONE" } ], "startTime": "2024-10-18T18:41:07.476926Z", "type": "JOB_TYPE_BATCH" } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
