See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Direct/808/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Add a multi-process shared utility.
[Robert Bradshaw] Add fastener dependency.
[Robert Bradshaw] Refactor to have an explicit acquire/release API.
[Robert Bradshaw] Drop a TODO about deferred construction parameterization.
[Robert Bradshaw] Fix unused import/var.
[Moritz Mack] Replace website references to deprecated aws / kinesis modules
with more
[noreply] Add database role to SpannerConfig for role-based access control.
[noreply] Remove obsolete and deprecated bigquery native read. (#23557)
------------------------------------------
[...truncated 7.77 MB...]
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/afc7c415-dca3-4750-914f-4acc99f5c057.
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/2bb2a235-0aa9-4f9d-a83f-496d3f9596ff.
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/76d5041f-38e9-4914-84da-354a2932c072.
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/be03aae7-1264-4413-8fa7-80d8522cd61b.
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/6ea9ae80-d17c-4cfc-af68-36bba301c889.
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/903561c0-fefc-45db-b854-c9e85a6bc7d9.
Oct 11, 2022 9:52:09 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21/3c80706b-b1af-448d-bf83-7a4e24965918.
Oct 11, 2022 9:53:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:53:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:53:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:53:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
pending. retrying.
Oct 11, 2022 9:53:28 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables
lambda$startLoad$79287d2b$1
INFO: Loading 8045 files into GenericData{classInfo=[datasetId, projectId,
tableId], {datasetId=combine_per_key_examples1665524845534,
projectId=apache-beam-testing, tableId=combine_per_key_examples_table}} using
job GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
location=US, projectId=apache-beam-testing}}, job id iteration 0
Oct 11, 2022 9:53:29 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: GenericData{classInfo=[jobId, location,
projectId],
{jobId=beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
location=US, projectId=apache-beam-testing}}.
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:29 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
started
Oct 11, 2022 9:53:29 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:31 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:32 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:35 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:39 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:45 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:53:53 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:54:03 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:54:29 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:54:50 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:55:14 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
Oct 11, 2022 9:56:15 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0,
location=US, projectId=apache-beam-testing}} completed in state DONE
Oct 11, 2022 9:56:15 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
succeeded. Statistics:
{"completionRatio":1.0,"creationTime":"1665525208967","endTime":"1665525356275","load":{"badRecords":"0","inputFileBytes":"586637","inputFiles":"8045","outputBytes":"401602","outputRows":"8045","timeline":[{"elapsedMs":"143303","totalSlotMs":"300686","pendingUnits":"4373","completedUnits":"3672","activeUnits":"1000","estimatedRunnableUnits":"0"},{"elapsedMs":"143805","totalSlotMs":"697795","pendingUnits":"4","completedUnits":"8041","activeUnits":"1000","estimatedRunnableUnits":"0"},{"elapsedMs":"145302","totalSlotMs":"815794","pendingUnits":"1","completedUnits":"8545","activeUnits":"1","estimatedRunnableUnits":"0"},{"elapsedMs":"146646","totalSlotMs":"816060","pendingUnits":"0","completedUnits":"8546","activeUnits":"0","estimatedRunnableUnits":"0"}]},"reservationUsage":[{"name":"default-pipeline","slotMs":"816060"}],"reservation_id":"default-pipeline","startTime":"1665525209407","totalSlotMs":"816060","finalExecutionDurationMs":"4276"}
Oct 11, 2022 9:56:15 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_LOAD_testpipelinejenkins1011214728484adc9b_ea75a3dc044644e490214d9a10872f21_6d6a0e577d8e07b2240f27b5b6dfcb46_00001_00000-0
completed successfully.
Oct 11, 2022 9:56:22 PM org.apache.beam.sdk.io.gcp.testing.BigqueryClient
deleteTable
INFO: Successfully deleted table: combine_per_key_examples_table
Oct 11, 2022 9:56:22 PM org.apache.beam.sdk.io.gcp.testing.BigqueryClient
deleteDataset
INFO: Successfully deleted dataset: combine_per_key_examples1665524845534
org.apache.beam.examples.cookbook.MaxPerKeyExamplesIT >
testE2EMaxPerKeyExamples STANDARD_ERROR
Oct 11, 2022 9:56:23 PM org.apache.beam.sdk.io.gcp.testing.BigqueryClient
createNewDataset
INFO: Successfully created new dataset :
apache-beam-testing:max_per_key_examples1665525382551
Oct 11, 2022 9:56:24 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
INFO: Starting BigQuery extract job:
beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3
Oct 11, 2022 9:56:24 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: GenericData{classInfo=[jobId, location,
projectId],
{jobId=beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3,
location=US, projectId=apache-beam-testing}}.
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3
Oct 11, 2022 9:56:24 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3
Oct 11, 2022 9:56:25 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3,
location=US, projectId=apache-beam-testing}} completed in state DONE
Oct 11, 2022 9:56:25 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
INFO: BigQuery extract job completed:
beam_bq_job_EXPORT_testpipelinejenkins1011215624f2a6d72c_4d0033a7f35f4774b52b0757e41cc9e3
Oct 11, 2022 9:56:25 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase split
INFO: Extract job produced 1 files
Oct 11, 2022 9:56:25 PM org.apache.beam.sdk.io.FileBasedSource createReader
INFO: Matched 1 files for pattern
gs://temp-storage-for-end-to-end-tests/BigQueryExtractTemp/4d0033a7f35f4774b52b0757e41cc9e3/000000000000.avro
Oct 11, 2022 9:56:25 PM org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4
getTempFilePrefix
INFO: Writing BigQuery temporary files to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/
before loading them.
Oct 11, 2022 9:56:25 PM org.apache.beam.sdk.io.FileBasedSource
getEstimatedSizeBytes
INFO: Filepattern
gs://temp-storage-for-end-to-end-tests/BigQueryExtractTemp/4d0033a7f35f4774b52b0757e41cc9e3/000000000000.avro
matched 1 files with total size 119755
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/dec834db-7f47-4a3d-8a7e-c4376c46448c.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/7d020833-91ff-4a46-bb1c-bf06e5b8fff1.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/1207640c-f932-46e5-9754-dabbf61ecf6d.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/332ca1b3-411a-405c-b5b5-4e44fb347600.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/3548a168-d134-45e6-b9c5-12cbd89e0d36.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/dddb652a-e43a-491d-8b2a-e2649693b3c4.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/d1cc0206-b1b1-47a9-8b18-558e6c229f01.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/7134f472-7060-4de4-9163-256d0590f494.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/a45162ef-d2ba-43ee-ac36-b46c4d487259.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/a77f9473-2ff9-42b2-8b98-2e9f8def51a8.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/45b0a365-310e-4118-af7d-9006184f1610.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad/fb426ed5-00c9-446d-b0e3-5a4bb43e1c00.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:56:27 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
pending. retrying.
Oct 11, 2022 9:56:28 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables
lambda$startLoad$79287d2b$1
INFO: Loading 12 files into GenericData{classInfo=[datasetId, projectId,
tableId], {datasetId=max_per_key_examples1665525382551,
projectId=apache-beam-testing, tableId=max_per_key_examples_table}} using job
GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0,
location=US, projectId=apache-beam-testing}}, job id iteration 0
Oct 11, 2022 9:56:28 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: GenericData{classInfo=[jobId, location,
projectId],
{jobId=beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0,
location=US, projectId=apache-beam-testing}}.
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
Oct 11, 2022 9:56:28 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
started
Oct 11, 2022 9:56:28 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
Oct 11, 2022 9:56:30 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
Oct 11, 2022 9:56:32 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0,
location=US, projectId=apache-beam-testing}} completed in state DONE
Oct 11, 2022 9:56:32 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
succeeded. Statistics:
{"completionRatio":1.0,"creationTime":"1665525388621","endTime":"1665525390976","load":{"badRecords":"0","inputFileBytes":"522","inputFiles":"12","outputBytes":"192","outputRows":"12","timeline":[{"elapsedMs":"1196","totalSlotMs":"364","pendingUnits":"1","completedUnits":"0","activeUnits":"1","estimatedRunnableUnits":"0"},{"elapsedMs":"2161","totalSlotMs":"758","pendingUnits":"0","completedUnits":"1","activeUnits":"1","estimatedRunnableUnits":"0"}]},"reservationUsage":[{"name":"default-pipeline","slotMs":"758"}],"reservation_id":"default-pipeline","startTime":"1665525388785","totalSlotMs":"758","finalExecutionDurationMs":"1202"}
Oct 11, 2022 9:56:32 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_LOAD_testpipelinejenkins1011215624f2a6d72c_de3d1ead45e148db8f90d5c304386dad_65b97d0cef6a223ec357a02cb661a2be_00001_00000-0
completed successfully.
org.apache.beam.examples.snippets.transforms.io.gcp.bigquery.BigQuerySamplesIT
> testTableIO STANDARD_ERROR
Oct 11, 2022 9:56:33 PM org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4
getTempFilePrefix
INFO: Writing BigQuery temporary files to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec/
before loading them.
Oct 11, 2022 9:56:33 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter <init>
INFO: Opening TableRowWriter to
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec/b205995f-b1a6-4149-bd46-adef08758d53.
Oct 11, 2022 9:56:33 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:56:33 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:56:33 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Waiting for jobs to complete.
Oct 11, 2022 9:56:33 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
pending. retrying.
org.apache.beam.examples.cookbook.MaxPerKeyExamplesIT >
testE2EMaxPerKeyExamples STANDARD_ERROR
Oct 11, 2022 9:56:33 PM org.apache.beam.sdk.io.gcp.testing.BigqueryClient
deleteTable
INFO: Successfully deleted table: max_per_key_examples_table
Oct 11, 2022 9:56:33 PM org.apache.beam.sdk.io.gcp.testing.BigqueryClient
deleteDataset
INFO: Successfully deleted dataset: max_per_key_examples1665525382551
org.apache.beam.examples.snippets.transforms.io.gcp.bigquery.BigQuerySamplesIT
> testTableIO STANDARD_ERROR
Oct 11, 2022 9:56:34 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables
lambda$startLoad$79287d2b$1
INFO: Loading 1 files into GenericData{classInfo=[datasetId, projectId,
tableId], {datasetId=beam_bigquery_samples_1665525392190_27,
projectId=apache-beam-testing, tableId=testTableIO}} using job
GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0,
location=US, projectId=apache-beam-testing}}, job id iteration 0
Oct 11, 2022 9:56:35 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: GenericData{classInfo=[jobId, location,
projectId],
{jobId=beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0,
location=US, projectId=apache-beam-testing}}.
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
Oct 11, 2022 9:56:35 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob runJob
INFO: job
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
started
Oct 11, 2022 9:56:35 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
Oct 11, 2022 9:56:36 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
Oct 11, 2022 9:56:37 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0,
location=US, projectId=apache-beam-testing}} completed in state DONE
Oct 11, 2022 9:56:37 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob pollJob
INFO: Load job
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
succeeded. Statistics:
{"completionRatio":1.0,"creationTime":"1665525395136","endTime":"1665525397378","load":{"badRecords":"0","inputFileBytes":"484","inputFiles":"1","outputBytes":"239","outputRows":"1","timeline":[{"elapsedMs":"1266","totalSlotMs":"264","pendingUnits":"0","completedUnits":"1","activeUnits":"0"},{"elapsedMs":"2056","totalSlotMs":"264","pendingUnits":"0","completedUnits":"1","activeUnits":"0","estimatedRunnableUnits":"0"}]},"reservationUsage":[{"name":"default-pipeline","slotMs":"264"}],"reservation_id":"default-pipeline","startTime":"1665525395289","totalSlotMs":"264","finalExecutionDurationMs":"1153"}
Oct 11, 2022 9:56:37 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager
waitForDone
INFO: Job
beam_bq_job_LOAD_bigquerysamplesit0testtableiojenkins1011215633126c02e4_3c01f42cf07a49d197cc1467258be9ec_df4a7692aeec789821fb9c630ae0d3a1_00001_00000-0
completed successfully.
Oct 11, 2022 9:56:39 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
INFO: Starting BigQuery extract job:
beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6
Oct 11, 2022 9:56:39 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: GenericData{classInfo=[jobId, location,
projectId],
{jobId=beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6,
location=US, projectId=apache-beam-testing}}.
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6
Oct 11, 2022 9:56:39 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job
beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6,
currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing
beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6
Oct 11, 2022 9:56:40 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6,
location=US, projectId=apache-beam-testing}} completed in state DONE
Oct 11, 2022 9:56:40 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
INFO: BigQuery extract job completed:
beam_bq_job_EXPORT_bigquerysamplesit0testtableiojenkins1011215639e549202a_2b53daf04ce445f88d8692978579f2c6
Oct 11, 2022 9:56:40 PM
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase split
INFO: Extract job produced 1 files
Oct 11, 2022 9:56:40 PM org.apache.beam.sdk.io.FileBasedSource createReader
INFO: Matched 1 files for pattern
gs://temp-storage-for-end-to-end-tests/BigQueryExtractTemp/2b53daf04ce445f88d8692978579f2c6/000000000000.avro
org.apache.beam.examples.snippets.transforms.io.gcp.bigquery.BigQuerySamplesIT
> testTableIO FAILED
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.VerifyException
at Verify.java:124
***********************************************************
***********************************************************
**********************Set Up Pubsub************************
The Pub/Sub topic has been set up for this example:
projects/apache-beam-testing/topics/testpipeline-jenkins-1011214537-cf70d877
The Pub/Sub subscription has been set up for this example:
projects/apache-beam-testing/subscriptions/testpipeline-jenkins-1011214537-cf70d877
*************************Tear Down*************************
The Pub/Sub topic has been deleted:
projects/apache-beam-testing/topics/testpipeline-jenkins-1011214537-cf70d877
The Pub/Sub subscription has been deleted:
projects/apache-beam-testing/subscriptions/testpipeline-jenkins-1011214537-cf70d877
***********************************************************
***********************************************************
17 tests completed, 1 failed
> Task :runners:direct-java:examplesIntegrationTest FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runners:direct-java:examplesIntegrationTest'.
> There were failing tests. See the report at:
> file://<https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Direct/ws/src/runners/direct-java/build/reports/tests/examplesIntegrationTest/index.html>
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 12m 12s
89 actionable tasks: 54 executed, 33 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/zud5lcrdubsvs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
