[
https://issues.apache.org/jira/browse/BEAM-10248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Graham Polley updated BEAM-10248:
---------------------------------
Attachment: Untitled document.pdf
> Beam does not set correct region for BigQuery when requesting load job status
> -----------------------------------------------------------------------------
>
> Key: BEAM-10248
> URL: https://issues.apache.org/jira/browse/BEAM-10248
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.22.0
> Environment: -Beam 2.22.0
> -DirectRunner & DataflowRunner
> -Java 11
> Reporter: Graham Polley
> Priority: P1
> Attachments: Untitled document.pdf
>
>
> I am using `FILE_LOADS` (micro-batching) from Pub/Sub and writing to
> BigQuery. My BigQuery dataset is in region `australia-southeast1`.
> If the load job into BigQuery fails for some reason (e.g. table does not
> exist, or schema has changed), then an error from BigQuery is returned for
> the load.
> Beam then enters into a loop of retrying the load job. However, instead of
> using a new job id (where the suffix is incremented by 1), it wrongly tries
> to reinsert the job using the same job id. This is because when it tries to
> look up if the job id already exists, it does not take into consideration the
> region where the dataset is. Instead, it defaults to the US but the job was
> created in `australia-southeast1`, so it returns `null`.
> Bug is on LN308 of `BigQueryHelpers.java`. It should not return `null`. It
> returns `null` because it's looking in the wrong region.
> If I *test* using a dataset in BigQuery that is in the US region, then it
> correctly finds the job id and begins to retry the job with a new job id
> (suffixed with the retry count). We have seen this bug in other areas of Beam
> before and in other tools/services on GCP e.g. Cloud Composer.
> However, that leads me to my next problem/bug. Even if that is fixed, the
> number of retries is set to `Integer.MX_VALUE`, so Beam will keep retrying
> the job 2,147,483,647 times. This is not good.
> The exception is swallowed up by the Beam SDK and never propagated back up
> the stack for users to catch and handle. So, if a load job fails there is no
> way to handle it and react for users.
> I attempted to set `withFailedInsertRetryPolicy()` to transient errors only,
> but it is not supported with `FILE_LOADS`. I also tried, using the
> `WriteResult` object returned from the bigQuery sink/write, to get a handle
> on the error but it does not work. Users need a way to catch and catch failed
> load jobs when using `FILE_LOADS`.
>
> {code:java}
> public class TemplatePipeline {
> private static final String TOPIC =
> "projects/etl-demo-269105/topics/test-micro-batching";
> private static final String BIGQUERY_DESTINATION_FILTERED =
> "etl-demo-269105:etl_demo_fun.micro_batch_test_xxx";
> public static void main(String[] args) throws Exception {
> try {
> PipelineOptionsFactory.register(DataflowPipelineOptions.class);
> DataflowPipelineOptions options = PipelineOptionsFactory
> .fromArgs(args)
> .withoutStrictParsing()
> .as(DataflowPipelineOptions.class);
> Pipeline pipeline = Pipeline.create(options);
> PCollection<PubsubMessage> messages = pipeline
> .apply(PubsubIO.readMessagesWithAttributes().fromTopic(String.format(TOPIC,
> options.getProject())))
> .apply(Window.into(FixedWindows.of(Duration.standardSeconds(1))));
> WriteResult result = messages.apply(ParDo.of(new RowToBQRow()))
> .apply(BigQueryIO.writeTableRows()
> .to(String.format(BIGQUERY_DESTINATION_FILTERED, options.getProject()))
> .withCreateDisposition(CREATE_NEVER)
> .withWriteDisposition(WRITE_APPEND)
> .withMethod(BigQueryIO.Write.Method.FILE_LOADS)
> .withTriggeringFrequency(Duration.standardSeconds(5))
> .withNumFileShards(1)
> .withExtendedErrorInfo()
> .withSchema(getTableSchema()));
> // result.getFailedInsertsWithErr().apply(ParDo.of(new
> DoFn<BigQueryInsertError, String>() {
> // @ProcessElement
> // public void processElement(ProcessContext c) {
> // for(ErrorProto err : c.element().getError().getErrors()){
> // throw new RuntimeException(err.getMessage());
> // }
> //
> // }
> // }));
> result.getFailedInserts().apply(ParDo.of(new DoFn<TableRow, String>() {
> @ProcessElement
> public void processElement(ProcessContext c) {
> System.out.println(c.element());
> c.output("foo");
> throw new RuntimeException("Failed to load");
> }
> @FinishBundle
> public void finishUp(FinishBundleContext finishBundleContextc){
> System.out.println("Got here");
> }
> }));
> pipeline.run();
> } catch (Exception e) {
> e.printStackTrace();
> throw new Exception(e);
> }
> }
> private static TableSchema getTableSchema() {
> List<TableFieldSchema> fields = new ArrayList<>();
> fields.add(new TableFieldSchema().setName("timestamp").setType("INTEGER"));
> fields.add(new TableFieldSchema().setName("payload").setType("STRING"));
> return new TableSchema().setFields(fields);
> }
> public static class RowToBQRow extends DoFn<PubsubMessage, TableRow> {
> @ProcessElement
> public void processElement(ProcessContext c) {
> String payload = new String(c.element().getPayload(),
> StandardCharsets.UTF_8);
> c.output(new TableRow()
> .set("timestamp", System.currentTimeMillis())
> .set("payload", payload)
> );
> }
> }
> }{code}
> Stack trace showing problem of same job id being used/inserted to BigQuery on
> each retry:
> {noformat}
> /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java
> -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:50370,suspend=y,server=n
> -ea
> -javaagent:/Users/grahampolley/Library/Caches/IdeaIC2018.2/groovyHotSwap/gragent.jar
>
> -javaagent:/Users/grahampolley/Library/Caches/IdeaIC2018.2/captureAgent/debugger-agent.jar=file:/private/var/folders/_t/1lxhy17n2mlc0_4rb3_3thq40001g7/T/capture.props
> -Dfile.encoding=UTF-8 -classpath
> "/Users/grahampolley/Dev/gcp-batch-ingestion-bigquery/out/production/classes:/Users/grahampolley/Dev/gcp-batch-ingestion-bigquery/out/production/resources:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.30/c21f55139d8141d2231214fb1feaf50a1edca95e/slf4j-log4j12-1.7.30.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-google-cloud-dataflow-java/2.22.0/f251d2508e2c55a6c44fdf4c2c66ce4407b84908/beam-runners-google-cloud-dataflow-java-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-direct-java/2.22.0/541d048b57a0bb189310ebc85c6266241a732952/beam-runners-direct-java-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-io-google-cloud-platform/2.22.0/cdd9b337c909cd5ed85e379b7ae1c1a26dfba55f/beam-sdks-java-io-google-cloud-platform-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-google-cloud-platform-core/2.22.0/c3509a7b95a3533775cbaa61e438a482e2cdd5bf/beam-sdks-java-extensions-google-cloud-platform-core-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-expansion-service/2.22.0/6180ba45374960b0b639df0ae47c3b3736045612/beam-sdks-java-expansion-service-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-java-fn-execution/2.22.0/cdeafa6cb9365caf63da4232e326b2eb2ddcde33/beam-runners-java-fn-execution-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-core-construction-java/2.22.0/94c4145c597f700a4f3807282f916427dab5ef70/beam-runners-core-construction-java-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-protobuf/2.22.0/580a4bf3424738b30f840e29d4e5946df7017a84/beam-sdks-java-extensions-protobuf-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-fn-execution/2.22.0/5da4c54468112faf1f8b7f8bbf5524fe49d6c8ef/beam-sdks-java-fn-execution-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-sdks-java-extensions-protobuf/2.22.0/ec7722d88ddceb57806a76f7285bd8c0a1cc7fcd/beam-vendor-sdks-java-extensions-protobuf-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-core/2.22.0/73d5b6a07c6ae1579506c6224895efbd53d69f3f/beam-sdks-java-core-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-model-pipeline/2.22.0/3b5b6715b2f9b6e7e35b51acd904e30bde77d10e/beam-model-pipeline-2.22.0.jar:/Users/grahampolley/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.bigtable/bigtable-client-core/1.13.0/a86858be799ad73f1139017f91c4777147552fd4/bigtable-client-core-1.13.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-model-job-management/2.22.0/c155afa58e1cc382e7de62b9a67e0b918f2f5c1c/beam-model-job-management-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-model-fn-execution/2.22.0/8f180cc8d7ed9b0bfbbede2901ce9c62f10b50fd/beam-model-fn-execution-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.dropwizard.metrics/metrics-core/3.2.6/62fe170cffeded1cef60e9e3402a93b45ce14327/metrics-core-3.2.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.30/b5a4b6d16ab13e34a88fae84c35cd5d68cac922c/slf4j-api-1.7.30.jar:/Users/grahampolley/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-guava-26_0-jre/0.1/f309c3047ca99428e567afa42d233fb3e839bde1/beam-vendor-guava-26_0-jre-0.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.bigdataoss/gcsio/2.1.3/6dfefdc343ae4e5d1ff6df480f0457c073bfc19c/gcsio-2.1.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.bigdataoss/util/2.1.3/3a0d8e8883268fd1d8ec4241084fdb290bf7782b/util-2.1.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-clouddebugger/v2-rev20200313-1.30.9/f08af56f1e6cf8673d9408b99874eda40e6960fe/google-api-services-clouddebugger-v2-rev20200313-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-dataflow/v1b3-rev20200305-1.30.9/66c26eb008746cc8003c9dad799b2420cfdd75ed/google-api-services-dataflow-v1b3-rev20200305-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-bigquerystorage/0.125.0-beta/4ade9a93e57bdf032d8325e3fc0bd3cad75f153b/google-cloud-bigquerystorage-0.125.0-beta.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-bigquery/1.106.0/bc7232727654ba78edd63f8c95dd0ea47c4950f8/google-cloud-bigquery-1.106.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-bigquery/v2-rev20191211-1.30.9/4c8b0534651f0000d3320307cea24e145c237a5e/google-api-services-bigquery-v2-rev20191211-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-healthcare/v1beta1-rev20200307-1.30.9/89a426386623daf7528b0869a710673c3bf7c156/google-api-services-healthcare-v1beta1-rev20200307-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-pubsub/v1-rev20200312-1.30.9/aad0ede541289a8ca1a280facd0fe159fa9cebd4/google-api-services-pubsub-v1-rev20200312-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.datastore/datastore-v1-proto-client/1.6.3/b0ffaeb68af9cb99772aa9933aa9baa0521568dc/datastore-v1-proto-client-1.6.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-cloudresourcemanager/v1-rev20200311-1.30.9/6c675cf826d657a08e2fece69317de4319d3192f/google-api-services-cloudresourcemanager-v1-rev20200311-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-storage/v1-rev20200326-1.30.9/5c0856380fff4fe8d11f5df03c653c16f69388e1/google-api-services-storage-v1-rev20200326-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client-java6/1.30.9/cef0e59174a3d6066598046209e44c9c34bb3132/google-api-client-java6-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client-jackson2/1.30.9/b6af6877119b2932ca293b14b33aa5f7393a358c/google-api-client-jackson2-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-core-http/1.92.4/d2776edaa94a4eee16332966adb575d8c5e63210/google-cloud-core-http-1.92.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client/1.30.9/8fc8641740ee0c26715738fb4d902779b96217d6/google-api-client-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-spanner/1.49.1/19dfd15e889a41c306b5dd62a622db6c471cb6fc/google-cloud-spanner-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-bigtable/1.9.1/8df06e49c314f524e6fdc11878ad5647f2a992b6/google-cloud-bigtable-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-core-grpc/1.92.2/af95d438cc2f05cdbc0c31101ca2fd17da753ee9/google-cloud-core-grpc-1.92.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/gax-grpc/1.54.0/93d75eab29a7923f407efdae59a87e032b86d9f7/gax-grpc-1.54.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-core/1.92.5/97ca5c35f34ee67dde8f2d4e9b940f0a96393c66/google-cloud-core-1.92.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/gax-httpjson/0.70.1/15b2c7844b9561ea036e009c153cf1ef70701afb/gax-httpjson-0.70.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/gax/1.54.0/1f1668868b8b3fd5fc248d80c16dd9f09afc9180/gax-1.54.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-alts/1.29.0/aa559eb5550bb772e42665c271cfe4dfcf22116f/grpc-alts-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.auth/google-auth-library-oauth2-http/0.20.0/f33d4d6c91a68826816606a2208990eea93fcb2a/google-auth-library-oauth2-http-0.20.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-all/1.27.2/fd3c66752c77b7f2c1e345f852862af92dd36a83/grpc-all-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-auth/1.29.0/aa89399565038ca930510b8428588cc919086eba/grpc-auth-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.auth/google-auth-library-credentials/0.20.0/87a91a373e64ba5c3cdf8cc5cf54b189dd1492f8/google-auth-library-credentials-0.20.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.oauth-client/google-oauth-client-java6/1.30.6/6c744e05b99325030625d9c7df7f3306f3e80a4e/google-oauth-client-java6-1.30.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.oauth-client/google-oauth-client/1.30.6/5c9a0e62e0cbda2bad495fa72147a331617f2e99/google-oauth-client-1.30.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-jackson2/1.34.2/fc5cd5e7b6252bae6a65ef2a1091476afb25b7c7/google-http-client-jackson2-1.34.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-protobuf/1.33.0/aa38d441440dc8ed72509b94580403c58d910caa/google-http-client-protobuf-1.33.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-appengine/1.34.1/d5d538385a2fa2b23d1831c35af66ac2e5445a0f/google-http-client-appengine-1.34.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client/1.34.2/58b4c500b5ce58271f1a6c5f6767aa8721158ea4/google-http-client-1.34.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.10.2/528de95f198afafbcfb0c09d2e43b6e0ea663ec/jackson-databind-2.10.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.10.2/3a13b6105946541b8d4181a0506355b5fae63260/jackson-annotations-2.10.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.10.2/73d4322a6bda684f676a2b5fe918361c4e5c7cca/jackson-core-2.10.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.10.5/7f1d89817cd20a32444d5ab4160f035ab9b864e7/joda-time-2.10.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-grpc-1_26_0/0.3/6871e7d0b92cd4983064166ee44d633e3800ef0f/beam-vendor-grpc-1_26_0-0.3.jar:/Users/grahampolley/.m2/repository/args4j/args4j/2.33/args4j-2.33.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-netty/1.27.2/dd4760eafb989dd1997c8f918eb376717b36e4aa/grpc-netty-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.opencensus/opencensus-contrib-grpc-util/0.24.0/1ac2f0c1a02c7ec0cd244e59228f687a90dfddab/opencensus-contrib-grpc-util-0.24.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-okhttp/1.27.2/5081585398fe003550717016928d18ec849f44f4/grpc-okhttp-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-testing/1.27.2/f0d7e35e5d0cb23a99afc26f2372e30b5e97cc01/grpc-testing-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-netty-shaded/1.29.0/80078c0aec3f2146ce94b5c0bddd722fa1f8e502/grpc-netty-shaded-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-grpclb/1.29.0/98635bf61b76cfdc3b38442290d4939120f9f51c/grpc-grpclb-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-core/1.29.0/b051a14a67c97bb9bbe0b9a03b5d7e7080e7b960/grpc-core-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.opencensus/opencensus-contrib-http-util/0.24.0/6d96406c272d884038eb63b262458df75b5445/opencensus-contrib-http-util-0.24.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.opencensus/opencensus-api/0.24.0/f974451b19007ce820f433311ce8adb88e2b7d2c/opencensus-api-0.24.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-cloud-pubsub-v1/1.85.1/5c59ba127376fc68b7c7f38f9b837e280b32b6dd/grpc-google-cloud-pubsub-v1-1.85.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-cloud-bigtable-v2/1.9.1/9edbe1087db616c9023b56f5581551002c95caad/grpc-google-cloud-bigtable-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-cloud-bigtable-admin-v2/1.9.1/974022119ccf41a7443a4a092c77559966a18b4d/grpc-google-cloud-bigtable-admin-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-common-protos/1.17.0/9c5b4135ff50dbc0d299ba8288c5f8e829470994/grpc-google-common-protos-1.17.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-protobuf/1.29.0/57a50ff72c0da333c31ad45616ea17f7d3b91b1c/grpc-protobuf-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-stub/1.29.0/2ded7df15689f2e606189456a91f7a9e1da9feee/grpc-stub-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-protobuf-lite/1.29.0/22e5ff00b942ee573c9dbadfc5e690b151a6d5e4/grpc-protobuf-lite-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-api/1.29.0/4f067a7b1657ad95c00fe958e8a66c8f8446c9f/grpc-api-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-context/1.29.0/1d8a441110f86f8927543dc3007639080441ea3c/grpc-context-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigtable-v2/1.9.1/a02ccc9174a00d97d9d1bad53a1bb9e456de8714/proto-google-cloud-bigtable-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-pubsub-v1/1.85.1/cddb71da7461d091effaaf51ed773fbc766263d6/proto-google-cloud-pubsub-v1-1.85.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-spanner-admin-database-v1/1.49.1/b1c18dc081a1858060577767af00dec47f0ddeeb/proto-google-cloud-spanner-admin-database-v1-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-datastore-v1/0.85.0/30d57cf3b3b0ae569975fcbf2dff36b958afb2c0/proto-google-cloud-datastore-v1-0.85.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigtable-admin-v2/1.9.1/fe8584d617c60d1f8cf203c33bd0358f4423e257/proto-google-cloud-bigtable-admin-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-spanner-admin-instance-v1/1.49.1/d0e502423bfed8985952dcb2f909cb8417b6923e/proto-google-cloud-spanner-admin-instance-v1-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-iam-v1/0.13.0/ed3d62b64aa23a3decf324c8988eb4aae9f36e94/proto-google-iam-v1-0.13.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-spanner-v1/1.49.1/d9d02615204a606caf87c91f67c0ff9c31e1ba52/proto-google-cloud-spanner-v1-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/api-common/1.8.1/e89befb19b08ad84b262b2f226ab79aefcaa9d7f/api-common-1.8.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java-util/3.11.4/99a6a669e55f3d587ac8eb61857f7b81d0bbd7f7/protobuf-java-util-3.11.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/29.0-jre/801142b4c3d0f0770dd29abea50906cacfddd447/guava-29.0-jre.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.42.Final/819e7b5f2005770cf7558c04276fff080331c6df/netty-codec-http2-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.42.Final/7b816d9f37ddcb68f6c1b9b0d7b5a98bfac40911/netty-handler-proxy-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.42.Final/5f71267aa784d0e6c5ec09fb988339d244b205a0/netty-codec-http-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.42.Final/fc6546be5df552d9729f008d8d41a6dee28127aa/netty-handler-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.17.Final/b1e5acbde8c444c656131238ac6ab9e73f694300/netty-tcnative-boringssl-static-2.0.17.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1beta1/0.90.0/29e65631a68262e36cc1868768029767bfd8bd72/proto-google-cloud-bigquerystorage-v1beta1-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1alpha2/0.90.0/2a1296e1b7d2dfe668813dbda443a3bcd477b35e/proto-google-cloud-bigquerystorage-v1alpha2-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1beta2/0.90.0/c59fa6741ab5d84a2de4c11f07f41510751e52c4/proto-google-cloud-bigquerystorage-v1beta2-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1/0.90.0/6c6431f01d5ed66f1c7cf4c5f95deda4f73876c4/proto-google-cloud-bigquerystorage-v1-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-common-protos/1.18.0/b69bc5942255dc9dd2d2de432c4b99a5b483679/proto-google-common-protos-1.18.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/3.11.4/7ec0925cc3aef0335bbc7d57edfd42b0f86f8267/protobuf-java-3.11.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-bytebuddy-1_10_8/0.1/16661d7186f2d10440f1bb1c1e27e51f5306abe8/beam-vendor-bytebuddy-1_10_8-0.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.perfmark/perfmark-api/0.19.0/2bfc352777fa6e27ad1e11d11ea55651ba93236b/perfmark-api-0.19.0.jar:/Users/grahampolley/.m2/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/Users/grahampolley/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.tukaani/xz/1.8/c4f7d054303948eb6a4066194253886c8af07128/xz-1.8.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.3.4/dac170e4594de319655ffb62f41cbd6dbb5e601e/error_prone_annotations-2.3.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.11/f6d42fee5110c227bac18a550a297e028f2fb21a/httpclient-4.5.11.jar:/Users/grahampolley/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.6.2/bd1b74a5d170686362091c7cf596bbc3adf5c09b/log4j-api-2.6.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.conscrypt/conscrypt-openjdk-uber/2.2.1/59a346d64c0ddca750c5c877e274d5e6278e53ce/conscrypt-openjdk-uber-2.2.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.65/be28c46df75712f1fece48639c05633c8217f71/classgraph-4.8.65.jar:/Users/grahampolley/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/grahampolley/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/grahampolley/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.20/b8df472b31e1f17c232d2ad78ceb1c84e00c641b/commons-compress-1.20.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.auto.value/auto-value-annotations/1.7.1/58d76a9ec581f7c6d33f3343de9b2ba04a0ae799/auto-value-annotations-1.7.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.flogger/google-extensions/0.5.1/4f1d862216754651fc1e1f2b614746810e68a4ff/google-extensions-0.5.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.13/853b96d3afbb7bf8cc303fe27ee96836a10c1834/httpcore-4.4.13.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.3/ba035118bc8bac37d7eff77700720999acd9986d/j2objc-annotations-1.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.flogger/flogger-system-backend/0.5.1/b66d3bedb14da604828a8693bb24fd78e36b0e9e/flogger-system-backend-0.5.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.threeten/threetenbp/1.4.1/8e0de17c5077523503812837fa069459103fb714/threetenbp-1.4.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.13/3f18e1aa31031d89db6f01ba05d501258ce69d2c/commons-codec-1.13.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-lang3/3.5/6c6c702c89bfff3cd9e80b04d668c5e190d588c6/commons-lang3-3.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.code.gson/gson/2.8.6/9180733b7df8542621dc12e21e87557e8c99b8cb/gson-2.8.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.android/annotations/4.1.1.4/a1678ba907bf92691d879fef34e1a187038f9259/annotations-4.1.1.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.42.Final/dd355f01dc00f93aaebe805b05026d7cf57c60ec/netty-codec-socks-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.42.Final/b1d5ed85a558fbbadc2783f869fbd0adcd32b07b/netty-codec-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.42.Final/857502e863c02c829fdafea61c3fda6bda01d0af/netty-transport-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.42.Final/6e6fc9178d1f1401aa0d6b843341efb91720f2cd/netty-buffer-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/javax.annotation/javax.annotation-api/1.3.2/934c04d3cfef185a8008e7bf34331b79730a9d43/javax.annotation-api-1.3.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.guava/failureaccess/1.0.1/1dcf1de382a0bf95a3d8b0849546c88bac1292c9/failureaccess-1.0.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/b421526c5f297295adef1c886e5246c39d4ac629/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.checkerframework/checker-qual/2.11.1/8c43bf8f99b841d23aadda6044329dad9b63c185/checker-qual-2.11.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.flogger/flogger/0.5.1/71d1e2cef9cc604800825583df56b8ef5c053f14/flogger-0.5.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.checkerframework/checker-compat-qual/2.5.5/435dc33e3019c9f019e15f01aa111de9d6b2b79c/checker-compat-qual-2.5.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.18/f7aa683ea79dc6681ee9fb95756c999acbb62f5d/animal-sniffer-annotations-1.18.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.squareup.okio/okio/1.13.0/a9283170b7305c8d92d25aff02a6ab7e45d06cbe/okio-1.13.0.jar:/Users/grahampolley/.m2/repository/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.jar:/Users/grahampolley/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.42.Final/ccaacf418a9e486b65e82c47bed66439119c5fdb/netty-resolver-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.42.Final/e02700b574d3a0e2100308f971f0753ac8700e7c/netty-common-4.1.42.Final.jar:/Users/grahampolley/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Applications/IntelliJ
> IDEA CE.app/Contents/lib/idea_rt.jar" org.polleyg.TemplatePipeline
> --project=etl-demo-269105 --runner=DirectRunner --streaming=true
> --tempLocation=gs://micro-batch-test/tmp
> --stagingLocation=gs://micro-batch-test/tmp
> --workerZone=australia-southeast1-a
> --region=asia-northeast1/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java
>
> -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:50370,suspend=y,server=n
> -ea
> -javaagent:/Users/grahampolley/Library/Caches/IdeaIC2018.2/groovyHotSwap/gragent.jar
>
> -javaagent:/Users/grahampolley/Library/Caches/IdeaIC2018.2/captureAgent/debugger-agent.jar=file:/private/var/folders/_t/1lxhy17n2mlc0_4rb3_3thq40001g7/T/capture.props
> -Dfile.encoding=UTF-8 -classpath
> "/Users/grahampolley/Dev/gcp-batch-ingestion-bigquery/out/production/classes:/Users/grahampolley/Dev/gcp-batch-ingestion-bigquery/out/production/resources:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.30/c21f55139d8141d2231214fb1feaf50a1edca95e/slf4j-log4j12-1.7.30.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-google-cloud-dataflow-java/2.22.0/f251d2508e2c55a6c44fdf4c2c66ce4407b84908/beam-runners-google-cloud-dataflow-java-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-direct-java/2.22.0/541d048b57a0bb189310ebc85c6266241a732952/beam-runners-direct-java-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-io-google-cloud-platform/2.22.0/cdd9b337c909cd5ed85e379b7ae1c1a26dfba55f/beam-sdks-java-io-google-cloud-platform-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-google-cloud-platform-core/2.22.0/c3509a7b95a3533775cbaa61e438a482e2cdd5bf/beam-sdks-java-extensions-google-cloud-platform-core-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-expansion-service/2.22.0/6180ba45374960b0b639df0ae47c3b3736045612/beam-sdks-java-expansion-service-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-java-fn-execution/2.22.0/cdeafa6cb9365caf63da4232e326b2eb2ddcde33/beam-runners-java-fn-execution-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-core-construction-java/2.22.0/94c4145c597f700a4f3807282f916427dab5ef70/beam-runners-core-construction-java-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-protobuf/2.22.0/580a4bf3424738b30f840e29d4e5946df7017a84/beam-sdks-java-extensions-protobuf-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-fn-execution/2.22.0/5da4c54468112faf1f8b7f8bbf5524fe49d6c8ef/beam-sdks-java-fn-execution-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-sdks-java-extensions-protobuf/2.22.0/ec7722d88ddceb57806a76f7285bd8c0a1cc7fcd/beam-vendor-sdks-java-extensions-protobuf-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-core/2.22.0/73d5b6a07c6ae1579506c6224895efbd53d69f3f/beam-sdks-java-core-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-model-pipeline/2.22.0/3b5b6715b2f9b6e7e35b51acd904e30bde77d10e/beam-model-pipeline-2.22.0.jar:/Users/grahampolley/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.bigtable/bigtable-client-core/1.13.0/a86858be799ad73f1139017f91c4777147552fd4/bigtable-client-core-1.13.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-model-job-management/2.22.0/c155afa58e1cc382e7de62b9a67e0b918f2f5c1c/beam-model-job-management-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-model-fn-execution/2.22.0/8f180cc8d7ed9b0bfbbede2901ce9c62f10b50fd/beam-model-fn-execution-2.22.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.dropwizard.metrics/metrics-core/3.2.6/62fe170cffeded1cef60e9e3402a93b45ce14327/metrics-core-3.2.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.30/b5a4b6d16ab13e34a88fae84c35cd5d68cac922c/slf4j-api-1.7.30.jar:/Users/grahampolley/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-guava-26_0-jre/0.1/f309c3047ca99428e567afa42d233fb3e839bde1/beam-vendor-guava-26_0-jre-0.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.bigdataoss/gcsio/2.1.3/6dfefdc343ae4e5d1ff6df480f0457c073bfc19c/gcsio-2.1.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.bigdataoss/util/2.1.3/3a0d8e8883268fd1d8ec4241084fdb290bf7782b/util-2.1.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-clouddebugger/v2-rev20200313-1.30.9/f08af56f1e6cf8673d9408b99874eda40e6960fe/google-api-services-clouddebugger-v2-rev20200313-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-dataflow/v1b3-rev20200305-1.30.9/66c26eb008746cc8003c9dad799b2420cfdd75ed/google-api-services-dataflow-v1b3-rev20200305-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-bigquerystorage/0.125.0-beta/4ade9a93e57bdf032d8325e3fc0bd3cad75f153b/google-cloud-bigquerystorage-0.125.0-beta.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-bigquery/1.106.0/bc7232727654ba78edd63f8c95dd0ea47c4950f8/google-cloud-bigquery-1.106.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-bigquery/v2-rev20191211-1.30.9/4c8b0534651f0000d3320307cea24e145c237a5e/google-api-services-bigquery-v2-rev20191211-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-healthcare/v1beta1-rev20200307-1.30.9/89a426386623daf7528b0869a710673c3bf7c156/google-api-services-healthcare-v1beta1-rev20200307-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-pubsub/v1-rev20200312-1.30.9/aad0ede541289a8ca1a280facd0fe159fa9cebd4/google-api-services-pubsub-v1-rev20200312-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud.datastore/datastore-v1-proto-client/1.6.3/b0ffaeb68af9cb99772aa9933aa9baa0521568dc/datastore-v1-proto-client-1.6.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-cloudresourcemanager/v1-rev20200311-1.30.9/6c675cf826d657a08e2fece69317de4319d3192f/google-api-services-cloudresourcemanager-v1-rev20200311-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-storage/v1-rev20200326-1.30.9/5c0856380fff4fe8d11f5df03c653c16f69388e1/google-api-services-storage-v1-rev20200326-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client-java6/1.30.9/cef0e59174a3d6066598046209e44c9c34bb3132/google-api-client-java6-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client-jackson2/1.30.9/b6af6877119b2932ca293b14b33aa5f7393a358c/google-api-client-jackson2-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-core-http/1.92.4/d2776edaa94a4eee16332966adb575d8c5e63210/google-cloud-core-http-1.92.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client/1.30.9/8fc8641740ee0c26715738fb4d902779b96217d6/google-api-client-1.30.9.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-spanner/1.49.1/19dfd15e889a41c306b5dd62a622db6c471cb6fc/google-cloud-spanner-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-bigtable/1.9.1/8df06e49c314f524e6fdc11878ad5647f2a992b6/google-cloud-bigtable-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-core-grpc/1.92.2/af95d438cc2f05cdbc0c31101ca2fd17da753ee9/google-cloud-core-grpc-1.92.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/gax-grpc/1.54.0/93d75eab29a7923f407efdae59a87e032b86d9f7/gax-grpc-1.54.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.cloud/google-cloud-core/1.92.5/97ca5c35f34ee67dde8f2d4e9b940f0a96393c66/google-cloud-core-1.92.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/gax-httpjson/0.70.1/15b2c7844b9561ea036e009c153cf1ef70701afb/gax-httpjson-0.70.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/gax/1.54.0/1f1668868b8b3fd5fc248d80c16dd9f09afc9180/gax-1.54.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-alts/1.29.0/aa559eb5550bb772e42665c271cfe4dfcf22116f/grpc-alts-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.auth/google-auth-library-oauth2-http/0.20.0/f33d4d6c91a68826816606a2208990eea93fcb2a/google-auth-library-oauth2-http-0.20.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-all/1.27.2/fd3c66752c77b7f2c1e345f852862af92dd36a83/grpc-all-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-auth/1.29.0/aa89399565038ca930510b8428588cc919086eba/grpc-auth-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.auth/google-auth-library-credentials/0.20.0/87a91a373e64ba5c3cdf8cc5cf54b189dd1492f8/google-auth-library-credentials-0.20.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.oauth-client/google-oauth-client-java6/1.30.6/6c744e05b99325030625d9c7df7f3306f3e80a4e/google-oauth-client-java6-1.30.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.oauth-client/google-oauth-client/1.30.6/5c9a0e62e0cbda2bad495fa72147a331617f2e99/google-oauth-client-1.30.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-jackson2/1.34.2/fc5cd5e7b6252bae6a65ef2a1091476afb25b7c7/google-http-client-jackson2-1.34.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-protobuf/1.33.0/aa38d441440dc8ed72509b94580403c58d910caa/google-http-client-protobuf-1.33.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-appengine/1.34.1/d5d538385a2fa2b23d1831c35af66ac2e5445a0f/google-http-client-appengine-1.34.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client/1.34.2/58b4c500b5ce58271f1a6c5f6767aa8721158ea4/google-http-client-1.34.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.10.2/528de95f198afafbcfb0c09d2e43b6e0ea663ec/jackson-databind-2.10.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.10.2/3a13b6105946541b8d4181a0506355b5fae63260/jackson-annotations-2.10.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.10.2/73d4322a6bda684f676a2b5fe918361c4e5c7cca/jackson-core-2.10.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.10.5/7f1d89817cd20a32444d5ab4160f035ab9b864e7/joda-time-2.10.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-grpc-1_26_0/0.3/6871e7d0b92cd4983064166ee44d633e3800ef0f/beam-vendor-grpc-1_26_0-0.3.jar:/Users/grahampolley/.m2/repository/args4j/args4j/2.33/args4j-2.33.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-netty/1.27.2/dd4760eafb989dd1997c8f918eb376717b36e4aa/grpc-netty-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.opencensus/opencensus-contrib-grpc-util/0.24.0/1ac2f0c1a02c7ec0cd244e59228f687a90dfddab/opencensus-contrib-grpc-util-0.24.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-okhttp/1.27.2/5081585398fe003550717016928d18ec849f44f4/grpc-okhttp-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-testing/1.27.2/f0d7e35e5d0cb23a99afc26f2372e30b5e97cc01/grpc-testing-1.27.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-netty-shaded/1.29.0/80078c0aec3f2146ce94b5c0bddd722fa1f8e502/grpc-netty-shaded-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-grpclb/1.29.0/98635bf61b76cfdc3b38442290d4939120f9f51c/grpc-grpclb-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-core/1.29.0/b051a14a67c97bb9bbe0b9a03b5d7e7080e7b960/grpc-core-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.opencensus/opencensus-contrib-http-util/0.24.0/6d96406c272d884038eb63b262458df75b5445/opencensus-contrib-http-util-0.24.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.opencensus/opencensus-api/0.24.0/f974451b19007ce820f433311ce8adb88e2b7d2c/opencensus-api-0.24.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-cloud-pubsub-v1/1.85.1/5c59ba127376fc68b7c7f38f9b837e280b32b6dd/grpc-google-cloud-pubsub-v1-1.85.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-cloud-bigtable-v2/1.9.1/9edbe1087db616c9023b56f5581551002c95caad/grpc-google-cloud-bigtable-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-cloud-bigtable-admin-v2/1.9.1/974022119ccf41a7443a4a092c77559966a18b4d/grpc-google-cloud-bigtable-admin-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/grpc-google-common-protos/1.17.0/9c5b4135ff50dbc0d299ba8288c5f8e829470994/grpc-google-common-protos-1.17.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-protobuf/1.29.0/57a50ff72c0da333c31ad45616ea17f7d3b91b1c/grpc-protobuf-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-stub/1.29.0/2ded7df15689f2e606189456a91f7a9e1da9feee/grpc-stub-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-protobuf-lite/1.29.0/22e5ff00b942ee573c9dbadfc5e690b151a6d5e4/grpc-protobuf-lite-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-api/1.29.0/4f067a7b1657ad95c00fe958e8a66c8f8446c9f/grpc-api-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-context/1.29.0/1d8a441110f86f8927543dc3007639080441ea3c/grpc-context-1.29.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigtable-v2/1.9.1/a02ccc9174a00d97d9d1bad53a1bb9e456de8714/proto-google-cloud-bigtable-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-pubsub-v1/1.85.1/cddb71da7461d091effaaf51ed773fbc766263d6/proto-google-cloud-pubsub-v1-1.85.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-spanner-admin-database-v1/1.49.1/b1c18dc081a1858060577767af00dec47f0ddeeb/proto-google-cloud-spanner-admin-database-v1-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-datastore-v1/0.85.0/30d57cf3b3b0ae569975fcbf2dff36b958afb2c0/proto-google-cloud-datastore-v1-0.85.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigtable-admin-v2/1.9.1/fe8584d617c60d1f8cf203c33bd0358f4423e257/proto-google-cloud-bigtable-admin-v2-1.9.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-spanner-admin-instance-v1/1.49.1/d0e502423bfed8985952dcb2f909cb8417b6923e/proto-google-cloud-spanner-admin-instance-v1-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-iam-v1/0.13.0/ed3d62b64aa23a3decf324c8988eb4aae9f36e94/proto-google-iam-v1-0.13.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-spanner-v1/1.49.1/d9d02615204a606caf87c91f67c0ff9c31e1ba52/proto-google-cloud-spanner-v1-1.49.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api/api-common/1.8.1/e89befb19b08ad84b262b2f226ab79aefcaa9d7f/api-common-1.8.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java-util/3.11.4/99a6a669e55f3d587ac8eb61857f7b81d0bbd7f7/protobuf-java-util-3.11.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/29.0-jre/801142b4c3d0f0770dd29abea50906cacfddd447/guava-29.0-jre.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.42.Final/819e7b5f2005770cf7558c04276fff080331c6df/netty-codec-http2-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.42.Final/7b816d9f37ddcb68f6c1b9b0d7b5a98bfac40911/netty-handler-proxy-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.42.Final/5f71267aa784d0e6c5ec09fb988339d244b205a0/netty-codec-http-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.42.Final/fc6546be5df552d9729f008d8d41a6dee28127aa/netty-handler-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.17.Final/b1e5acbde8c444c656131238ac6ab9e73f694300/netty-tcnative-boringssl-static-2.0.17.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1beta1/0.90.0/29e65631a68262e36cc1868768029767bfd8bd72/proto-google-cloud-bigquerystorage-v1beta1-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1alpha2/0.90.0/2a1296e1b7d2dfe668813dbda443a3bcd477b35e/proto-google-cloud-bigquerystorage-v1alpha2-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1beta2/0.90.0/c59fa6741ab5d84a2de4c11f07f41510751e52c4/proto-google-cloud-bigquerystorage-v1beta2-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-cloud-bigquerystorage-v1/0.90.0/6c6431f01d5ed66f1c7cf4c5f95deda4f73876c4/proto-google-cloud-bigquerystorage-v1-0.90.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.api.grpc/proto-google-common-protos/1.18.0/b69bc5942255dc9dd2d2de432c4b99a5b483679/proto-google-common-protos-1.18.0.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/3.11.4/7ec0925cc3aef0335bbc7d57edfd42b0f86f8267/protobuf-java-3.11.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-vendor-bytebuddy-1_10_8/0.1/16661d7186f2d10440f1bb1c1e27e51f5306abe8/beam-vendor-bytebuddy-1_10_8-0.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.perfmark/perfmark-api/0.19.0/2bfc352777fa6e27ad1e11d11ea55651ba93236b/perfmark-api-0.19.0.jar:/Users/grahampolley/.m2/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/Users/grahampolley/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.tukaani/xz/1.8/c4f7d054303948eb6a4066194253886c8af07128/xz-1.8.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.3.4/dac170e4594de319655ffb62f41cbd6dbb5e601e/error_prone_annotations-2.3.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.11/f6d42fee5110c227bac18a550a297e028f2fb21a/httpclient-4.5.11.jar:/Users/grahampolley/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.6.2/bd1b74a5d170686362091c7cf596bbc3adf5c09b/log4j-api-2.6.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.conscrypt/conscrypt-openjdk-uber/2.2.1/59a346d64c0ddca750c5c877e274d5e6278e53ce/conscrypt-openjdk-uber-2.2.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.65/be28c46df75712f1fece48639c05633c8217f71/classgraph-4.8.65.jar:/Users/grahampolley/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/grahampolley/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/grahampolley/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.20/b8df472b31e1f17c232d2ad78ceb1c84e00c641b/commons-compress-1.20.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.auto.value/auto-value-annotations/1.7.1/58d76a9ec581f7c6d33f3343de9b2ba04a0ae799/auto-value-annotations-1.7.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.flogger/google-extensions/0.5.1/4f1d862216754651fc1e1f2b614746810e68a4ff/google-extensions-0.5.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.13/853b96d3afbb7bf8cc303fe27ee96836a10c1834/httpcore-4.4.13.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.3/ba035118bc8bac37d7eff77700720999acd9986d/j2objc-annotations-1.3.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.flogger/flogger-system-backend/0.5.1/b66d3bedb14da604828a8693bb24fd78e36b0e9e/flogger-system-backend-0.5.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.threeten/threetenbp/1.4.1/8e0de17c5077523503812837fa069459103fb714/threetenbp-1.4.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.13/3f18e1aa31031d89db6f01ba05d501258ce69d2c/commons-codec-1.13.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-lang3/3.5/6c6c702c89bfff3cd9e80b04d668c5e190d588c6/commons-lang3-3.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.code.gson/gson/2.8.6/9180733b7df8542621dc12e21e87557e8c99b8cb/gson-2.8.6.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.android/annotations/4.1.1.4/a1678ba907bf92691d879fef34e1a187038f9259/annotations-4.1.1.4.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.42.Final/dd355f01dc00f93aaebe805b05026d7cf57c60ec/netty-codec-socks-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.42.Final/b1d5ed85a558fbbadc2783f869fbd0adcd32b07b/netty-codec-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.42.Final/857502e863c02c829fdafea61c3fda6bda01d0af/netty-transport-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.42.Final/6e6fc9178d1f1401aa0d6b843341efb91720f2cd/netty-buffer-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/javax.annotation/javax.annotation-api/1.3.2/934c04d3cfef185a8008e7bf34331b79730a9d43/javax.annotation-api-1.3.2.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.guava/failureaccess/1.0.1/1dcf1de382a0bf95a3d8b0849546c88bac1292c9/failureaccess-1.0.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/b421526c5f297295adef1c886e5246c39d4ac629/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.checkerframework/checker-qual/2.11.1/8c43bf8f99b841d23aadda6044329dad9b63c185/checker-qual-2.11.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.google.flogger/flogger/0.5.1/71d1e2cef9cc604800825583df56b8ef5c053f14/flogger-0.5.1.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.checkerframework/checker-compat-qual/2.5.5/435dc33e3019c9f019e15f01aa111de9d6b2b79c/checker-compat-qual-2.5.5.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.18/f7aa683ea79dc6681ee9fb95756c999acbb62f5d/animal-sniffer-annotations-1.18.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/com.squareup.okio/okio/1.13.0/a9283170b7305c8d92d25aff02a6ab7e45d06cbe/okio-1.13.0.jar:/Users/grahampolley/.m2/repository/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.jar:/Users/grahampolley/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.42.Final/ccaacf418a9e486b65e82c47bed66439119c5fdb/netty-resolver-4.1.42.Final.jar:/Users/grahampolley/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.42.Final/e02700b574d3a0e2100308f971f0753ac8700e7c/netty-common-4.1.42.Final.jar:/Users/grahampolley/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Applications/IntelliJ
> IDEA CE.app/Contents/lib/idea_rt.jar" org.polleyg.TemplatePipeline
> --project=etl-demo-269105 --runner=DirectRunner --streaming=true
> --tempLocation=gs://micro-batch-test/tmp
> --stagingLocation=gs://micro-batch-test/tmp
> --workerZone=australia-southeast1-a --region=asia-northeast1Connected to the
> target VM, address: '127.0.0.1:50370', transport: 'socket'15:25:23,603 0
> [main] WARN org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource -
> Created subscription
> projects/etl-demo-269105/subscriptions/test-micro-batching_beam_-2917199999199772085
> to topic projects/etl-demo-269105/topics/test-micro-batching. Note this
> subscription WILL NOT be deleted when the pipeline terminates15:25:35,827
> 12224 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads - Writing BigQuery temporary
> files to
> gs://micro-batch-test/tmp/BigQueryWriteTemp/beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8/
> before loading them.15:34:52,585 568982 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryRowWriter - Opening
> TableRowWriter to
> gs://micro-batch-test/tmp/BigQueryWriteTemp/beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8/ae94e5dc-68eb-4bb1-a1e2-1e747110be8e.15:35:00,216
> 576613 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Waiting for jobs to
> complete.15:35:00,216 576613 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:01,532 577929 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:02,408 578805 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - Started BigQuery
> job: GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}.bq show -j
> --format=prettyjson --project_id=etl-demo-269105
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:02,408
> 578805 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:02,909 579306 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE15:35:03,659 580056 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - No BigQuery job
> with job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> found.15:35:03,659 580056 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> not found, so retrying with that id15:35:03,676 580073
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Load job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed, will retry: {"errorResult":{"message":"Not found: Table
> etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in location
> australia-southeast1","reason":"notFound"},"errors":[{"message":"Not found:
> Table etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in
> location australia-southeast1","reason":"notFound"}],"state":"DONE"}. Next
> job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:03,676
> 580073 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:05,818 582215 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:06,475 582872 [direct-runner-worker] WARN
> org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer - Request
> failed with code 409, performed 0 retries due to IOExceptions, performed 0
> retries due to unsuccessful status codes, HTTP framework says request can be
> retried, (caller responsible for retrying):
> https://bigquery.googleapis.com/bigquery/v2/projects/etl-demo-269105/jobs.
> 15:35:06,505 582902 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} already exists,
> will not retry inserting
> it:com.google.api.client.googleapis.json.GoogleJsonResponseException: 409
> Conflict{ "code" : 409, "errors" : [ { "domain" : "global", "message"
> : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "reason" : "duplicate" } ], "message" : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "status" : "ALREADY_EXISTS"} at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:444)
> at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:244)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:229)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.lambda$startLoad$79287d2b$1(WriteTables.java:408)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:274)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source) at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:237)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:124)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.finishBundle(ParDoEvaluator.java:265)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.finishBundle(DoFnLifecycleManagerRemovingTransformEvaluator.java:73)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:188)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)15:35:06,523 582920
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:07,041 583438 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE15:35:07,730 584127 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - No BigQuery job
> with job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> found.15:35:07,731 584128 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> not found, so retrying with that id15:35:07,731 584128
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Load job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed, will retry: {"errorResult":{"message":"Not found: Table
> etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in location
> australia-southeast1","reason":"notFound"},"errors":[{"message":"Not found:
> Table etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in
> location australia-southeast1","reason":"notFound"}],"state":"DONE"}. Next
> job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:07,731
> 584128 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:10,461 586858 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:11,096 587493 [direct-runner-worker] WARN
> org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer - Request
> failed with code 409, performed 0 retries due to IOExceptions, performed 0
> retries due to unsuccessful status codes, HTTP framework says request can be
> retried, (caller responsible for retrying):
> https://bigquery.googleapis.com/bigquery/v2/projects/etl-demo-269105/jobs.
> 15:35:11,098 587495 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} already exists,
> will not retry inserting
> it:com.google.api.client.googleapis.json.GoogleJsonResponseException: 409
> Conflict{ "code" : 409, "errors" : [ { "domain" : "global", "message"
> : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "reason" : "duplicate" } ], "message" : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "status" : "ALREADY_EXISTS"} at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:444)
> at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:244)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:229)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.lambda$startLoad$79287d2b$1(WriteTables.java:408)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:274)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source) at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:237)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:124)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.finishBundle(ParDoEvaluator.java:265)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.finishBundle(DoFnLifecycleManagerRemovingTransformEvaluator.java:73)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:188)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)15:35:11,098 587495
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:11,520 587917 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE15:35:12,502 588899 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - No BigQuery job
> with job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> found.15:35:12,503 588900 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> not found, so retrying with that id15:35:12,503 588900
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Load job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed, will retry: {"errorResult":{"message":"Not found: Table
> etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in location
> australia-southeast1","reason":"notFound"},"errors":[{"message":"Not found:
> Table etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in
> location australia-southeast1","reason":"notFound"}],"state":"DONE"}. Next
> job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:12,503
> 588900 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:14,707 591104 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:15,273 591670 [direct-runner-worker] WARN
> org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer - Request
> failed with code 409, performed 0 retries due to IOExceptions, performed 0
> retries due to unsuccessful status codes, HTTP framework says request can be
> retried, (caller responsible for retrying):
> https://bigquery.googleapis.com/bigquery/v2/projects/etl-demo-269105/jobs.
> 15:35:15,274 591671 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} already exists,
> will not retry inserting
> it:com.google.api.client.googleapis.json.GoogleJsonResponseException: 409
> Conflict{ "code" : 409, "errors" : [ { "domain" : "global", "message"
> : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "reason" : "duplicate" } ], "message" : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "status" : "ALREADY_EXISTS"} at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:444)
> at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:244)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:229)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.lambda$startLoad$79287d2b$1(WriteTables.java:408)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:274)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source) at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:237)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:124)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.finishBundle(ParDoEvaluator.java:265)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.finishBundle(DoFnLifecycleManagerRemovingTransformEvaluator.java:73)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:188)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)15:35:15,274 591671
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:15,659 592056 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE15:35:16,224 592621 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - No BigQuery job
> with job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> found.15:35:16,224 592621 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> not found, so retrying with that id15:35:16,224 592621
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Load job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed, will retry: {"errorResult":{"message":"Not found: Table
> etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in location
> australia-southeast1","reason":"notFound"},"errors":[{"message":"Not found:
> Table etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in
> location australia-southeast1","reason":"notFound"}],"state":"DONE"}. Next
> job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:16,224
> 592621 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:22,555 598952 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:23,194 599591 [direct-runner-worker] WARN
> org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer - Request
> failed with code 409, performed 0 retries due to IOExceptions, performed 0
> retries due to unsuccessful status codes, HTTP framework says request can be
> retried, (caller responsible for retrying):
> https://bigquery.googleapis.com/bigquery/v2/projects/etl-demo-269105/jobs.
> 15:35:23,195 599592 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} already exists,
> will not retry inserting
> it:com.google.api.client.googleapis.json.GoogleJsonResponseException: 409
> Conflict{ "code" : 409, "errors" : [ { "domain" : "global", "message"
> : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "reason" : "duplicate" } ], "message" : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "status" : "ALREADY_EXISTS"} at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:444)
> at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:244)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:229)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.lambda$startLoad$79287d2b$1(WriteTables.java:408)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:274)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source) at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:237)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:124)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.finishBundle(ParDoEvaluator.java:265)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.finishBundle(DoFnLifecycleManagerRemovingTransformEvaluator.java:73)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:188)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)15:35:23,195 599592
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:23,697 600094 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE15:35:24,254 600651 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - No BigQuery job
> with job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> found.15:35:24,254 600651 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> not found, so retrying with that id15:35:24,254 600651
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Load job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed, will retry: {"errorResult":{"message":"Not found: Table
> etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in location
> australia-southeast1","reason":"notFound"},"errors":[{"message":"Not found:
> Table etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in
> location australia-southeast1","reason":"notFound"}],"state":"DONE"}. Next
> job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:24,254
> 600651 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:30,901 607298 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:31,542 607939 [direct-runner-worker] WARN
> org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer - Request
> failed with code 409, performed 0 retries due to IOExceptions, performed 0
> retries due to unsuccessful status codes, HTTP framework says request can be
> retried, (caller responsible for retrying):
> https://bigquery.googleapis.com/bigquery/v2/projects/etl-demo-269105/jobs.
> 15:35:31,542 607939 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} already exists,
> will not retry inserting
> it:com.google.api.client.googleapis.json.GoogleJsonResponseException: 409
> Conflict{ "code" : 409, "errors" : [ { "domain" : "global", "message"
> : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "reason" : "duplicate" } ], "message" : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "status" : "ALREADY_EXISTS"} at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:444)
> at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:244)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:229)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.lambda$startLoad$79287d2b$1(WriteTables.java:408)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:274)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source) at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:237)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:124)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.finishBundle(ParDoEvaluator.java:265)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.finishBundle(DoFnLifecycleManagerRemovingTransformEvaluator.java:73)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:188)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)15:35:31,543 607940
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:32,042 608439 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE15:35:32,588 608985 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - No BigQuery job
> with job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> found.15:35:32,588 608985 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> not found, so retrying with that id15:35:32,588 608985
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Load job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed, will retry: {"errorResult":{"message":"Not found: Table
> etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in location
> australia-southeast1","reason":"notFound"},"errors":[{"message":"Not found:
> Table etl-demo-269105:etl_demo_fun.micro_batch_test_xxx was not found in
> location australia-southeast1","reason":"notFound"}],"state":"DONE"}. Next
> job id
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-015:35:32,588
> 608985 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - Job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> failed. retrying.15:35:44,165 620562 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables - Loading 1 files into
> GenericData{classInfo=[datasetId, projectId, tableId],
> {datasetId=etl_demo_fun, projectId=etl-demo-269105,
> tableId=micro_batch_test_xxx}} using job GenericData{classInfo=[jobId,
> location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}}, job id iteration
> 015:35:44,688 621085 [direct-runner-worker] WARN
> org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer - Request
> failed with code 409, performed 0 retries due to IOExceptions, performed 0
> retries due to unsuccessful status codes, HTTP framework says request can be
> retried, (caller responsible for retrying):
> https://bigquery.googleapis.com/bigquery/v2/projects/etl-demo-269105/jobs.
> 15:35:44,689 621086 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} already exists,
> will not retry inserting
> it:com.google.api.client.googleapis.json.GoogleJsonResponseException: 409
> Conflict{ "code" : 409, "errors" : [ { "domain" : "global", "message"
> : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "reason" : "duplicate" } ], "message" : "Already Exists: Job
> etl-demo-269105:australia-southeast1.beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0",
> "status" : "ALREADY_EXISTS"} at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:444)
> at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:244)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:229)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:170)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.lambda$startLoad$79287d2b$1(WriteTables.java:408)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:209)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:274)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source) at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:237)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:124)
> at
> org.apache.beam.runners.direct.ParDoEvaluator.finishBundle(ParDoEvaluator.java:265)
> at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.finishBundle(DoFnLifecycleManagerRemovingTransformEvaluator.java:73)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:188)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)15:35:44,689 621086
> [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers - job
> beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0
> started15:35:45,758 622155 [direct-runner-worker] INFO
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl - BigQuery job
> GenericData{classInfo=[jobId, location, projectId],
> {jobId=beam_load_templatepipelinegrahampolley0612052535e06af6fe_7e9fba10c9a24d50b81cdd6364d913c8_18825aa47e9218cc06def366031f7280_00001_00000-0,
> location=australia-southeast1, projectId=etl-demo-269105}} completed in
> state DONE{noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
