Well, that is odd. It looks like the underlying client is closed, which is unexpected.
Do you see any retries in your pipeline? Also, what runner are you using? @Ahmed Abualsaud <[email protected]> this might be interesting to you too On Tue, Dec 5, 2023 at 9:39 PM [email protected] <[email protected]> wrote: > I'm using version 2.51.0 and The configuration is like this > > write > .withoutValidation() > .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER) > .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND) > .withExtendedErrorInfo() > .withMethod(Write.Method.STORAGE_WRITE_API) > .withTriggeringFrequency(Duration.standardSeconds(10)) > .withAutoSharding().optimizedWrites() > .withFailedInsertRetryPolicy(retryTransientErrors()); > > > On Tue, Dec 5, 2023 at 11:20 AM John Casey via user <[email protected]> > wrote: > >> Hi, >> >> Could you add some more detail? Which beam version are you using? >> >> >> On Tue, Dec 5, 2023 at 1:52 PM [email protected] <[email protected]> wrote: >> >>> Any one has experience in writing to BQ using storage api >>> >>> I tried to use it because according to the document it is more efficient >>> but I got error below >>> >>> 2023-12-05 04:01:29.741 PST >>> Error message from worker: java.lang.RuntimeException: >>> java.lang.IllegalStateException >>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$12(StorageApiWritesShardedRecords.java:573) >>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:790) >>> Caused by: java.lang.IllegalStateException >>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:496) >>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$1.pin(BigQueryServicesImpl.java:1403) >>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$12(StorageApiWritesShardedRecords.java:565) >>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:790) >>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn$DoFnInvoker.invokeProcessElement(Unknown >>> Source) >>> >>
