[
https://issues.apache.org/jira/browse/BEAM-12359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347221#comment-17347221
]
Brian Hulette commented on BEAM-12359:
--------------------------------------
The "was not shutdown properly!!!" indicates we're not properly managing the
lifecycle of our BQ Storage client. We should be calling close() on the client
created
[here|https://github.com/apache/beam/blob/1eaa8694528f44c8254992ad30da98aa8fb999cf/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L520]
I'm not sure how serious of an issue this is. In my experience with other GCP
clients it's usually harmless, but the fact that it fails so loudly makes me
worry there could be rare but serious effects.
> BigQuery Storage Read API source throws NullPointerException when a source
> table is not found
> ---------------------------------------------------------------------------------------------
>
> Key: BEAM-12359
> URL: https://issues.apache.org/jira/browse/BEAM-12359
> Project: Beam
> Issue Type: Bug
> Components: extensions-java-gcp
> Affects Versions: 2.29.0
> Reporter: Filip Popić
> Assignee: Vachan Shetty
> Priority: P2
>
> When Reading from BigQueryIO using DIRECT_READ getting
> {code:java}
> ~ Channel ManagedChannelImpl{logId=1,
> target=bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~
> Make sure to call shutdown()/shutdownNow() and wait until awaitTermination()
> returns true. {code}
> {code:java}
> 2021-05-12 13:25:49.319 CEST Error message from worker:
> java.lang.NullPointerException
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:105)
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:40)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:294)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:216)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:200)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:179)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:82)
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
> Sbt/Scala minimal example to reproduce [https://github.com/fpopic/BEAM-12359]
> (if required can try to make it in Java).
> Relates to user mailing list
> [question|https://lists.apache.org/x/thread.html/r705ca8f0ea9517f148637be59e509ce80fa75b22c98ad63acd551065@%3Cuser.beam.apache.org%3E].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
