[
https://issues.apache.org/jira/browse/BEAM-12356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457008#comment-17457008
]
Joachim Isaksson commented on BEAM-12356:
-----------------------------------------
Sorry for the delay on the stack trace, I have a couple;
The first one in PolicyTagManager, sorry for the cutoff but the rest of the
stack trace contains info I can't share (if it's important I can anonymize and
reshare that part too)
{code:java}
2021-12-10T08:51:17.5680978Z SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=7,
target=datacatalog.googleapis.com:443} was not shutdown properly!!! ~*~*~*
2021-12-10T08:51:17.5681797Z Make sure to call shutdown()/shutdownNow() and
wait until awaitTermination() returns true.
2021-12-10T08:51:17.5682542Z java.lang.RuntimeException: ManagedChannel
allocation site
2021-12-10T08:51:17.5683413Z at
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
2021-12-10T08:51:17.5684193Z at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
2021-12-10T08:51:17.5684923Z at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
2021-12-10T08:51:17.5685642Z at
io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:625)
2021-12-10T08:51:17.5686557Z at
io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:264)
2021-12-10T08:51:17.5687379Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:383)
2021-12-10T08:51:17.5688208Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1900(InstantiatingGrpcChannelProvider.java:82)
2021-12-10T08:51:17.5689528Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:239)
2021-12-10T08:51:17.5690369Z at
com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
2021-12-10T08:51:17.5691106Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:249)
2021-12-10T08:51:17.5691889Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:227)
2021-12-10T08:51:17.5692624Z at
com.google.api.gax.rpc.ClientContext.create(ClientContext.java:205)
2021-12-10T08:51:17.5693347Z at
com.google.cloud.datacatalog.v1beta1.stub.GrpcPolicyTagManagerStub.create(GrpcPolicyTagManagerStub.java:223)
2021-12-10T08:51:17.5694156Z at
com.google.cloud.datacatalog.v1beta1.stub.PolicyTagManagerStubSettings.createStub(PolicyTagManagerStubSettings.java:313)
2021-12-10T08:51:17.5695534Z at
com.google.cloud.datacatalog.v1beta1.PolicyTagManagerClient.<init>(PolicyTagManagerClient.java:144)
2021-12-10T08:51:17.5696475Z at
com.google.cloud.datacatalog.v1beta1.PolicyTagManagerClient.create(PolicyTagManagerClient.java:125)
2021-12-10T08:51:17.5697332Z at
com.google.cloud.datacatalog.v1beta1.PolicyTagManagerClient.create(PolicyTagManagerClient.java:116)
{code}
The second one in DatasetServiceImpl/extractFiles;
{code:java}
2021-12-10T08:56:26.8375862Z SEVERE: *~*~*~ Channel
ManagedChannelImpl{logId=15, target=bigquerystorage.googleapis.com:443} was not
shutdown properly!!! ~*~*~*
2021-12-10T08:56:26.8376960Z Make sure to call shutdown()/shutdownNow() and
wait until awaitTermination() returns true.
2021-12-10T08:56:26.8377785Z java.lang.RuntimeException: ManagedChannel
allocation site
2021-12-10T08:56:26.8378602Z at
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
2021-12-10T08:56:26.8379484Z at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
2021-12-10T08:56:26.8380368Z at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
2021-12-10T08:56:26.8381637Z at
io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:625)
2021-12-10T08:56:26.8382550Z at
io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:264)
2021-12-10T08:56:26.8383507Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:383)
2021-12-10T08:56:26.8384512Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1900(InstantiatingGrpcChannelProvider.java:82)
2021-12-10T08:56:26.8385513Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:239)
2021-12-10T08:56:26.8386426Z at
com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
2021-12-10T08:56:26.8387599Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:249)
2021-12-10T08:56:26.8389010Z at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:227)
2021-12-10T08:56:26.8390053Z at
com.google.api.gax.rpc.ClientContext.create(ClientContext.java:205)
2021-12-10T08:56:26.8390933Z at
com.google.cloud.bigquery.storage.v1beta2.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:136)
2021-12-10T08:56:26.8392547Z at
com.google.cloud.bigquery.storage.v1beta2.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:145)
2021-12-10T08:56:26.8393642Z at
com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.<init>(BigQueryWriteClient.java:128)
2021-12-10T08:56:26.8394532Z at
com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.create(BigQueryWriteClient.java:109)
2021-12-10T08:56:26.8395431Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1271)
2021-12-10T08:56:26.8396337Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:144)
2021-12-10T08:56:26.8397268Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:518)
2021-12-10T08:56:26.8399990Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:463)
2021-12-10T08:56:26.8403381Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:180)
2021-12-10T08:56:26.8404228Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:117)
2021-12-10T08:56:26.8404879Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead$2.processElement(BigQueryIO.java:1123)
2021-12-10T08:56:26.8405690Z at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead$2$DoFnInvoker.invokeProcessElement(Unknown
Source)
2021-12-10T08:56:26.8407260Z at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:232)
2021-12-10T08:56:26.8408344Z at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:191)
2021-12-10T08:56:26.8409180Z at
org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79)
2021-12-10T08:56:26.8409966Z at
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244)
2021-12-10T08:56:26.8410834Z at
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
2021-12-10T08:56:26.8411561Z at
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
2021-12-10T08:56:26.8412197Z at
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
2021-12-10T08:56:26.8412782Z at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2021-12-10T08:56:26.8413320Z at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2021-12-10T08:56:26.8413862Z at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2021-12-10T08:56:26.8414450Z at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2021-12-10T08:56:26.8415193Z at
java.base/java.lang.Thread.run(Thread.java:829) {code}
> BigQueryWriteClient in DatasetServiceImpl is not closed, which causes
> "ManagedChannel allocation site" exceptions
> -----------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-12356
> URL: https://issues.apache.org/jira/browse/BEAM-12356
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.29.0, 2.32.0, 2.33.0
> Reporter: Minbo Bae
> Assignee: Reuven Lax
> Priority: P2
> Fix For: 2.34.0
>
> Attachments: bigquery_grpc.log
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> [BigQueryWriteClient|https://github.com/apache/beam/blob/v2.29.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L461]
> in DatasetServiceImpl (added at [https://github.com/apache/beam/pull/14309)]
> is not closed. This causes the error logs in gRPC orphan channel clean up.
> See "bigquery_grpc.log" in attachments which is extracted from GCP Dataflow.
> I don't think this issue affect pipeline runs except the error logs, but
> could you take a look at that?
> A similar issue is reported for {{CloudBigtableIO}} at
> [https://github.com/googleapis/java-bigtable-hbase/issues/2658]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
