[
https://issues.apache.org/jira/browse/BEAM-12356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510783#comment-17510783
]
Claire McGinty edited comment on BEAM-12356 at 3/22/22, 5:31 PM:
-----------------------------------------------------------------
[~bhulette] Sure, here's an example! They're also all over Scio's [integration
test logs|https://github.com/spotify/scio/actions/workflows/it-tests.yml] if
you want further context.
{{2022-03-22 11:17:11.415-0400 error [ManagedChannelOrphanWrapper] {*}{{*}}{*}~
Channel \{0} was not shutdown properly!!!{*} *~*}}{{ Make sure to call
shutdown()/shutdownNow() and wait until awaitTermination() returns true.
}}{{{}java.lang.RuntimeException: ManagedChannel allocation site{}}}{{{}at
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93){}}}{{{}at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53){}}}{{{}at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44){}}}{{{}at
io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630){}}}{{{}at
io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:395){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1900(InstantiatingGrpcChannelProvider.java:82){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:240){}}}{{{}at
com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:250){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:228){}}}{{{}at
com.google.api.gax.rpc.ClientContext.create(ClientContext.java:205){}}}{{{}at
com.google.cloud.bigquery.storage.v1.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:132){}}}{{{}at
com.google.cloud.bigquery.storage.v1.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:144){}}}{{{}at
com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.<init>(BigQueryWriteClient.java:130){}}}{{{}at
com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.create(BigQueryWriteClient.java:111){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1353){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:145){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:518){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:463){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:180){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:117){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:155){}}}{{{}at
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:216){}}}{{{}at
org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:88){}}}{{{}at
org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:80){}}}{{{}at
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:159){}}}{{{}at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213){}}}{{{}at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67){}}}{{{}at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:323){}}}{{{}at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:309){}}}
was (Author: clairemcginty):
[~bhulette] Sure, here's an example! They're also all over our [integration
test logs|https://github.com/spotify/scio/actions/workflows/it-tests.yml] if
you want further context.
{{2022-03-22 11:17:11.415-0400 error [ManagedChannelOrphanWrapper] {*}{*}{*}~
Channel \{0} was not shutdown properly!!!{*} *~*}}{{ Make sure to call
shutdown()/shutdownNow() and wait until awaitTermination() returns true.
}}{{{}java.lang.RuntimeException: ManagedChannel allocation site{}}}{{{}at
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93){}}}{{{}at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53){}}}{{{}at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44){}}}{{{}at
io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630){}}}{{{}at
io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:395){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1900(InstantiatingGrpcChannelProvider.java:82){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:240){}}}{{{}at
com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:250){}}}{{{}at
com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:228){}}}{{{}at
com.google.api.gax.rpc.ClientContext.create(ClientContext.java:205){}}}{{{}at
com.google.cloud.bigquery.storage.v1.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:132){}}}{{{}at
com.google.cloud.bigquery.storage.v1.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:144){}}}{{{}at
com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.<init>(BigQueryWriteClient.java:130){}}}{{{}at
com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.create(BigQueryWriteClient.java:111){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1353){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:145){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:518){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:463){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:180){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:117){}}}{{{}at
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:155){}}}{{{}at
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:216){}}}{{{}at
org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:88){}}}{{{}at
org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:80){}}}{{{}at
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:159){}}}{{{}at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213){}}}{{{}at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67){}}}{{{}at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:323){}}}{{{}at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:309){}}}
> BigQueryWriteClient in DatasetServiceImpl is not closed, which causes
> "ManagedChannel allocation site" exceptions
> -----------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-12356
> URL: https://issues.apache.org/jira/browse/BEAM-12356
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.29.0, 2.32.0, 2.33.0
> Reporter: Minbo Bae
> Assignee: Reuven Lax
> Priority: P2
> Fix For: 2.34.0
>
> Attachments: bigquery_grpc.log
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> [BigQueryWriteClient|https://github.com/apache/beam/blob/v2.29.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L461]
> in DatasetServiceImpl (added at [https://github.com/apache/beam/pull/14309)]
> is not closed. This causes the error logs in gRPC orphan channel clean up.
> See "bigquery_grpc.log" in attachments which is extracted from GCP Dataflow.
> I don't think this issue affect pipeline runs except the error logs, but
> could you take a look at that?
> A similar issue is reported for {{CloudBigtableIO}} at
> [https://github.com/googleapis/java-bigtable-hbase/issues/2658]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
