[
https://issues.apache.org/jira/browse/BEAM-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Boenders updated BEAM-6708:
----------------------------------
Description:
When '_read_from_pubsub' (runnners/direct/transform_evaluator.py:448) is being
invoked, it will create a Pubsub SubscriberClient. This will create a grpc
channel which is not being closed. When you run a pipeline with the direct
runner which has a pubsub source in it, it will run eventually into an error:
"too many open files". This is because these grpc channels are never being
closed.
The suggestion is to wrap the code after the creation of the client in a
try/finally block and close the channel (exposed on
sub_client.api.transport._channel) in the finally block.
Some extra info about the issue:
https://github.com/googleapis/google-cloud-python/issues/5523
was:
When '_read_from_pubsub' (runnners/direct/transform_evaluator.py:448) is being
invoked, it will create a Pubsub SubscriberClient. This will create a grpc
channel which is not being closed. When you run a pipeline with the direct
runner which has a pubsub source in it, it will run eventually into an error:
"too many open files". This is because these grpc channels are never being
closed.
The suggestion is to wrap the code after the creation of the client in a
try/finally block and close the channel (exposed on
sub_client.api.transport._channel) in the finally block.
> Too many open files due grpc channel not being closed (direct runnner Pubsub
> source)
> ------------------------------------------------------------------------------------
>
> Key: BEAM-6708
> URL: https://issues.apache.org/jira/browse/BEAM-6708
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.10.0
> Reporter: Stefan Boenders
> Priority: Major
>
> When '_read_from_pubsub' (runnners/direct/transform_evaluator.py:448) is
> being invoked, it will create a Pubsub SubscriberClient. This will create a
> grpc channel which is not being closed. When you run a pipeline with the
> direct runner which has a pubsub source in it, it will run eventually into an
> error: "too many open files". This is because these grpc channels are never
> being closed.
> The suggestion is to wrap the code after the creation of the client in a
> try/finally block and close the channel (exposed on
> sub_client.api.transport._channel) in the finally block.
>
> Some extra info about the issue:
> https://github.com/googleapis/google-cloud-python/issues/5523
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
