[
https://issues.apache.org/jira/browse/BEAM-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan updated BEAM-6708:
-------------------------
Description:
When '_read_from_pubsub' (runnners/direct/transform_evaluator.py:448) is being
invoked, it will create a Pubsub SubscriberClient. This will create a grpc
channel which is not being closed. When you run a pipeline with the direct
runner which has a pubsub source in it, it will run eventually into an error:
"too many open files". This is because these grpc channels are never being
closed.
The suggestion is to wrap the code after the creation of the client in a
try/finally block and close the channel (exposed on
sub_client.api.transport._channel) in the finally block.
was:
When
_read_from_pubsub is being invoked, it will create a Pubsub SubscriberClient.
This will create a grpc channel which is not being closed. When you run a
pipeline with the direct runner which has a pubsub source in it, it will run
eventually into an error: "too many open files". This is because these grpc
channels are never being closed.
The suggestion is to wrap the code after the creation of the client in a
try/finally block and close the channel (exposed on
sub_client.api.transport._channel) in the finally block.
> Too many open files due grpc channel not being closed (direct runnner Pubsub
> source)
> ------------------------------------------------------------------------------------
>
> Key: BEAM-6708
> URL: https://issues.apache.org/jira/browse/BEAM-6708
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.10.0
> Reporter: Stefan
> Priority: Major
>
> When '_read_from_pubsub' (runnners/direct/transform_evaluator.py:448) is
> being invoked, it will create a Pubsub SubscriberClient. This will create a
> grpc channel which is not being closed. When you run a pipeline with the
> direct runner which has a pubsub source in it, it will run eventually into an
> error: "too many open files". This is because these grpc channels are never
> being closed.
> The suggestion is to wrap the code after the creation of the client in a
> try/finally block and close the channel (exposed on
> sub_client.api.transport._channel) in the finally block.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
