[
https://issues.apache.org/jira/browse/BEAM-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073230#comment-16073230
]
ASF GitHub Bot commented on BEAM-2551:
--------------------------------------
GitHub user rangadi opened a pull request:
https://github.com/apache/beam/pull/3492
[BEAM-2551] KafkaIO reader blocks indefinitely in case of network issues
If the KafaIO source reader on the worker can't reach the server, Kafka
consumer blocks forever inside UnboundedReader.start(). Users have no
indication of the error. It is better if start() fails with an error.
It is easy to reproduce with Kafka command line. I reported it on Kafka
users list here :
https://lists.apache.org/thread.html/98cebefacbd65b0d6c6817fe0b5197e26bc60252e72d05fced91e628@%3Cusers.kafka.apache.org%3E
It blocks inside Kafka client. Fortunately it can be unblocked with
KafkaConsumer.wakeup(). This PR initializes partition on a separate thread and
cancels it if it does not complete in one minute. Added a unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rangadi/beam timeout
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3492.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3492
----
commit 9238c36263b4c885f53b43dc26ff5c5f820ed908
Author: Raghu Angadi <[email protected]>
Date: 2017-07-04T06:54:10Z
intialize partition on a separate thread.
commit 65a4bc15d48d25f2729bb6a6b8ac42bca5ea9999
Author: Raghu Angadi <[email protected]>
Date: 2017-07-04T06:00:12Z
Add a unit test
----
> KafkaIO reader blocks indefinitely if servers are not reachable
> ---------------------------------------------------------------
>
> Key: BEAM-2551
> URL: https://issues.apache.org/jira/browse/BEAM-2551
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> If the KafaIO source reader on the worker can't reach the server, Kafka
> consumer blocks forever inside {{UnboundedReader.start()}}. Users have no
> indication of the error. It is better if start() fails with an error.
> It is easy to reproduce in Kafka. I reported it on Kafka users list here :
> https://lists.apache.org/thread.html/98cebefacbd65b0d6c6817fe0b5197e26bc60252e72d05fced91e628@%3Cusers.kafka.apache.org%3E
> It blocks inside Kafka client. Fortunately it can be unblocked with
> KafkaConsumer.wakeup(). We could run initialization in another thread and
> cancel it if takes longer than a minute.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
