GitHub user rangadi opened a pull request:
https://github.com/apache/beam/pull/3492
[BEAM-2551] KafkaIO reader blocks indefinitely in case of network issues
If the KafaIO source reader on the worker can't reach the server, Kafka
consumer blocks forever inside UnboundedReader.start(). Users have no
indication of the error. It is better if start() fails with an error.
It is easy to reproduce with Kafka command line. I reported it on Kafka
users list here :
https://lists.apache.org/thread.html/98cebefacbd65b0d6c6817fe0b5197e26bc60252e72d05fced91e628@%3Cusers.kafka.apache.org%3E
It blocks inside Kafka client. Fortunately it can be unblocked with
KafkaConsumer.wakeup(). This PR initializes partition on a separate thread and
cancels it if it does not complete in one minute. Added a unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rangadi/beam timeout
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3492.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3492
----
commit 9238c36263b4c885f53b43dc26ff5c5f820ed908
Author: Raghu Angadi <[email protected]>
Date: 2017-07-04T06:54:10Z
intialize partition on a separate thread.
commit 65a4bc15d48d25f2729bb6a6b8ac42bca5ea9999
Author: Raghu Angadi <[email protected]>
Date: 2017-07-04T06:00:12Z
Add a unit test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
