[
https://issues.apache.org/jira/browse/CRUNCH-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Micah Whitacre updated CRUNCH-621:
----------------------------------
Attachment: CRUNCH-621b.patch
Took a different approach where it would check after repeated empty failures
because Kafka could still drop data after checking proactively. Also will cut
down on calls to Kafka only when it fails vs calling from each task.
(hopefully spacing nits not a problem this time)
> Kafka no data to consume could get stuck
> ----------------------------------------
>
> Key: CRUNCH-621
> URL: https://issues.apache.org/jira/browse/CRUNCH-621
> Project: Crunch
> Issue Type: Bug
> Components: IO
> Reporter: Micah Whitacre
> Assignee: Micah Whitacre
> Attachments: CRUNCH-621.patch, CRUNCH-621b.patch
>
>
> With the KafkaSource and specifically the KafkaRecordReader, if there is no
> data to consume from a partition because all data fell off Kafka then we can
> end up in a situation where the KafkaRecordReader will poll forever as it
> never gets data to reach its ending offset.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
