[
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Sela updated BEAM-744:
---------------------------
Summary: UnboundedKafkaReader should return as soon as it can. (was: A
runner should be able to override KafkaIO max wait properties.)
> UnboundedKafkaReader should return as soon as it can.
> -----------------------------------------------------
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}},
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}},
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the
> DirectRunner, and I can add that they are now limiting the Spark runner
> (which reads in defined time frames, which may be smaller then the wait time
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which
> may be passed when creating the reader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
