[ https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585107#comment-15585107 ]
Amit Sela commented on BEAM-744: -------------------------------- That's Correct, I updated the JIRA, thanks! > A runner should be able to override KafkaIO max wait properties. > ---------------------------------------------------------------- > > Key: BEAM-744 > URL: https://issues.apache.org/jira/browse/BEAM-744 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions > Reporter: Amit Sela > > KafkaIO has two "wait" properties: > {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, > default: 5 seconds. > {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, > default: 10 msec. > [~rangadi] mentioned some of these were set to due to limitations of the > DirectRunner, and I can add that they are now limiting the Spark runner > (which reads in defined time frames, which may be smaller then the wait time > and so never actually read). > This feels like defaults should be set for optimal read from Kafka, while a > runner may override those if it needs to. > [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which > may be passed when creating the reader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)