Amit Sela created BEAM-744:
------------------------------
Summary: A runner should be able to override KafkaIO max wait
properties.
Key: BEAM-744
URL: https://issues.apache.org/jira/browse/BEAM-744
Project: Beam
Issue Type: Bug
Components: sdk-java-extensions
Reporter: Amit Sela
Assignee: James Malone
KafkaIO has three "wait" properties:
{{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}},
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}},
default: 10 msec.
[~rangadi] mentioned some of these were set to due to limitations of the
DirectRunner, and I can add that they are now limiting the Spark runner (which
reads in defined time frames, which may be smaller then the wait time and so
never actually read).
This feels like defaults should be set for optimal read from Kafka, while a
runner may override those if it needs to.
[~rangadi] also mentioned that this could be set in {{PipelineOptions}} which
may be passed when creating the reader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
