[
https://issues.apache.org/jira/browse/BEAM-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313394#comment-17313394
]
Chamikara Madhusanka Jayalath commented on BEAM-11880:
------------------------------------------------------
There are two issues that will prevent Kafka for working for DirectRunner today.
(1) Portable direct runner does not support streaming pipelines:
https://issues.apache.org/jira/browse/BEAM-7514.
(2) SDF doesn't work for unbounded reads on portable runner:
https://issues.apache.org/jira/browse/BEAM-11998
This should work if you try to use x-lang Kafka IO on a batch pipeline with
max_num_records or max_read_time.
See here for a previous thread on this:
https://lists.apache.org/x/thread.html/rd6ad2a0a44d7cb5ea45184cecacc4711af7fc87a2b56da0eb973f6d1@%3Cuser.beam.apache.org%3E
> KafkaIO doesn’t work with Direct Runner on Python SDK
> -----------------------------------------------------
>
> Key: BEAM-11880
> URL: https://issues.apache.org/jira/browse/BEAM-11880
> Project: Beam
> Issue Type: Bug
> Components: cross-language, io-py-kafka, sdk-py-core
> Reporter: Nir Gazit
> Priority: P2
>
> When running a pipeline that reads from KAFKA it seems to be stuck in a loop
> resetting the offset all the time but never actually reading anything.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
