Chamikara Madhusanka Jayalath created BEAM-11991:
----------------------------------------------------
Summary: Python Kafka source not emitting messages for streaming
pipelines
Key: BEAM-11991
URL: https://issues.apache.org/jira/browse/BEAM-11991
Project: Beam
Issue Type: Bug
Components: cross-language, runner-flink
Reporter: Chamikara Madhusanka Jayalath
Assignee: Boyuan Zhang
To reproduce:
* Startup a local Kafka cluster: [https://kafka.apache.org/quickstart]
* Setup topics:
bin/kafka-console-consumer.sh --topic mytopic1 --from-beginning
--bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic mytopic2 --from-beginning
--bootstrap-server localhost:9092
* Setup a Beam virtualenv and run a pipeline that reads from Kafka. For
example: [https://wtools.io/paste-code/b4je
]python ./pipeline.py --bootstrap_servers=localhost:9092 --in_topic=mytopic1
--out_topic=mytopic2 --runner=FlinkRunner --streaming
* Publish messages as kv pairs.
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic1
--property "parse.key=true" --property "key.separator=:"
>a:b
>c:d
>e:f
* Messages do not get pushed to subsequent steps.
* Following seems to be working fine.
* X-lang Bounded read with Flink
* X-lang Kafka sink and with Flink
[~boyuanz] could you take a look to rule out any SDF/unbounded read related
issues ?
cc: [~mxm] and [~angoenka] for Flink issues.
Beam user thread:
https://lists.apache.org/x/thread.html/r9c74a8a7efa4b14f2f1d5f77ce8f12128cf1071861c1627f00702415@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
