Jacek Wislicki created FLINK-30681:
--------------------------------------
Summary: Pulsar-Flink connector corrupts its output topic
Key: FLINK-30681
URL: https://issues.apache.org/jira/browse/FLINK-30681
Project: Flink
Issue Type: Bug
Components: Connectors / Pulsar
Affects Versions: 1.15.3
Reporter: Jacek Wislicki
When PulsarSink writes a message to its output topic, the topic gets
permanently corrupted and cannot be used anymore (even with newly created
subscriptions).
We have isolated this behaviour to a minimal project demonstrating the problem
available on [GitHub|https://github.com/JacekWislicki/vp-test5]:
# There are 2 topics: IN and OUT
# IN is subscribed by a Flink's InToOutJob (with PulsarSource) and writes to
OUT (with PulsarSink)
# OUT is subscribed by a Pulsar's OutReadFunction
# When we write directly to OUT (e.g., with OutTopicProducer), OutReadFunction
gets each message from its backlog and processes it with no issue (the ledger
position updates)
# When we write to IN (e.g., with InTopicProducer), InToOutJob reads the
message, processes it and writes to OUT
# OutReadFunction reads the message, the ledger position updates, but nothing
happens
## Further messages written to OUT are not read as OUT is blocked on the last
message from Flink
## Truncating OUT does not help, neither does unsubscribing or creating a new
subscription
Reproduced with Pulsar 2.9.1, 2.9.2 and 2.10.2.
The issue does not occur when we use our custom temporary old SinkFunction
implementation based on a Pulsar producer writing to OUT.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
