[
https://issues.apache.org/jira/browse/FLINK-33729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17837157#comment-17837157
]
Yufan Sheng commented on FLINK-33729:
-------------------------------------
[~Weijie Guo] Yep, this is a valid ticket. But it's not from the Flink side. We
need to figure out why the transaction didn't works on Pulsar. So I think this
ticket should be submitted to Pulsar community.
[~rtrojczak] I can't find any checkpoint configuration from your sample code. I
can only see two line of codes that enable checkpoint. I think this is the main
reason that your code fails as expect. Flink checkpoint needs a lot of
configuration to use. Such as the storage. You can check the link below to get
your application proper configured.
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/#enabling-and-configuring-checkpointing
> Events are getting lost when an exception occurs within a processing function
> -----------------------------------------------------------------------------
>
> Key: FLINK-33729
> URL: https://issues.apache.org/jira/browse/FLINK-33729
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
> Affects Versions: 1.15.3
> Reporter: Rafał Trójczak
> Priority: Major
>
> We have a Flink job using a Pulsar source that reads from an input topic, and
> a Pulsar sink that is writing to an output topic. Both Flink and Pulsar
> connector are of version 1.15.3. The Pulsar version that I use is 2.10.3.
> Here is a simple project that is intended to reproduce this problem:
> [https://github.com/trojczak/flink-pulsar-connector-problem/]
> All of my tests were done on my local Kubernetes cluster using the Flink
> Kubernetes Operator and Pulsar is running on my local Docker. But the same
> problem occurred on a "normal" cluster.
> Expected behavior: When an exception is thrown within the code (or a
> TaskManager pod is restarted for any other reason, e.g. OOM exception), the
> processing should be picked up from the last event sent to the output topic.
> Actual behavior: The events before the failure are sent correctly to the
> output topic, next some of the events from the input topic are missing, then
> from some point the events are being processed normally until the next
> exception is thrown, and so on. Finally, from 100 events that should be sent
> from the input topic to the output topic, only 40 are sent.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
