[
https://issues.apache.org/jira/browse/FLINK-27399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufan Sheng updated FLINK-27399:
--------------------------------
Description:
The Pulsar connector didn't use the consuming position from the checkpoint.
They just commit the position to Pulsar after the checkpoint is complete. And
the connector starts to consume message from Pulsar directly by the offset
stored on the Pulsar subscription.
This causes the test could be failed in some situations. The start cursor
(position on Pulsar) would be reset to the wrong position, which caused the
results didn't match the desired records.
This issue fixes
# FLINK-23944
# FLINK-24872
# FLINK-25815
# FLINK-25884
# FLINK-26177
# FLINK-26237
# FLINK-26721
# FLINK-27833
was:
The Pulsar connector didn't use the consuming position from the checkpoint.
They just commit the position to Pulsar after the checkpoint is complete. And
the connector start to consume message from Pulsar directly by the offset
stored on the Pulsar subscription.
This causes the test could be failed in some situations. The start cursor
(position on Pulsar) would be reset to the wrong position which causes the
results didn't match the desired records.
This issue fixes
# FLINK-23944
# FLINK-24872
# FLINK-25815
# FLINK-25884
# FLINK-26177
# FLINK-26237
# FLINK-26721
# FLINK-27833
Although the test failure message could be various. They are truly sharing the
same cause.
h2. How to fix this issue:
SourceEvent protocol for limiting the {{Consumer.seek}} operation.
The Pulsar source needs to seek the desired consuming position when
bootstrapping. The seeking action couldn’t be executed concurrently. We have
designed a [new
mechanism|https://github.com/apache/flink/pull/17119#pullrequestreview-746035072].
> Pulsar connector didn't set start consuming position correctly
> --------------------------------------------------------------
>
> Key: FLINK-27399
> URL: https://issues.apache.org/jira/browse/FLINK-27399
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
> Affects Versions: 1.14.4, 1.15.0, 1.16.0
> Reporter: Yufan Sheng
> Assignee: Yufan Sheng
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.16.0, 1.15.2, 1.14.6
>
>
> The Pulsar connector didn't use the consuming position from the checkpoint.
> They just commit the position to Pulsar after the checkpoint is complete. And
> the connector starts to consume message from Pulsar directly by the offset
> stored on the Pulsar subscription.
> This causes the test could be failed in some situations. The start cursor
> (position on Pulsar) would be reset to the wrong position, which caused the
> results didn't match the desired records.
> This issue fixes
> # FLINK-23944
> # FLINK-24872
> # FLINK-25815
> # FLINK-25884
> # FLINK-26177
> # FLINK-26237
> # FLINK-26721
> # FLINK-27833
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
